Corrupted journal in a Linux LVM How to recover

Not real Linux expert but here is the problem.

Shutdown this machine and then it would not reboot.

From everything I can tell it looks like the journal file in the logical volume is corrupted. I have tried everything I can think of to get the volume mounted.

Anybody have any ideas on how to recover this, of course there is no backup of the filesystem but that is another story.

Here is what I think is all of the pertinent.

Booting 'Red Hat Enterprise Linux ES (2.6.9-42.ELsmp)'

root (hd0,0)
Filesystem type is ext2fs partion type 0x83
kernel /vmlinux-2.6.9-42-ELsmp ro root=/dev/VolGroup00/LogVol00 rhgb quiet
[Linux-bz Image,setup=0x1400, size=0x15f464]
initrd /initrd-2.6.42.ELsmp,img
[linux-initrd@0x37ecc0000,0x123cce bytes]

Uncompressing Linux ... OK, booting the kernel.
Red Hat nash version 4.2.1.8 starting
Reading all physical volumes. This may take awhile ...
Found volume group "VolGroup00" using metadata type lvm2
2 logical volume(s) in volume group "VolGroup00" now active

EXT3-fs error (device dm-0) ext3_check_descriptor: Block bitmap for
group 0 not in group (block 41471)
EXT3-fs group descriptors corrupted
mount: error 22 mounting ext3
mount: error 2 mounting none
switchroot: /initrd/dev failed: 2
Kernel panic not syncing: Attempt to kill init

Booted from SystemRescueCD

found nothing in /dev/mapper

Performed the following steps

root@sysresccd% dmsetup mknodes
root@sysresccd% vgmknodes
root@sysresccd% vgscan
Reading all physical volumes. This may take awhile ...
Found volume group "VolGroup00" using metadata type lvm2
root@sysresccd% vgchange -ay
2 logical volume(s) in volume group "VolGroup00" now active

Now I could see the logical volumes,

/dev/VolGroup00/LogVol01 is swap
/dev/VolGroup00/LogVol00 should be /

Performed:

root@sysresccd% mkdir /mnt/mydir
root@sysresccd% mount -t ext3 /dev/VolGroup00/LogVol00 /mnt/mydir
mount: wrong fs type, bad option, bad superblock on /dev/mapper/VolGroup00/LogVol00
missing codepage or helper program, or other errors
In some cases useful info is found in syslog - try
dmesg | tail or so.

Then performed:

root@sysresccd% fsck /dev/VolGroup00/LogVol00

fsck 1.41.3 (12-oct-2008)
e2fsck 1.41.3 (12-Oct-2008)
fsck.ext3: Group descriptors look bad.. trying backup blocks
fsck.ext3: Invalid argument while checking ext3 journal for /dev/VolGroup00/LogVol00

Then ran mk2fs -n /dev/VolGroup/LogVol00 to get alternate superblocks

Tried running fsck again using alternate superblocks but still get the same errors.

When trying to mount with -o noload get these errors in syslog

EXT3-fs error (device dm-0) ext3_check_descriptor: Block bitmap for group 0 not in group (block 41471)
EXT3-fs group descriptors corrupted!

root@sysresccd /mnt/nsips % vgdisplay -v
Finding all volume groups
Finding volume group "VolGroup00"
Fixing up missing size (68.26 GB) for PV /dev/sdb2
Fixing up missing size (68.36 GB) for PV /dev/sdc1
Fixing up missing size (68.26 GB) for PV /dev/sdb2
Fixing up missing size (68.36 GB) for PV /dev/sdc1

--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 136.59 GB
PE Size 32.00 MB
Total PE 4371
Alloc PE / Size 4369 / 136.53 GB
Free PE / Size 2 / 64.00 MB
VG UUID JINuth-kuv3-NG9k-3q34-wIHJ-wkQR-4fk1tW

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID fOHDQv-ekvG-dZQU-xus0-Tlbt-0kok-OgmmwT
LV Write Access read/write
LV Status available
# open 0
LV Size 134.59 GB
Current LE 4307
Segments 2
Allocation inherit
Read ahead sectors auto

• currently set to 256
  Block device 253:0

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID LavZiq-3mE3-Awhj-Oyfz-zlXW-kBEG-01j00w
LV Write Access read/write
LV Status available
# open 0
LV Size 1.94 GB
Current LE 62
Segments 1
Allocation inherit
Read ahead sectors auto

• currently set to 256
  Block device 253:1

--- Physical volumes ---
PV Name /dev/sdb2
PV UUID udJZJb-kkzA-RBxr-rWoI-3qaf-3xtb-gFtgUO
PV Status allocatable
Total PE / Free PE 2184 / 2

PV Name /dev/sdc1
PV UUID Wwu9xr-HARk-knlw-cqfT-pRQo-5zb9-UqrwZn
PV Status allocatable
Total PE / Free PE 2187 / 0

root@sysresccd /mnt/nsips % pvscan -v
Wiping cache of LVM-capable devices
Wiping internal VG cache
Walking through all physical volumes
Fixing up missing size (68.26 GB) for PV /dev/sdb2
Fixing up missing size (68.36 GB) for PV /dev/sdc1
Fixing up missing size (68.26 GB) for PV /dev/sdb2
Fixing up missing size (68.36 GB) for PV /dev/sdc1
PV /dev/sdb2 VG VolGroup00 lvm2 [68.25 GB / 64.00 MB free]
PV /dev/sdc1 VG VolGroup00 lvm2 [68.34 GB / 0 free]
Total: 2 [136.59 GB] / in use: 2 [136.59 GB] / in no VG: 0 [0 ]

root@sysresccd /mnt/nsips % dumpe2fs /dev/VolGroup00/LogVol00
dumpe2fs 1.41.3 (12-Oct-2008)
dumpe2fs: Invalid argument while reading journal inode
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 04f9f61a-fdb8-4205-88ff-172cf08e89f5
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Default mount options: (none)
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 17645568
Block count: 35282944
Reserved block count: 1764147
Free blocks: 4365169
Free inodes: 15808913
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Fri Nov 28 17:09:50 2008
Last mount time: Fri Dec 19 17:26:17 2008
Last write time: Wed Jan 14 12:24:04 2009
Mount count: 8
Maximum mount count: -1
Last checked: Fri Nov 28 17:09:50 2008
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
First orphan inode: 17416274
Default directory hash: tea
Directory Hash Seed: 189832ed-6e78-40b6-8cac-a95d8a2bb09d
Journal backup: inode blocks
dumpe2fs: Invalid argument while reading journal inode

root@sysresccd /mnt/nsips % tune2fs -O ^has_journal /dev/VolGroup00/LogVol00
tune2fs 1.41.3 (12-Oct-2008)
tune2fs: Invalid argument while reading journal inode

root@sysresccd /mnt/nsips % debugfs /dev/VolGroup00/LogVol00
debugfs 1.41.3 (12-Oct-2008)
/dev/VolGroup00/LogVol00: Can't read an inode bitmap while reading inode bitmap
debugfs:

You can mount ext3 filesystems as ext2 filesystems.

Tried mounting as an ext2 file:

mount -t ext2 /dev/VolGroup00/LogVol00 /ramdisk/mydir

This fails with wrong fs type, bad option, bad superblock on /dev/VolGroup00/LogVol00,
missing codepage or other error.

Why didnt you use vgcfgrestore command?

What happens when you run e2fsck against the filesystem?

Edit: I see you tried running e2fsck already......

More than likely he did not make a backup with vgcfgbackup

Neo,

Correct.

Have you considered something like Linux Data Recovery Software - Data Recovery Tools ?

Linux Data Recovery Software Tools. Recover Linux Partition Data. LVM Recovery

Also, see: Recovering lost ext2 Linux filesystems

http://www.unixwiz.net/techtips/recovering-ext2.html

Before you do anything drastic, did you try locating a good superblock?

Linux maintains multiple copies of the superblock in every file system and you can use backup copies to restore a damaged primary super block.

For example, the following command displays primary and backup superblock location on /dev/sda2:

dumpe3fs /dev/hda2 | grep -i superblock

(or dumpe2fs)

Yes I have been trying to use this product. The Linux based version keeps locking up.
The windows version reports invalid group descriptor block.

Trying other vendors products now.

What complicates this is that you are running a logical volume.

I don't think these ext2/3 recovery tools are designed to work against logical volumes, unfortunately.

Neo,

Yes I did try alternate superblocks

Running that command though gives an error:

bad magic number in superblock while trying to open /dev/sda2
Couldn't find valid filesystem superblock

I only used /dev/sda2 as an example.

You need to use your device file ..... not sure if this will work (more than likely not), but can you try:

dumpe3fs /dev/VolGroup00/LogVol00 | grep -i superblock

Neo,

I am getting that impression about these tools. May have to go to a professional for this. Hate to do as we really don't have the budget.

Neo,

Actually that is the device on this system.

That is the first partition in the volume.

I also got the same results with /dev/VolGroup00/LogVol00

Make sure you do a disk-to-disk (dd) first and save a backup copy of the entire disk .....

First thing I did. Did the logical volume and the physical disks.

Well, here is a long shot, but I don't think it will work on a logical volume, try:

mkfs.ext2 -n /dev/sda2

Of course, if we are lucky, you might recover with:

e2fsck -b backup_superblock /dev/sda2

... anyway, I don't think this will work on a logical volume, unfortunately.....

BTW, can you post your /etc/lvmtab file?

Also, can you post the output of

lvdisplay -v /dev/VolGroup00/LogVol00