Replaying log stuck during boot

UNIX AIX
1

I had some problems with NFS see here: http://www.unix.com/302790997-post9.html

and now I cannot boot the partition

when I boot I get stuck here

Saving Base Customize Data to boot disk                                         
Starting the sync daemon                                                        
Mounting the platform dump file system, /var/adm/ras/platform                   
mount: 0506-324 Cannot mount /dev/fwdump on /var/adm/ras/platform: A file or dir
ectory in the path name does not exist.                                         
Starting the error daemon                                                       
System initialization completed.                                                
Setting tunable parameters...complete                                           
Starting Multi-user Initialization                                              
 Performing auto-varyon of Volume Groups                                        
 Activating all paging spaces                                                   
0517-075 swapon: Paging device /dev/hd6 is already active.                      
swapon: Paging device /dev/paging00 activated.                                  
                                                                                
The current volume is: /dev/hd1                                                 
Primary superblock is valid.                                                    
                                                                                
The current volume is: /dev/hd10opt                                             
Primary superblock is valid.                                                    
 Performing all automatic mounts                                                
Replaying log for /dev/fslv02.                                                  
Replaying log for /dev/fslv01.                                                  
Replaying log for /dev/fslv00.

How I can resolve this issue ?

2

Boot to a rescue shell and do a "fsck", then try to boot normally again.

I hope this helps.

bakunin

3

Thanks, but even fsck is taking quite lot of time, it has been doing this for the last 1 day and still no outcome

fsck -y /dev/fslv00



The current volume is: /dev/fslv00
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/fslv00

Size of fslv00 = 150GB

Any other way for it ?

4

If this is an extra volume group you could use chvg -an vgName to turn off automount while in a boot/maint mode, and then continue the boot.

If these are filesystems in rootvg try doing chfs -A no /filesystem/name and booting. At least you are up and can do the fsck while up and running.

Hope this helps.

5

Thanks Bakunin and Michael,
I was able to bring the system online following the steps both of you mentioned,
however, doing fsck is taking lots of time. Is this normal behavior ?
How can one check if fsck is working okay or it is stuck (hanged)

6

iostat -d (just to ensure some regular i/o is occuring)
lvmstat (after enabling data collection - watch file system and jfs(2)log activity
errpt -c - maybe disk errors are being reported?

Hope this helps!

7

THanks,

# errpt -c
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

 

LABEL:          J2_FS_FULL
IDENTIFIER:     CAC5259E
CAC5259E   0314185013 I O SYSJ2          UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

# errpt -a  CAC5259E   
Date/Time:       Thu Mar 14 18:50:37  2013
Sequence Number: 624
Machine Id:      00C7780E4C00
Node Id:         rmandb
Class:           O
Type:            INFO
Resource Name:   SYSJ2

Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

Probable Causes
FILE SYSTEM FULL

        Recommended Actions
        INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
        REMOVE UNNECESSARY DATA FROM FILE SYSTEM
        USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED

Detail Data
JFS2 MAJOR/MINOR DEVICE NUMBER
000A 0004


iostat -d

System configuration: lcpu=2 drives=4
                " Disk history since boot not available. "

Should I increase the JFS2LOG or the Filesystem itself, I am afraid If I increase the size will it be able to replay the log or would I lose data ?

8

The JFS log (and the JFS2 log too) is of fixed size (1 PP). I have never tried, but i doubt that it even can be increased. You should increase the filesystem itself.

Just an observation: don't you have any monitoring script in place? The error message is from more than 3 weeks ago, so the FS was full for some time already (with nobody perhaps noticing).

I hope this helps.

bakunin