The root filesystem was mirrored, someone/something stopped mirroring, and increased / and /home to ridiculous values (/ got increased to 102gb and its only using 4.3gb, so 98gb is free). Can I reduce the / (/dev/hd4) filesytem down WITHOUT corrupting the the OS?
I would do a: chfs -a size=10g /dev/hd4.
This is AIX 6.1 TL9 SP8 (6100-09-08-1642). I would even reduce the Filesystem size by 1gb @10 times if I had to.
The ultimate end is to remove hdisk1 from the rootvg and re-add it back as a mirror.
These are 146gb disks
lsvg -l rootvg (and added up the PP's) shows I have 538PP's allocated, with 7PP free with a PPsize of 256mb (which is 1.7gb approx.). hd10opt is occupying 40PP's and is on hdisk1 (as seen with lslv -l hd10opt)
I have already reduced /home to only occupy hdisk0 (as shown with lslv -m hd9). / is on hdisk 0 as well.
I need to migratepv -l hd10opt hdisk1 hdisk0 but it reports not enough space (of course due to only 1.7gb free on hdisk0).
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 1092 (279552 megabytes)
MAX LVs: 256 FREE PPs: 513 (131328 megabytes)
LVs: 13 USED PPs: 579 (148224 megabytes)
OPEN LVs: 11 QUORUM: 2 (Enabled)
TOTAL PVs: 2 VG DESCRIPTORS: 3
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 2 AUTO ON: yes
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
Not a straightforward fix, I'm afraid. I don't think you can reduce the filesystem in a simple way. If it is in-use, then there is pretty much no chance.
I've done this on Linux and it's messy. I ended up having to boot from media and forcing it from there, but I can't remember if AIX let's you do that anyway. If I was to be faced with this on AIX, then I would have two options but both of them are effectively Disaster Recovery options:-
An alternate to the first one might be to use savevg & restvg whilst adjusting the LV description file, but it's essentially the same.
They are pretty serious and obviously requires a chunk of down-time (as does the way I did with Linux) however you might be able to use one of these methods to clone your server to alternate hardware (or LPAR) and attach the application/data disks to it.
I'm sorry, but it's not a nice place to be.
When you say that you want to migrate hd10opt, may I ask why? I would hope that this is mirrored, or has someone managed to remove the mirrors to give themselves lots of free space to grow in to? Are these real, local disks? If they are virtualised by something, does that protect you against hardware failure? It might be, but that can be done wrong too.
Can you share the output from:-
lsvg -p rootvg
lsvg -l rootvg
lspv -l hdisk0 (and hdisk1)
lscfg -vl hdisk0
Thanks, in advance,
Robin
Thanks Robin, however, chfs actually worked without issue, as long as I did not reduce into the data. One thing I am happy with AIX, is the disk management.
Maybe it's an extra I was not aware of. I only got to AIX 6.1 TL7 before the hosts were removed and it's not something I would have gone looking for, so thanks for teaching me something today.
What state are your disks in now? If the rootvg are real local disks, I presume you want them mirrored and both bootable. What do you have for the output of:-
lspv -l hdisk0
lspv -l hdisk1
bootlist -o normal
Kind regards,
Robin
trust me, it was gut wrenching doing the chfs.
At this moment, I have a disk failing so I had to forego the re-mirroring and use the disk as a backup of the failing disk while a new disk is being procured. Once I reduced the / filesystem to be reasonable, the migratepv -l <hd10opt> hdisk1 hdisk0 worked. then I was able to reducevg rootvg hdisk1.
hdisk0:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
hd6 2 2 00..00..00..00..02 N/A
hd5 1 1 01..00..00..00..00 N/A
hd9 41 41 00..40..00..00..01 /home
hd3 8 8 00..00..00..00..08 /tmp
hd10opt 40 40 00..00..40..00..00 /opt
hd7 34 34 00..00..00..00..34 N/A
hd4 41 41 00..00..41..00..00 /
livedump 1 1 00..00..00..00..01 /var/adm/ras/livedump
hd8 1 1 00..00..01..00..00 N/A
hd9var 9 9 00..00..04..00..05 /var
lg_dumplv 16 16 00..16..00..00..00 N/A
hd2 24 24 00..00..00..00..24 /usr
hd11admin 1 1 00..00..01..00..00 /admin
hdisk1:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
newvol 546 546 110..109..109..109..109 /mnt
Here is the bootlist info.
cd0
hdisk0 blv=hd5 pathid=0
Well done for getting the disk out of the VG. That's usually the hardest part.
Assuming that you have now swapped the drive, you just need to add the new disk into rootvg, mirror the logical volumes, write the boot block and set the boot list to have both available.
This doc is quite good at leading you through:- IBM Mirroring the rootvg Volume Group for AIX - United States
Kind regards,
Robin
So when the person pulled the disk out of the mirror, open lv's were on both disks in a quasi operating mode/separation. The issues with the migrate was because when I tried to run reducevg (-d)(-f) rootvg hdisk1 it complained about the lv's being open on that PV. hd10opt was one of the lv's left on hdisk1 and I had no room on hdisk0 due to the extremely large root fs allocation.