Hello,
Running AIX 7.2 on Power9 bare-metal (no LPAR and no NIM server), in the process of creating a guide on MKSYSB process.
I understand that MKSYSB is a backup of the rootvg and we can exclude stuff via exclude.rootvg file, the rest of the data volumes are mapped to the system as LUNs via multi-path FC connection to a SAN and that is backed up separately. The rootvg does not boot off SAN, it has two internal NVMe disks.
In the event that rootvg is corrupt and we need to restore rootvg , the process would be to modify the bootlist and reboot into SMS via SSH/Console or using the ASMI service processor.
Are there any critical steps we need to do pre or post-rootvg restore?
- Currently the rootvg is mirrored, once restored will it remain mirrored?
- Will i need to mess with the bootlist at all?
- Will the configuration of the external filesystems be preserved and mounted ready to use (if set to auto-mount at boot)?
- Running a power system in 24x7 prodution after a MKSYSB restore is viable or should one plan for a fresh re-install of AIX from install media at some point?
- Does one need access to AIX 7.2 install media at any point during a MKSYSB restore?
The Power9 system has a LTO tape library which is where I was going to write the MKSYSB backup to and boot off, there is no CD-ROM but there is USB - is it possible to boot MKSYSB off USB? Any cons or pros going with tape vs. USB? Most of the guides out there refer to a tape or dvd, though many of them were written years ago.
My plan is to learn from experience and create a MKSYSB via mksysb -iev /dev/rmt0, corrupt rootvg by rm -rf /etc, and reboot. I am pretty sure it will become very unhappy with a missing /etc - so proceed with another reboot (via FPO using Control Panel or ASMI) and boot to SMS for a MKSYSB restore.
Mainly looking for best practices / tips / suggestions around the process.
Hello c3rb3rus,
The AIX tool mksysb is very good and tries to do a lot for you. To answer your specific questions:-
- Currently the rootvg is mirrored, once restored will it remain mirrored?
The process reads a file that it is sensible to recreate before your run mksysb (or its created if it does not exist) called mkszfile The intermediate steps in building the recoverable image will include definitions for all sorts of things including the rootvg layout. Whilst you might not get it back block for block, the volume group will be restored to provide the defined capabilities. If you are mirrored, then a restore will require you to use at least two disks and it will recreate the mirror.
- Will i need to mess with the bootlist at all?
The bootlist should be set up for you during the restore process. It is easy to check afterwards anyway with bootlist -o normal so you can run it now to have a look and then be familiar with it after a recovery. Do you have both disks currently in the boot list? If it's just hdisk0 and you lose that, it will be awkward to get running on (for instance) hdisk1
You might also need to consider splitting the dump logical volume. I've struggled (maybe I was doing it wrong :o) where a system had sysdumpdevlv (or whatever it's now called) on a failed disk, whether it was mirrored or not. I set up the dump devices as two separate LVs, one on each PV. This can be difficult to enforce though if someone later comes along and either mirrors them, extends them or moves them both to the same disk.
- Will the configuration of the external filesystems be preserved and mounted ready to use (if set to auto-mount at boot)?
If the non-rootvg VGs are available when the server boots, it should recognise them (the VG serial number) and bring them online. If it refuses, you should be able to use importvg to bring them on. It can be awkward if you have sub-mounted filesystems from different volume groups.
- Running a power system in 24x7 prodution after a MKSYSB restore is viable or should one plan for a fresh re-install of AIX from install media at some point?
The mksysb should recreate everything you need. No further re-install should be required. If you rely on a third party backup product or home grown scripts, make sure that you have a small filesystem in the rootvg that contains everything you need to get them running again so you can restore your data if you have to move to a separate (cold) site. This may not need the backup catalogs if you can read these back in from the media. You would, of course, need to know your most recent backup media, so dare I suggest paper listing are sent out with any off-site process.
- Does one need access to AIX 7.2 install media at any point during a MKSYSB restore?
If you are recovering to dissimilar hardware then it would be good to have the install media available, however if you are using a recovery service company, they would normally help you through this anyway. If you are recovering over the existing machine then you have already got everything you need.
Further to this, should you choose to fire up a clone of your OS for whatever reason, you need to ensure that the network cable (physical or virtual) is unplugged else you may get IP conflicts as the clone server boots because everything that you have configured should be brought back. You can get access via the console and change the settings before connecting the network up again.
I hope that this helps.
Kind regards,
Robin
1 Like
Thanks for the very detailed response Robin, this relieves some of my "unknown pain points" around the process. The P9 is in pre-production so I wanted to take the time to blow away rootvg and restore it (for documentation purposes), better now than when the system is live and something goes sideways resulting in needing to restore rootvg for which I have no proven guide to rely on.
My plan is to restore MKSYSB to the same hardware and document, so should not have any IP conflicts and such.
Into the unknown I go 
I cannot seem to boot off tape in SMS.
I created a MKSYSB backup to tape using: mksysb -iev /dev/rmt0
Verified it using: lsmksysb -l -f /dev/rmt0
Booted to SMS menu, selected Tape as the boot device and set Media Type to SCSI, it throws an error: The selected devices were not detected in the system.
I then try using the "List All Devices" for Media Type, it does a scan and comes back with the tape drive....
PowerPC Firmware
Version FW910.20 (VL910_122)
SMS (c) Copyright IBM Corp. 2000,2017 All rights reserved.
-------------------------------------------------------------------------------
Select Device
Device Current Device
Number Position Name
1. - FC Tape drive
-------------------------------------------------------------------------------
Navigation keys:
M = return to Main Menu
ESC key = return to previous screen X = eXit System Management Services
-------------------------------------------------------------------------------
Type menu item number and press Enter or select Navigation key: 1
This seems to work in that the next screen I get a select task option where I choose the Service Media Boot, this fails with: No Operating System Found
I am stuck, seems that I cannot boot off the newly created MKSYSB tape.
Any ideas what I can try? I have since escaped from here and booted back to AIX normally.
Opened a ticket with IBM, according to the ATAPE readme it states that:
- Booting from a SAS attached tape drive is not supported on Power servers.
- Booting from a Fibre attached tape drive is not supported on Power servers.
So there goes being able to do a MKSYSB restore from a tape backup. yikes.
Do you have a DVD-RAM drive? That's an option (if I can remember the syntax and process)
Alternately, you can back up over the network to a NIM server, then recover form that.
Do either of these give you an option?
Robin
I don't have a cd-rom, but i do have a a few USB 3.0 inputs and after a call to IBM the route I went with was to create a USB MKSYSB (mksysb -iev /dev/usbms0), was able to restore the rootvg using this method.
1 Like
Excellent. Good to know you are sorted.
Just to clarify, I do mean DVD-RAM . A ROM would be Read Only (hence the O)
It's different to DVD-RW & DVD+RW and there is a bit of discussion on a Wikipedia page : DVD-RAM - Wikipedia
To get it all to work, we had to make a mksysb image on a local disk and then use mkdvd appropriately. Oddly it eventually calls readcd with the write flag set, but you probably won't need to worry about that. It did need a chunk of free space in the rootvg too, just over twice the size of the mksysb image. I also had to adjust the LV creating algorithm in mkdvd (I think) because it estimated too small and we were continually running out of space in the temporary LVs it creates.
Is there anything else we can try to help with?
Robin