The crisis is over. I am just doing postmortem on how we handled it. So this is just an exercise in academics.
We have a mission critical system running on RED Hat Linux. It is a turnkey system "managed 100% by the vendor". I put this is quotes because we had an event last night that dramatically reduced that percentage for the evening!!
Evidently the Vendor tech was trying to resolve an issue with the internal ultrium tape drive. Backups were not functioning... so he decided to install something to fix the tape drive.. that something wiped out C++ libraries... (??). Evidently this trashed the system. We had a login at the console but it would not authenticate. The application interfaces with other systems, instruments were failing.
He asked for our assistance to rebuild C libraries. He had us download redhat isos and put them on CDs. He had us boot off cd using "linux rescue" at the boot: prompt. Then he had us attempt to reinstall C libs using rpm. It seems the rpm command could not find the rpm database. I think it was looking for rpm database in /var. Var was not mounted. So following the direction of vendor who really owned this cluster, we reinstalled red-Hat formatting boot, root, and swap partitions and preserving the data file systems
1) Could we have attempted to mount /var file system?
and
2) And had we been able to do #1, might the re-install / upgrade of these C+++ libshave worked?
Here are the install commands we were given . The only difference between this and the versions on the CD were not the same as these:
rpm -Uvh /mnt/sources/RedHat/RPMS/glibc-2.3.2-5.i386.rpm --nodeps
rpm -Uvh /mnt/sources/RedHat/RPMS/glibc-common-2.3.2-5.i386.rpm --node
I don't have the output errors about rpm not being able to find the rpm database.
I don't know what the "old" existing red hat was but the CDs Were RH 2.1 ES.
SkyyBugg