I need help figuring out why I lose /usr/bin directory sometimes it goes down 3-4 times a day and I have to link them back using ln - /usr/bin bin I cannot figure out why this is happenning
Why are you linking to the /usr/bin directory ... and where??
We are not linking to it or should I say our backoffice system does. I had IBM look at it and our software vendor. They only tell me that someone or program according to the logs is deleting the links. No one as of yet can tell me why or how to stop it...
What do you mean "linking" to it???
I should correct myself I don't if the backoffice system is linking to it or not. I just know that something is deleting the /usr/bin and don't know why
check the filemon utility, or write a script that checks if the directory is still there in periodic intervals, and write to a logfile with timestamp
then you have the specific time
I would check first:
-
automated process on the machine, per cron perhaps
check every users crontab (crontab -l) that has the rights to delete the folder, and check for user created scripts running -
automated process outside the machine
triggered per ssh, telnet, rsh...
you can compare the time from your scripts logfile I wrote above with the login logs on your server an find out the IP (last, syslog) -
manually triggered by a user, but seems not very likely
like step 2, analyse login logs for that
very scary 
I still don't get how something is deleting /usr/bin ! 1. Hoes does it "reappear"? 2. When it's deleted, can you still run commands?
We don't either!
1) We go to the console and run the following command: ln -s /usr/bin bin to get it back
and we also have to relink these /usr/lib /u and /usr/lib/boot/unix_64
2) Yes but only on the console you cannot connect from a remote client
Then you are linking TO /usr/bin. So /usr/bin is always there, right? What directory are you IN (cwd/pwd) when you run the ln command??
you mean you get /usr/bin back, when running ln -s /usr/bin bin?
I don't think so
you link /usr/bin to your local directory
it's getting a bit weird
We don't either!!
1) We relink them at the console it only work on the console since it has all users lock out.
2) Yes but only on the console
What you are saying makes absolutely no sense. Please answer our questions:
- What is the current directory when you run this "ln" command
- What exactly is the ln command?
- What is the purpose of this ln command?
Reply to Funksen
That is the command we run every time the link goes down, we have our accounting software running on the system as well not sure that is why we run that command but it was the one our software vendor recommended to use.
1) The root directory
2) ln -s /usr/bin bin
3) No one can connect to the software package or even telnet
Then it is the "/bin" directory that keeps disappearing?
Yes that is the one...
Please copy/paste (within code tags or as a TXT attachment) the following commands:
ps -ef
find /var/spool/cron/crontabs -ls
The first is to get the entire process list. The second shows all files that are used by cron.
AIX 5.3 (and perhaps earlier) has an audit feature, which allows you to record the process and/or command that changes, for instance, a particular file. You can read about that at Setting up auditing
Ok, this makes more sense, because /bin is a link to /usr/bin on AIX systems.
EDIT: Setting up auditing is a good idea, otheus.
bakunin
Thanks for your response but I have ran the auditing process with IBM support and send them the logs. The only thing they told me is that something is going out and deleting the links. They thought it was a job being launch by the user root. However they can't tell what the job is and how it is deleting the links. I have cahnged the root password three different times per their recommendation still having the issue.
First, the output of ps and find please. Then we'll get to auditing if it's needed