Our AIX system is currently having an issue where calls to various executables are returning errors such as
ksh: /usr/bin/oslevel: not found.
This is even happening with system files, third party scripts, and with the full path included.
Telnet connections are also closing immediately upon hitting the server with the following error, despite telnetd seemingly running fine.
Connection closed by host
Lots of other executables are running fine however and seem to be entirely unaffected. The hardware isn't showing any issue and the disks aren't full.
This started yesterday, when there wasn't any users on the system. The only work that was being done on the system was running cp on a UniData database over night, which started the night before the error occurred.
Is the binary /usr/bin/oslevel still there? You know what the copy command was, maybe something went wrong, wrong destination or something?
Any entries in errpt?
Did you check if the PATH is still correct?
The scripts we were calling were all there. I would cd to their directory and check they were there with an ls before calling them.
The cp command was completely successful. The thousands of files were all copied to the other directory without issue. We had a nightly backup fail that same night so that might be more related to the actual problem than the copy command, unless there was some sort of struggle for control between them. However they've run simultaneously in the past without issue.
Nothing showed in errpt. I didn't check PATH but someone else might have.
To resolve the problem, our networking team did a reboot of the server. From what I understand, just to do that they had to do some sort of adjustment to /etc/rc.shutdown. After the reboot, everything was back to normal.
I was concerned that something had gone horribly wrong, such as issuing a find and delete somewhere that had a symbolic link across to /usr/bin. Don't laugh, we've done a similar thing here - though fortunately not deleting OS files. :o