We have 2 solaris 10 t5240 servers with static IP addresses on nxge0 I/F which were rebooted a few days back with a known good config that has been in place for years (for /etc/hosts, /etc/hostname.nxge0, /etc/netmasks, etc)
They are not using dhcp.
About the same time today, both of their IPs changed to that of a 3rd server and the I/F's went down as seen in ifconfig -a. We had to login from the console and change them back and up the I/F.
I didn't see in bash_history that someone had typed an ifconfig nxge0 statement. I didn't see anything much in /var/adm/messages.
Had the config files for the network configuration changed? I regret that I can't remember where it is defined on Solaris (the last Solaris server I had was 2.6 :o)
A few thoughts:-
Have you examined the output from dmesg which is written before the syslog daemon starts?
Did they DHCP somehow?
Do they boot from local disk or is it SAN provided?
Has anyone run sysunconfigure, which would destroy the identity of the server (hostname, network config, host lookup, etc.)
I'm not sure how helpful these will be, but it might bring something to light.
Have you examined the output from dmesg which is written before the syslog daemon starts?
dmesg > somefile looks like /var/adm/messages did: normal operations then a bunch of 390400 kern messages which could have been the results of the network change when the localhost IP changed. The server was booted days ago and used the proper IP. It was in-flight days later that it did this -- on two independent servers at the same time. Each had been rebooted a day from the other after DRPC patching.
Did they DHCP somehow? No log of such and daemon is not running and there are no dhcp config files
Do they boot from local disk or is it SAN provided?
Local disk
Has anyone run sysunconfigure, which would destroy the identity of the server (hostname, network config, host lookup, etc.)
No. All the files /etc/hosts /etc/hostname.nxge0 /etc/netmasks etc untouched with old timestamps. Also bash_history doesn't show commands were entered.
but that means someone has to mess with the network config files and have the rights to restart the service to have the changes take effect. They'd need root rights for that, me thinks. A bit of a mystery!
I'd check what's in root's crontabs, and also the modification time of the crontab files in /var/spool/crontab.
And then I'd enable auditing.
The mystery here isn't "what", the mystery is "who". The real problem is failing to own up to a mistake - we all make them.
It's the hiding of the mistake that's an indication of a problem admin, because now you start wondering what other mistakes are being hidden because you now know you can't trust that admin to admit mistakes.
Checked the shell history file of all that logged in that day and found: ifconfig -a xx.xx.xx.xx which was the wrong IP. The -a option, when used with no other parms, lists all configs, else edits all, which is what resulted.