Syslog - solaris 9

HI admins,

I am facing an issue with syslog on solaris9.

It stopped the logging of messages..I tried all ways to start the syslog--/etc/init.d/syslog start, /usr/sbin/syslogd......etc....
But ps -eaf | grep sys not showing the syslog processes..........
When i start syslog using /etc/init.d/syslog start, /var/adm/messages reports messages as follows:
krtld: [ID 472681 kern.notice] WARNING: mod_load: cannot load module 'nfs'
krtld: [ID 819705 kern.notice] /kernel/sys/sparcv9/doorfs: undefined symbol

I didn't patched recently, but I just had installed syslog-ng
on the host. For syslog-ng, I also added some dependancies such as
eventlog-
libpcap
pcre-8.10
glib-2.20.
libol-

I started syslog-ng once using /etc/init.d/syslog-ng start and then syslog stopped.
Then I stopped syslog-ng, and tried to restart syslog,but no luck.

I think, bcoz of some dependancies, it won't work....Plz suggest ASAP.

Thanks guys

Have you tried a "truss" to see if you can see what it is failing on ?
Might help

well, actually i didn't have exp with truss commands....

I know that truss -p <pid> will give something but as ps -eaf not showing any sys process.....so no pid...

how to use it for this syslog??

Thanks

you might be having a syntax problem in your syslog.conf file. Can you tail /var/adm/messages and try to start the syslogd again and send the logs here ?

Can you see if any of the relevant files have changed with the syslog-ng install?
Please check:-

/etc/init.d/syslog:
This should have a Copyright "Sun Microsystems" and should start /usr/sbin/syslogd

/usr/sbin/syslogd
On my box it is dated Apr 1 2006, and "/bin/sum" output is
56116 155 /usr/sbin/syslogd
This may be different if you have patches installed.

/etc/syslog.conf
Again should be copyright "Sun Microsystems"

/dev/log
My "ls -l /dev/log" looks like:
lrwxrwxrwx 1 root other 27 May 19 2008 /dev/log -> ../devices/pseudo/log@0:log

/devices/pseudo/log@0:log:
My "ls -l /devices/pseudo/log\@0\:log" looks like:
crw-r----- 1 root sys 21, 5 May 19 2008 /devices/pseudo/log@0:log

Also could you post the contents of your /etc/syslog.conf ?

Thx.

# file /usr/sbin/syslogd
-r-xr-xr-x 1 root sys 79396 Jun 10 2008 /usr/sbin/syslogddynamically linked, stripped

# ls -l /usr/sbin/syslogd
-r-xr-xr-x 1 root sys 79396 Jun 10 2008 /usr/sbin/syslogd

# ls -l /dev/log
lrwxrwxrwx 1 root other 27 Sep 29 2006 /dev/log -> ../devices/pseudo/log@0:log

# ls -l /devices/pseudo/log\@0\:log
crw-r----- 1 root sys 21, 5 Sep 29 2006 /devices/pseudo/log@0:log

# /etc/init.d/syslog start
syslog service starting.
# tail -5 /var/adm/messages
Aug 17 09:06:06 eric krtld: [ID 819705 kern.notice] /kernel/sys/sparcv9/nfs: undefined symbol
Aug 17 09:06:06 eric krtld: [ID 826211 kern.notice] 'crsetugid'
Aug 17 09:06:06 eric krtld: [ID 472681 kern.notice] WARNING: mod_load: cannot load module 'nfs'
Aug 18 08:20:36 eric krtld: [ID 819705 kern.notice] /kernel/sys/sparcv9/doorfs: undefined symbol
Aug 18 08:20:36 eric krtld: [ID 826211 kern.notice] 'copyout_nowatch'

Here eric is my host name

# cat /etc/syslog.conf
#ident "@(#)syslog.conf 1.5 98/12/14 SMI" /* SunOS 5.0 */
#
# Copyright (c) 1991-1998 by Sun Microsystems, Inc.
# All rights reserved.
#
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names
# that match m4 reserved words. Also, within ifdef's, arguments
# containing commas must be quoted.
#
*.err;kern.notice;auth.notice /dev/sysmsg
*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages

*.alert;kern.err;daemon.err operator
*.alert root

*.emerg *

# if a non-loghost machine chooses to have authentication messages
# sent to the loghost machine, un-comment out the following line:
#auth.notice ifdef(`LOGHOST', /var/log/authlog, @loghost)

mail.debug ifdef(`LOGHOST', /var/log/syslog, @loghost)

#
# non-loghost machines will use the following lines to cause "user"
# log messages to be logged locally.
#
ifdef(`LOGHOST', ,
user.err /dev/sysmsg
user.err /var/adm/messages
user.alert `root, operator'
user.emerg *
)

#Save oracle rdbms audit trail to oracle_audit.log
local0.warning /var/log/oracle/oracle_audit.log
#Send oracle rdbms audit trail to remote syslog server
#local0.info @

I think all files are correct as they should be...Is it because of the dependancies that I mentioned in previous posts...???

---------- Post updated at 04:38 PM ---------- Previous update was at 08:23 AM ----------

Hi Citaylor/Niyazi,

Could you please go through my last post.....
It might help to recognize the solution.....

thnx

I think you have a version mismatch between the kernel and kernel modules on your machine. The doorfs kernel module needs 'copyout_nowatch' to be present in the kernel, but it is not. The same is true for the nfs kernel module and 'crsetugid'. Maybe you have more error messages like these in your /var/adm/messages file.

Because syslogd needs to open a door file (var/run/syslog_door) and the kernel is not able to load the appropriate driver, syslogd can't run.

To resolve this issue, you need to identify all files that were overwritten by mistake and replace them with the correct ones (hopefully from a backup).

I guess you ran into trouble after rebooting the machine. The kernel modules might have been replaced a long time ago but it had no immediate effect, because the old ones were still in memory. But after rebooting ...

what files do I need to check....i mean from any specific folder/directory....

let me know..

thnx

That's hard to tell from the distance. At least you have to check:

/kernel/sys/sparcv9/doorfs
/kernel/sys/sparcv9/nfs
/kernel/fs/sparcv9/nfs

Please check /var/adm/messages for other errors from the kernel runtime loader.

But those are executable files....how to check those....

What if I will replace those files directly using cd or the patch that contains those files.....???

For example, your doofs kernel module requires the kernel function copyout_nowatch.

$ /usr/ccs/bin/nm /kernel/sys/sparcv9/doorfs | grep copyout_nowatch
[175]   |             0|           0|FUNC |GLOB |0    |UNDEF  |copyout_nowatch

In my case (a Solaris 10 installation), the size and date of the module is

$ ls -l /kernel/sys/sparcv9/doorfs
-rwxr-xr-x   1 root     sys        61592 Oct  3  2008 /kernel/sys/sparcv9/doorfs

Now, you have to check your backups for an older version of this file. Restore it to a temporary location and check the symbol table with nm and verify, that copyout_nowatch is not an undefined symbol.

I have the following outputs...

# ls -l /kernel/sys/sparcv9/doorfs
-rwxr-xr-x 1 root sys 49496 Feb 12 2010 /kernel/sys/sparcv9/doorfs

# /usr/ccs/bin/nm /kernel/sys/sparcv9/doorfs | grep copyout_nowatch
[74] | 0| 0|FUNC |GLOB |0 |UNDEF |copyout_nowatch

I chacked these on the server on which syslog is not working and also on the server on which syslog is working...

both have the same date and both showing the same "UNDEF" o/p as above......I have solaris -9

What to do???

Hmm, was the server where syslog is not working rebooted and the server with the working syslog not rebooted?

Besides that, I am out of ideas at the moment. It seems, that somehow kernel modules that do not match the kernel have found their way into the system. With regular patching such a mismatch should not have occured. But that's only a diagnosis from the far distance.

nope.........after installation of syslog-ng with dependancies, it wasn't rebooted yet....as its production server....

And the server with syslog working was rebooted 3 month ago (production server too) ...not recently....

I m just hanged....