Hello,
The scenario is as follows, I have a background process running initially for which i know the PID on machine1. I use ssh from machine 2 to execute a script in machine 1. For some reason the back ground process is terminated. I would like to know which process caused the termination of this background task. is there any ideas to monitor this ?
Thanks in advance
Well, it might be a bit of a job. Signals like kill -9 do not involve a party that logs, the sender sends the signal the the recipient dies.
But, there is always a way. You can replace the entire libc or whetver lib kill(2) and signal() or whatever send the signals lives, with a library you augment with the desired logging if a signal is sent. All calls in the new libc have to call syscall() to get to the original calls in the original, renamed library. I read about this in a product that created restart points with no cooperation of the applications! Heady stuff!
If you are using Solaris 10, then this will show who killed your process and how:
dtrace -n 'proc:::signal-send/args[1]->pr_pid==your_proc_pid/{printf ("user with UID %d sent signal %d using %s",uid,args[2],execname);}'
After the fact, from a log? I guess, before the pid recycles, or all such messages for that pid?
If you are asking about that DTrace one-liner - it should be run when the process is alive, and it should be kept running (preferably in background, redirecting the output to some file). When the process in question receives any signal, then DTrace will detect that call and print the information about who sent it.
Ok, it is somewhat like using a live process with truss/tusc/strace or a debugger. Is it low overhead? Truss is definitely not!
It may seem like running debugger on an application, but it most certainly is not. Debuggers intercept all the application's instructions introducing huge delays. DTrace on the other hand is built deeply into Solaris kernel, firing off its code only when instructed to trace some particular event, which is done by DTrace probes. By default all the probes are disabled. When DTrace code like above is executed, Solaris kernel substitutes small piece of its own assembly code with instructions informing DTrace framework about particular event. This code is only executed when traced event happens. This behavior results in literally no overhead when tracing rare events like sending signals to applications and very small overhead when tracing larger number of very often occurring events.