pstack file analysis

UNIX Solaris
1

Hi ...
Can you please share how to analyse pstack file and various options for core dump analysis

2

I usually use gdb bt or dbx where on core, after using file and strings to figure out what it was running. If optimized, expect inlined calls to disappear. I am more into distrusting inputs, good logging, debug messages, error handling, all-cases action and good structure, with an occasional side of truss/tusc/strace. I stopped stepping when I left hardware in the late 80's.

I used to run core dump hunters on my servers, where each core dump was stack traced and analyzed by script, saved compressed in a dir with 8th day purge, removed and reported by email. Many prod mysteries are core dumps. Some are not fixable, but you get to know their signatures.

3

Hi DGPickett,

can you please send me your core dump analysis scripts that you mentioned(which runs continuously and analyzes core files and send report)

Thanks and Regards
Madhu

4

This is a basic one. Adding gdb or dbx to it requires taking the find output and the $PATH expected or extracted from the core to find the executable. One nice trick with a core file is to run it through 'strings' or 'cat -v' and look for the environment, usually near the end and in ASCII: PATH=.... Using gdb for where (stack trace) solved licensing issues (dbx only ran on dev boxes).

$ cat bin/coremail
#!/usr/bin/ksh
if [ $# = 0 ]
then
 echo "Usage: ${0##*/} <email_addr_list>" >&2
 exit 1
fi
(
 touch ~/.core_mail.mark.new
 if [ -f ~/.core_mail.mark ]
 then
  find / -name core -type f -newer ~/.core_mail.mark
 else
  echo "No ~/.core_mail.mark, all core files included." >&2
  find / -name core -type f
 fi 2>/dev/null | while read l
 do
  date "+%Y-%m-%d %H:%M:%S ($$) Processing: $l"
  (
   echo "Core file found: $l"
   echo
   ls -l "$l"
   echo
   if [ -r "$l" -a -s "$l" ]
   then
    file "$l" | tee /dev/stderr |read zexe
    echo
    zexe="${zexe#*'}"
    zexe="${zexe%'*}"
    if [ "$zexe" != "" ]
    then
      echo ========== strings with $zexe in $l =========
      strings "$l" | fgrep "$zexe"
      echo
    fi
    export zsn=/tmp/core_found_$(date "+%Y-%m-%d_%H:%M:%S").gz
    echo "Saving gzip'd copy to: $zsn"
    echo
    (
     while [ "$(/usr/sbin/fuser "$l" 2>/dev/null)" != "" ]
     do
      sleep 1
     done
     umask 077
     gzip -7 <"$l" >$zsn
     )&
   fi
   ) 2>&1 |tee /dev/stderr|mailx -s core_mail.`date "+%Y-%m-%d_%H:%M:%S"` $@ &
  sleep 2
  date "+%Y-%m-%d %H:%M:%S ($$) Finished $l"
  echo
 done
 mv -f ~/.core_mail.mark.new ~/.core_mail.mark
 )>>/tmp/core_mail_$(date '+%Y%m%d').log 2>&1