This application is eating up the CPU

pandu345 · August 7, 2006, 9:51pm

Hi,
I am not very much fmiliar with Solaris OS. My main concern for posting is One application is eating 50% of CPU and I cannot run that application, If I perform any action in that application it takes real long time.

I have solaris installed on my development machine.I have my application IDM installed on this solaris box.It is a commercial application from SUN , is set up on weblogic server.The application was running fine but recently it has started eating CPU rapidly, every time I see on the machine , the JVM corresponding to this application is consuming 50%.
I cannot run the application any more.If I perform any small action in the application , it takes real long time and fails.

When I do a prstat on the solaris box it shows this thing

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
9768 javasrvr 728M 590M cpu1 0 10 659:58.28 38% java/81

This is my OS

SunOS 5.8 Generic_117350-23 sun4u sparc SUNW,Sun-Fire-280RSystem = SunOS
Node = pandu
Release = 5.8
KernelID = Generic_117350-23

I have java 1.3 installed on this machine.
can somebody get me out of this trouble

System_Shock · August 7, 2006, 10:55pm

Java apps are notorious for using up CPU. The specs you mention don't show what CPU you have or how many.
What's IDM? There's no IDM software mentioned in Sun's site. Did you mean to say it is a commercial application FOR Sun? If that's the case, may want to check the CPU requirements for it, or it may be a badly written app.

pandu345 · August 8, 2006, 1:30am

Actually the machine is my development machine at my work place, How can I get Those information. I am sure it should be a very good configuration , because my company has good machines in all the places. And IDM is a commercial aplication from sun used for creatinng user accounts and managing them in corporates.please let me know if you have any other questions.

thanks,
pandu

izy100 · August 10, 2006, 10:36am

1st you have to verify your claim that it is eating a lot of CPU.

Can you do these command and paste here?

vmstat 3 10
iostat -xntcz 3 10
/usr/ucb/ps aux | sort -nrk 3
/usr/ucb/ps aux | sort -nrk 4

pandu345 · August 10, 2006, 1:03pm

Hi,
Thanks for your response,I have the responses for all the commands.

Machine-name> vmstat 3 10
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s6 sd sd --   in   sy   cs us sy id
 0 0 0 5065080 4974464 1  5  6  0  0  0  0  0  1  1  0  208  561  636 42  1 57
 0 0 0 3373920 3817920 0 14  0  0  0  0  0  0  0  0  0 5180 1682 1871 51  1 48
 0 0 0 3373920 3817920 0 13  0  0  0  0  0  0  0  0  0 6675 2023 2090 54  1 45
 0 0 0 3373920 3817920 0 12  0  0  0  0  0  0  1  1  0 5423 1768 1975 53  2 45
 0 0 0 3373920 3817920 0  8  0  0  0  0  0  0  0  0  0 3962 1374 1664 52  0 48
 0 0 0 3373920 3817920 0 10  0  0  0  0  0  0  1  1  0 5208 1697 1852 51  2 47
 0 0 0 3373920 3817920 0 12  0  0  0  0  0  0  1  1  0 5449 1752 1970 52  1 47
 0 0 0 3373920 3817928 0  9  0  0  0  0  0  0  0  0  0 4787 1582 1807 52  1 46
 0 0 0 3373920 3817928 0 12  0  0  0  0  0  0  0  0  0 5601 1824 1930 52  2 46
 0 0 0 3374000 3818008 0 10  0  0  0  0  0  0  0  0  0 5243 1709 1951 51  2 47

_________________________________________________________________

Machine-name> iostat -xntcz 3 10
   tty         cpu
 tin tout  us sy wt id
   0    2  42  1  0 56
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t6d0
    0.2    0.6    4.3    4.9  0.0  0.0    2.3   11.4   0   0 c1t0d0
    0.2    0.6    4.1    4.9  0.0  0.0    2.5   10.7   0   0 c1t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0   11.4   0   0 toaster:/vol/vol1/tools/solaris8-sparc
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    9.0   0   0 toaster:/vol/vol1/weblogicDEV
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.4   0   0 nas-rwc-filer6-nfs:/vol/Sales/Spocrde/spocrde_dev
   tty         cpu
 tin tout  us sy wt id
   0  236  52  1  0 47
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   tty         cpu
 tin tout  us sy wt id
   0   66  50  3  0 46
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.3    0.0    9.7  0.0  0.0    0.0    7.0   0   1 c1t0d0
    0.0    1.3    0.0    9.7  0.0  0.0    0.0    6.4   0   1 c1t1d0
   tty         cpu
 tin tout  us sy wt id
   0  112  55  2  0 43
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   tty         cpu
 tin tout  us sy wt id
   0   66  53  2  0 44
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.3    0.0    0.5  0.0  0.0    0.0   12.2   0   0 c1t0d0
    0.0    0.3    0.0    0.5  0.0  0.0    0.0    9.1   0   0 c1t1d0
   tty         cpu
 tin tout  us sy wt id
   0  112  52  4  0 44
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.3    0.0    2.7  0.0  0.0    0.0    9.4   0   0 c1t0d0
    0.0    0.3    0.0    2.7  0.0  0.0    0.0    9.6   0   0 c1t1d0
   tty         cpu
 tin tout  us sy wt id
   0  112  54  3  0 43
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   tty         cpu
 tin tout  us sy wt id
   0   66  54  3  0 42
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.7    0.0    3.3  0.0  0.0    0.0    7.2   0   0 c1t0d0
    0.0    0.7    0.0    3.3  0.0  0.0    0.0    6.9   0   0 c1t1d0
   tty         cpu
 tin tout  us sy wt id
   0  112  54  4  0 42
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   tty         cpu
 tin tout  us sy wt id
   0   66  54  2  0 44
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.7    0.0    3.0  0.0  0.0    0.0   12.3   0   1 c1t0d0
    0.0    0.7    0.0    3.0  0.0  0.0    0.0    9.7   0   1 c1t1d0
Machine-name>

_________________________________________________________________


Machine-name> /usr/ucb/ps aux | sort -nrk 3
javasrvr  9768 49.4  7.5744968603512 ?        O   Jun 27 51042:50 /opt/java142_08/bi
javasrvr  5734  0.4  4.5729656356536 ?        S 13:01:37 16:13 /opt/weblogic/8.1S
javasrvr  3364  0.3  6.7732368534752 ?        S   Jun 22 132:29 /opt/weblogic/8.1S
gwpuser   3135  0.3  4.2466800339072 pts/1    R   Aug 07 370:37 /opt/weblogic/8.1S
javasrvr  6195  0.2  2.7442040218552 ?        S 16:34:49  4:04 /opt/weblogic/8.1S
root      7142  0.1  0.1 1856 1392 pts/2    O 09:49:40  0:00 /usr/ucb/ps aux
javasrvr  1605  0.1  7.41440624597784 ?        S   Jul 14 56:42 /opt/weblogic/8.1S
sridharm  7093  0.0  0.1 2248 1760 pts/2    R 09:41:50  0:00 -csh
sridharm  7091  0.0  0.1 6984 2104 ?        S 09:41:50  0:00 /usr/local/sbin/ss
root     18943  0.0  0.0 1680 1152 ?        S   Jul 27  0:00 in.tnamed
root      8004  0.0  0.1 1800 1272 ?        S   Jul 19  0:00 rpc.rstatd
root      7089  0.0  0.1 7000 3648 ?        S 09:41:44  0:00 /usr/local/sbin/ss
root      2957  0.0  0.1 7000 3576 ?        S   Aug 07  0:00 /usr/local/sbin/ss
root       608  0.0  0.1 1888 1280 console  S   May 25  0:00 /usr/lib/saf/ttymo
root       605  0.0  0.1 3760 2464 ?        S   May 25  8:55 /usr/local/samba/b
root       603  0.0  0.1 4368 2304 ?        S   May 25  0:00 /usr/local/samba/b
root       535  0.0  0.1 4008 1824 ?        S   May 25  0:00 /usr/local/sbin/ss
root       521  0.0  0.0    0    0          Z           0:00  <defunct>
root       520  0.0  0.412663230056 ?        S   May 25  0:31 /opt/centerrun/N1_
root       499  0.0  0.0 1784 1216 ?        S   May 25  0:00 vxnotify -f -w 15
root       498  0.0  0.0  720  432 ?        S   May 25  0:00 /sbin/sh - /usr/li
root       472  0.0  0.0  720  464 ?        S   May 25  0:00 /sbin/sh - /usr/li
root       467  0.0  0.1 4464 3264 ?        S   May 25  0:12 /usr/sbin/nsrexecd
root       466  0.0  0.1 3752 1952 ?        S   May 25  0:03 /usr/sbin/nsrexecd
root       462  0.0  0.1 4720 1888 ?        S   May 25  0:02 /usr/lib/sendmail
root       457  0.0  0.0 1080  776 ?        S   May 25  0:00 /usr/lib/utmpd
root       448  0.0  0.1 2264 1280 ?        S   May 25  0:02 /usr/lib/inet/xntp
root       445  0.0  0.1 8496 3456 ?        S   May 25  0:00 mntrpc_server  -S1
root       432  0.0  0.1 1712 1376 ?        S   May 25  0:01 /opt/rational/clea
root       430  0.0  0.1 4192 2392 ?        S   May 25  0:03 /opt/rational/clea
root       407  0.0  0.1 2024 1360 ?        S   May 25  0:04 /usr/sbin/cron
root       394  0.0  0.1 4856 2888 ?        S   May 25  0:32 /usr/sbin/syslogd
root       382  0.0  0.1 4400 2904 ?        S   May 25  0:01 /usr/lib/autofs/au
root       368  0.0  0.1 2320 1776 ?        S   May 25  0:02 /usr/sbin/inetd -s
root       366  0.0  0.1 1952 1424 ?        S   May 25  0:00 /usr/lib/nfs/lockd
root       340  0.0  0.1 2432 1528 ?        S   May 25  0:00 /usr/sbin/rpcbind
root       289  0.0  0.44931232960 ?        S   May 25  1:06 /opt/VRTSob/bin/vx
root       212  0.0  0.1 3752 3024 ?        S   May 25  2:07 /usr/lib/picl/picl
root        84  0.0  0.1 2640 1448 ?        S   May 25  0:00 devfsadmd
root        72  0.0  0.0 2288 1104 ?        S   May 25  0:00 /usr/lib/sysevent/
root        14  0.0  0.32758425152 ?        S   May 25  0:05 vxconfigd -m boot
root         3  0.0  0.0    0    0 ?        S   May 25 10:25 fsflush
root         2  0.0  0.0    0    0 ?        S   May 25  0:00 pageout
root         1  0.0  0.0  872  352 ?        S   May 25  0:05 /etc/init -
root         0  0.0  0.0    0    0 ?        T   May 25  0:01 sched
mmcatam   2961  0.0  0.1 2248 1688 pts/1    S   Aug 07  0:00 -csh
mmcatam   2959  0.0  0.1 6976 2184 ?        S   Aug 07  0:00 /usr/local/sbin/ss
javasrvr  5905  0.0  2.2278376171720 ?        S 13:24:56  1:27 /opt/weblogic/8.1S
gwpuser   2983  0.0  0.1 3776 3008 pts/1    S   Aug 07  0:00 tcsh
daemon     367  0.0  0.1 3136 2400 ?        S   May 25  0:00 /usr/lib/nfs/statd
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
Machine-name>


_________________________________________________________________


Machine-name> /usr/ucb/ps aux | sort -nrk 4
javasrvr  9768 48.3  7.5744968603512 ?        O   Jun 27 51044:44 /opt/java142_08/bi
javasrvr  1605  0.1  7.41440624597784 ?        S   Jul 14 56:42 /opt/weblogic/8.1S
javasrvr  3364  0.1  6.7732368534752 ?        S   Jun 22 132:29 /opt/weblogic/8.1S
javasrvr  5734  0.5  4.5729656356536 ?        S 13:01:37 16:14 /opt/weblogic/8.1S
gwpuser   3135  0.3  4.2466800339072 pts/1    R   Aug 07 370:45 /opt/weblogic/8.1S
javasrvr  6195  0.2  2.7442040218552 ?        S 16:34:49  4:05 /opt/weblogic/8.1S
javasrvr  5905  0.0  2.2278376171720 ?        S 13:24:56  1:27 /opt/weblogic/8.1S
root       289  0.0  0.44931232960 ?        S   May 25  1:06 /opt/VRTSob/bin/vx
root       520  0.0  0.412663230056 ?        S   May 25  0:31 /opt/centerrun/N1_
root        14  0.0  0.32758425152 ?        S   May 25  0:05 vxconfigd -m boot
sridharm  7093  0.0  0.1 2248 1760 pts/2    S 09:41:50  0:00 -csh
sridharm  7091  0.1  0.1 6984 2112 ?        S 09:41:50  0:00 /usr/local/sbin/ss
root      8004  0.0  0.1 1800 1272 ?        S   Jul 19  0:00 rpc.rstatd
root      7144  0.1  0.1 1856 1392 pts/2    O 09:51:36  0:00 /usr/ucb/ps aux
root      7089  0.0  0.1 7000 3648 ?        S 09:41:44  0:00 /usr/local/sbin/ss
root      2957  0.0  0.1 7000 3576 ?        S   Aug 07  0:00 /usr/local/sbin/ss
root       608  0.0  0.1 1888 1280 console  S   May 25  0:00 /usr/lib/saf/ttymo
root       605  0.0  0.1 3760 2464 ?        S   May 25  8:55 /usr/local/samba/b
root       603  0.0  0.1 4368 2304 ?        S   May 25  0:00 /usr/local/samba/b
root       535  0.0  0.1 4008 1824 ?        S   May 25  0:00 /usr/local/sbin/ss
root       467  0.0  0.1 4464 3264 ?        S   May 25  0:12 /usr/sbin/nsrexecd
root       466  0.0  0.1 3752 1952 ?        S   May 25  0:03 /usr/sbin/nsrexecd
root       462  0.0  0.1 4720 1888 ?        S   May 25  0:02 /usr/lib/sendmail
root       448  0.0  0.1 2264 1280 ?        S   May 25  0:02 /usr/lib/inet/xntp
root       445  0.0  0.1 8496 3456 ?        S   May 25  0:00 mntrpc_server  -S1
root       432  0.0  0.1 1712 1376 ?        S   May 25  0:01 /opt/rational/clea
root       430  0.0  0.1 4192 2392 ?        S   May 25  0:03 /opt/rational/clea
root       407  0.0  0.1 2024 1360 ?        S   May 25  0:04 /usr/sbin/cron
root       394  0.0  0.1 4856 2888 ?        S   May 25  0:32 /usr/sbin/syslogd
root       382  0.0  0.1 4400 2904 ?        S   May 25  0:01 /usr/lib/autofs/au
root       368  0.0  0.1 2320 1776 ?        S   May 25  0:02 /usr/sbin/inetd -s
root       366  0.0  0.1 1952 1424 ?        S   May 25  0:00 /usr/lib/nfs/lockd
root       340  0.0  0.1 2432 1528 ?        S   May 25  0:00 /usr/sbin/rpcbind
root       212  0.0  0.1 3752 3024 ?        S   May 25  2:07 /usr/lib/picl/picl
root        84  0.0  0.1 2640 1448 ?        S   May 25  0:00 devfsadmd
mmcatam   2961  0.0  0.1 2248 1688 pts/1    S   Aug 07  0:00 -csh
mmcatam   2959  0.0  0.1 6976 2184 ?        S   Aug 07  0:00 /usr/local/sbin/ss
gwpuser   2983  0.0  0.1 3776 3008 pts/1    S   Aug 07  0:00 tcsh
daemon     367  0.0  0.1 3136 2400 ?        S   May 25  0:00 /usr/lib/nfs/statd
sridharm  7145  0.1  0.0 6760  904 pts/2    S 09:51:36  0:00 sort -nrk 4
root     18943  0.0  0.0 1680 1152 ?        S   Jul 27  0:00 in.tnamed
root       521  0.0  0.0    0    0          Z           0:00  <defunct>
root       499  0.0  0.0 1784 1216 ?        S   May 25  0:00 vxnotify -f -w 15
root       498  0.0  0.0  720  432 ?        S   May 25  0:00 /sbin/sh - /usr/li
root       472  0.0  0.0  720  464 ?        S   May 25  0:00 /sbin/sh - /usr/li
root       457  0.0  0.0 1080  776 ?        S   May 25  0:00 /usr/lib/utmpd
root        72  0.0  0.0 2288 1104 ?        S   May 25  0:00 /usr/lib/sysevent/
root         3  0.0  0.0    0    0 ?        S   May 25 10:25 fsflush
root         2  0.0  0.0    0    0 ?        S   May 25  0:00 pageout
root         1  0.0  0.0  872  352 ?        S   May 25  0:05 /etc/init -
root         0  0.0  0.0    0    0 ?        T   May 25  0:01 sched
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
Machine-name>

Can you briefly tell me what each command does.
And why I am telling this application is eatig CPU is when I run the prstat command .

 PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  9768 javasrvr  728M  589M cpu0     0   10 799:40.35  49% java/75
  5734 javasrvr  713M  348M sleep   21    0   0:00.40 0.4% java/65
  3135 gwpuser   456M  331M sleep   22    0   0:00.52 0.3% java/65
  6195 javasrvr  432M  213M sleep   20    0   0:00.56 0.1% java/53
  7150 sridharm 1872K 1496K cpu1    58    0   0:00.00 0.0% prstat/1
  1605 javasrvr 1407M  584M sleep   28   10   0:01.42 0.0% java/63
  3364 javasrvr  715M  522M sleep   28   10   0:01.40 0.0% java/71
   448 root     2264K 1280K sleep  100    -   0:00.02 0.0% xntpd/1
   457 root     1080K  776K sleep   59    0   0:00.00 0.0% utmpd/1
   368 root     2320K 1776K sleep   58    0   0:00.02 0.0% inetd/1
   407 root     2024K 1360K sleep   48    0   0:00.04 0.0% cron/1
   382 root     4400K 2904K sleep   59    0   0:00.00 0.0% automountd/5
   367 daemon   3136K 2400K sleep   59    0   0:00.00 0.0% statd/4
   499 root     1784K 1216K sleep   50    0   0:00.00 0.0% vxnotify/1
   289 root       48M   32M sleep   59    0   0:00.00 0.0% vxsvc/13
   366 root     1952K 1424K sleep   58    0   0:00.00 0.0% lockd/1
   340 root     2432K 1528K sleep   58    0   0:00.00 0.0% rpcbind/1
   467 root     4464K 3264K sleep   59    0   0:00.12 0.0% nsrexecd/3
    84 root     2640K 1448K sleep   51    0   0:00.00 0.0% devfsadm/7
   212 root     3752K 3024K sleep   23    0   0:00.00 0.0% picld/10
   445 root     8496K 3456K sleep   58    0   0:00.00 0.0% mntrpc_server/1
Total: 47 processes, 597 lwps, load averages: 1.13, 1.03, 0.99

This line
9768 javasrvr 728M 589M cpu0 0 10 799:40.35 49% java/75,
corresponds to the Sun Identity Manager (IdM) application.

The way I decided that this is the JVM of this application is, When ever I need to restart IdM , I need to restart weblogic server on which IdM is installed, When I run that script to kill the weblogic server, it says

Machine-name> audit kill-identity_m1
Machine-name> KILLING PID 9768 SUCCESSFUL

Thanks for your response.Please let me know if you have other questions.
Pandu

izy100 · August 11, 2006, 7:34am

Looks weird that it is eating 50% of CPU time consistently.

Can you do a mpstat 3 10?
I think the application is not utilizing ALL the CPUS efficiently.

pandu345 · August 11, 2006, 12:47pm

Hi,

Thanks for your response,If you dont mind , can you briefly go over what you found from the previous results and how you came to know that application is not utlilising CPU efficiently

Machine-Name> clear
Machine-Name> mpstat 3 10
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    3   0   86   551  422  533   83  100   24    1   485   51   1   0  47
  1    2   0   36   501    1  124   78  100   21    1   100   33   1   0  66
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    7   0  117   731  596  687  114  142   33    0   560   68   2   0  30
  1    6   0   48  4786    1 1280   97  145   28    0  1249   38   2   0  60
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    7   0   91   724  587  688  122  135   23    0   513   73   2   0  25
  1    2   0   42  4655    1 1289   71  133   33    0  1248   30   1   0  69
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    7   0   93   766  634  641  126  122   29    0   431   83   3   0  13
  1    3   0   44  5253    1 1466   73  124   16    0  1470   22   1   1  77
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    9   0  132   837  696  590  133  133   43    0   426   82   2   0  16
  1    5   0   50  6046    1 1614  107  136   48    0  1742   26   1   0  73
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    7   0   85   678  548  592  122  141   34    0   404   81   2   0  17
  1    1   0   36  4177    1 1303   70  132   29    0  1228   24   1   0  75
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    7   0   87   718  588  688  123  123   22    0   533   82   3   0  16
  1    3   0   38  4661    1 1323   72  128   28    0  1271   24   0   0  76
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   11   0  115   876  736  730  134  131   39    0   608   78   2   0  20
  1    4   0   51  6365    1 1533   93  131   43    0  1599   29   3   0  68
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    7   0   93   761  625  671  133  135   39    0   473   81   1   0  18
  1    3   0   42  5139    1 1397   76  131   24    0  1417   27   2   0  72
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    9   0   89   779  649  742  121  125   22    0   591   79   2   0  20
  1    4   0   37  5449    1 1405   68  128   18    0  1428   26   1   0  73
Machine-Name>

Also can I know some thing, I have like 10 applications on weblogic other than idm, they are small inhouse apps.Can we find out which is the culprit.

Thanks,
pandu

izy100 · August 11, 2006, 2:37pm

To understand all those commands, go and read Adrian Cockroft book, SUN Performance and Tuning.

I can tell you these facts:
1) You have enough CPUs resources. No processes are waiting for your CPU time slice.

2) You have enough memory as indicated by your scan rate (sr)

3) You are not IO Bound.

4) Your application is utilizing both CPUs.

5) javasrvr consuming 50% of the CPU util is NOT an issue at all.

Bottomline: Your application is not stressing the server at all.

It is likely that the application is written in a serial (single threaded) manner and unable to utilize ALL the avail server resources.

You said, "If I perform any action in that application it takes real long time."
Have you try to run any other command/application in the server and compare it with another server of the same specs that does not have this javasrvr application?

My guess is the perf of both server should be about the same.

pandu345 · August 11, 2006, 5:30pm

Hi,

I have the following questions:
If my application is not stressing the CPU , why it consumes 50 % in prstat.
In all the other servers where I have same application installed it consumes only 5 -10% of CPU.
Even in this bad server initially when i start it uses 5-10% of CPU when I start doing some action in the IdM application it rises to 50% and is slower.

The other inhouse applications deployed on the same server work well even if the CPU consumption is 50 %.

If javasrvr consume 5-10% , the application works fine as it is in other servers, but when it rises to 50 % the application stops working in this bad server.

What are culprits, Is it the JVM or the IdM application.
If we know about the lwps or light weight processes used by this application IDm will it give us any other information.

Thanks for responding constantly.

pandu

reborg · August 11, 2006, 7:53pm

if you do:

kill -QUIT <pid of java process>

You will get a thread dump for the process.

izy100 · August 11, 2006, 10:19pm

See my reply inline:

If my application is not stressing the CPU , why it consumes 50 % in prstat.
>> Even if it is 100% CPU util, it doesn't mean that you don't have enough CPU resources. You run out of cpu ONLY when there is a queue of runnable processess waiting for CPU time slice. In short, 100% CPU util means the CPU is working hard for your processes (doesn't mean there is not enough).

In all the other servers where I have same application installed it consumes only 5 -10% of CPU. Even in this bad server initially when i start it uses 5-10% of CPU when I start doing some action in the IdM application it rises to 50% and is slower.
>> See my reply above. 50% does not mean much.

The other inhouse applications deployed on the same server work well even if the CPU consumption is 50 %.
>> Exactly. If other application on the same server has no performance problem running, this means you have ample CPU resources on the SAME server.

If javasrvr consume 5-10% , the application works fine as it is in other servers, but when it rises to 50 % the application stops working in this bad server.
>> My guess is there is some performance bug with a particular function that is triggering the workload. You have to talk to the developer of the application and get them to draw some workflow diagram to understand where is the problem.

pandu345 · August 12, 2006, 11:43pm

Hi,

I have got the thread dump of the JVM thats consuming 50% of CPU, will it help us in any way.If you get a chance please go through it.If you need any further information, please let me know.

<Aug 12, 2006 12:04:06 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):

"LDAPConnThread-95 ldap://ldap-it.gene.com:389" daemon prio=5 tid=0x0078c9f8 nid=0x68cbb runnable [ca601000..ca6019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe11737a0> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-94 ldap://ldap-it.gene.com:389" daemon prio=5 tid=0x00ac1f20 nid=0x68cb6 runnable [cbe01000..cbe019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe1171898> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-93 ldap://ldap-it.gene.com:389" daemon prio=5 tid=0x00c1d198 nid=0x68caa runnable [cae01000..cae019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe115aff8> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-87 ldap://ldap-it.gene.com:389" daemon prio=5 tid=0x00b67d08 nid=0x68c70 runnable [cad01000..cad019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe10e93f8> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-81 ldap://ldap-it.gene.com:389" daemon prio=5 tid=0x00ef2ea8 nid=0x68c49 runnable [ca901000..ca9019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe10e8018> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-75 ldap://ldap-it.gene.com:389" daemon prio=5 tid=0x00ec3da8 nid=0x68bfe runnable [caa01000..caa019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe10c0498> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"Source Adapter Task" daemon prio=5 tid=0x003d69b8 nid=0x6859a waiting on condition [cf301000..cf3019c0]
	at java.lang.Thread.sleep(Native Method)
	at com.waveset.adapter.SourceAdapterTask.doRealWork(SourceAdapterTask.java:86)
	at com.waveset.task.Executor.execute(Executor.java:157)
	at com.waveset.task.TaskThread.run(TaskThread.java:116)

"ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x00bc18a8 nid=0x68597 in Object.wait() [cab01000..cab019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0fda900> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x01aa70d0 nid=0x68596 in Object.wait() [ce001000..ce0019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0fda980> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x01708b38 nid=0x68595 in Object.wait() [cf201000..cf2019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0fdaa00> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"LdapContextFactory.Reaper" daemon prio=5 tid=0x00ae50a8 nid=0x3b10 waiting on condition [cf101000..cf1019c0]
	at java.lang.Thread.sleep(Native Method)
	at com.waveset.util.jndi.LdapContextFactory$ReaperThread.run(LdapContextFactory.java:358)

"Reconciler Work Thread" daemon prio=5 tid=0x00cdab38 nid=0x4f in Object.wait() [cb001000..cb0019c0]
	at java.lang.Object.wait(Native Method)
	at com.waveset.util.BlockingQueue.waitToDequeue(BlockingQueue.java:39)
	- locked <0xe0a3dd38> (a com.waveset.util.BlockingQueue)
	at com.waveset.recon.ReconTask.getNextWorkItem(ReconTask.java:3816)
	at com.waveset.recon.ReconTask.access$1300(ReconTask.java:105)
	at com.waveset.recon.ReconTask$WorkerThread.run(ReconTask.java:2935)

"Reconciler Work Thread" daemon prio=5 tid=0x00248d18 nid=0x4e in Object.wait() [cb101000..cb1019c0]
	at java.lang.Object.wait(Native Method)
	at com.waveset.util.BlockingQueue.waitToDequeue(BlockingQueue.java:39)
	- locked <0xe0a3dd38> (a com.waveset.util.BlockingQueue)
	at com.waveset.recon.ReconTask.getNextWorkItem(ReconTask.java:3816)
	at com.waveset.recon.ReconTask.access$1300(ReconTask.java:105)
	at com.waveset.recon.ReconTask$WorkerThread.run(ReconTask.java:2935)

"Reconciler Resource Thread" daemon prio=5 tid=0x017f1118 nid=0x4d in Object.wait() [cb201000..cb2019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0a3dea0> (a com.waveset.util.BlockingQueue)
	at com.waveset.util.BlockingQueue.waitToDequeue(BlockingQueue.java:39)
	- locked <0xe0a3dea0> (a com.waveset.util.BlockingQueue)
	at com.waveset.recon.ReconTask$ResourceThread.run(ReconTask.java:520)

"Reconciler" daemon prio=5 tid=0x00969758 nid=0x4b waiting on condition [cb401000..cb4019c0]
	at java.lang.Thread.sleep(Native Method)
	at com.waveset.recon.ReconTask.doRealWork(ReconTask.java:3321)
	at com.waveset.task.Executor.execute(Executor.java:157)
	at com.waveset.task.TaskThread.run(TaskThread.java:116)

"ListenThread.Default" prio=5 tid=0x01723ff0 nid=0x4a runnable [cb501000..cb5019c0]
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
	- locked <0xe09850e0> (a java.net.PlainSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:448)
	at java.net.ServerSocket.accept(ServerSocket.java:419)
	at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:24)
	at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:739)
	at weblogic.t3.srvr.ListenThread.run(ListenThread.java:292)

"com.waveset.logging.event.AuditEventHandler:" daemon prio=5 tid=0x0044e900 nid=0x48 in Object.wait() [cb701000..cb7019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0985250> (a java.lang.Object)
	at java.lang.Object.wait(Object.java:429)
	at com.ibm.logging.Handler.run(Handler.java:1053)
	- locked <0xe0985250> (a java.lang.Object)
	at java.lang.Thread.run(Thread.java:534)

"Task Scheduler" daemon prio=5 tid=0x01700b68 nid=0x47 runnable [cb800000..cb8019c0]
	at weblogic.apache.xerces.util.XMLStringBuffer.append(XMLStringBuffer.java:191)
	at weblogic.apache.xerces.util.XMLStringBuffer.append(XMLStringBuffer.java:205)
	at weblogic.apache.xerces.impl.XMLScanner.scanAttributeValue(XMLScanner.java:907)
	at weblogic.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(XMLDocumentFragmentScannerImpl.java:818)
	at weblogic.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:740)
	at weblogic.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1477)
	at weblogic.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:329)
	at weblogic.apache.xerces.parsers.DTDConfiguration.parse(DTDConfiguration.java:525)
	at weblogic.apache.xerces.parsers.DTDConfiguration.parse(DTDConfiguration.java:581)
	at weblogic.apache.xerces.parsers.XMLParser.parse(XMLParser.java:152)
	at weblogic.apache.xerces.parsers.DOMParser.parse(DOMParser.java:257)
	at weblogic.apache.xerces.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:201)
	at weblogic.xml.jaxp.RegistryDocumentBuilder.parse(RegistryDocumentBuilder.java:149)
	at com.waveset.util.XmlParser.parseString(XmlParser.java:496)
	- locked <0xe11d3870> (a com.waveset.util.XmlParser)
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:442)
	- locked <0xe0601de8> (a com.waveset.util.XmlUtil)
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:456)
	at com.waveset.repository.AbstractDataStore.deserializeItemXml(AbstractDataStore.java:6090)
	at com.waveset.repository.AbstractDataStore.deserializeItem(AbstractDataStore.java:5889)
	at com.waveset.repository.AbstractDataStore$Item.getObject(AbstractDataStore.java:707)
	at com.waveset.repository.AbstractDataStore.checkout(AbstractDataStore.java:2375)
	at com.waveset.repository.ServerRepository.checkout(ServerRepository.java:2865)
	at com.waveset.server.InternalSession.checkoutObject(InternalSession.java:536)
	at com.waveset.task.Scheduler.applyEvent(Scheduler.java:3282)
	at com.waveset.task.Scheduler.processEvent(Scheduler.java:3150)
	at com.waveset.task.Scheduler.processEvents(Scheduler.java:3076)
	at com.waveset.task.Scheduler.processReadyTasks(Scheduler.java:2983)
	at com.waveset.task.Scheduler.processTasks(Scheduler.java:1130)
	at com.waveset.task.Scheduler.run(Scheduler.java:956)

"Scheduler.PollThread" daemon prio=5 tid=0x014b1ee8 nid=0x46 waiting on condition [cb901000..cb9019c0]
	at java.lang.Thread.sleep(Native Method)
	at com.waveset.task.Scheduler$PollThread.run(Scheduler.java:5003)

"ChangeNotifier update thread" daemon prio=5 tid=0x00eb0a18 nid=0x45 in Object.wait() [cba01000..cba019c0]
	at java.lang.Object.wait(Native Method)
	at com.waveset.server.ChangeNotifier$ChangeNotifierUpdaterThread.run(ChangeNotifier.java:321)
	- locked <0xe0985478> (a com.waveset.server.ChangeNotifier$ChangeNotifierUpdaterThread)
	at java.lang.Thread.run(Thread.java:534)

"AuthorizationTypes watcher thread" daemon prio=5 tid=0x0129ea20 nid=0x44 waiting for monitor entry [cbb01000..cbb019c0]
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:438)
	- waiting to lock <0xe0601de8> (a com.waveset.util.XmlUtil)
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:456)
	at com.waveset.repository.AbstractDataStore.deserializeItemXml(AbstractDataStore.java:6090)
	at com.waveset.repository.AbstractDataStore.deserializeItem(AbstractDataStore.java:5889)
	at com.waveset.repository.AbstractDataStore$Item.getObject(AbstractDataStore.java:707)
	at com.waveset.repository.AbstractDataStore.checkoutIfExists(AbstractDataStore.java:2461)
	at com.waveset.repository.AbstractDataStore.getIfExists(AbstractDataStore.java:2414)
	at com.waveset.repository.ServerRepository.getIfExists(ServerRepository.java:2836)
	at com.waveset.server.ObjectWatcher.run(ObjectWatcher.java:67)

"System Configuration Watcher Thread" daemon prio=5 tid=0x00628750 nid=0x43 waiting for monitor entry [cbc01000..cbc019c0]
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:438)
	- waiting to lock <0xe0601de8> (a com.waveset.util.XmlUtil)
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:456)
	at com.waveset.repository.AbstractDataStore.deserializeItemXml(AbstractDataStore.java:6090)
	at com.waveset.repository.AbstractDataStore.deserializeItem(AbstractDataStore.java:5889)
	at com.waveset.repository.AbstractDataStore$Item.getObject(AbstractDataStore.java:707)
	at com.waveset.repository.AbstractDataStore.checkoutIfExists(AbstractDataStore.java:2461)
	at com.waveset.repository.AbstractDataStore.getIfExists(AbstractDataStore.java:2414)
	at com.waveset.repository.ServerRepository.getIfExists(ServerRepository.java:2836)
	at com.waveset.server.ObjectWatcher.run(ObjectWatcher.java:67)

"Thread-12" prio=5 tid=0x015ccab8 nid=0x42 in Object.wait() [cbd01000..cbd019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe05ef7b0> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:429)
	- locked <0xe05ef7b0> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"Thread-10" prio=5 tid=0x009090e8 nid=0x40 in Object.wait() [cc581000..cc5819c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe05eeaf8> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:429)
	- locked <0xe05eeaf8> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"Thread-9" prio=5 tid=0x00d6be88 nid=0x3f in Object.wait() [ce601000..ce6019c0]
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:429)
	- locked <0xe053f670> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"Thread-8" prio=5 tid=0x017f3178 nid=0x3e in Object.wait() [ce701000..ce7019c0]
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:429)
	- locked <0xe04e87b8> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"Thread-7" prio=5 tid=0x00b841b8 nid=0x3d in Object.wait() [ce801000..ce8019c0]
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:429)
	- locked <0xe04c5980> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"Thread-6" daemon prio=5 tid=0x015ce648 nid=0x3c in Object.wait() [cec01000..cec019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe04720b8> (a java.util.TaskQueue)
	at java.lang.Object.wait(Object.java:429)
	at java.util.TimerThread.mainLoop(Timer.java:403)
	- locked <0xe04720b8> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"weblogic.health.CoreHealthMonitor" daemon prio=5 tid=0x00a64810 nid=0x3b waiting on condition [cee01000..cee019c0]
	at java.lang.Thread.sleep(Native Method)
	at weblogic.t3.srvr.CoreHealthMonitorThread.run(CoreHealthMonitorThread.java:127)

"Thread-5" prio=5 tid=0x00df3208 nid=0x3a in Object.wait() [cef01000..cef019c0]
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:429)
	- locked <0xe02ecb18> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:382)

"ExecuteThread: '0' for queue: 'Multicast'" daemon prio=5 tid=0x012bc5a0 nid=0x39 in Object.wait() [cf001000..cf0019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe02d69b8> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"Thread-4" prio=5 tid=0x00d40178 nid=0x35 waiting on condition [cf401000..cf4019c0]
	at java.lang.Thread.sleep(Native Method)
	at com.netegrity.siteminder.agentcommon.framework.bw.run(DashoA8550)

"LDAPConnThread-0 ldap://128.137.22.79:10110" daemon prio=5 tid=0x012d0838 nid=0x32 runnable [cf601000..cf6019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
	- locked <0xe0231768> (a java.io.BufferedInputStream)
	at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
	at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
	at java.lang.Thread.run(Thread.java:534)

"VDE Transaction Processor Thread" prio=2 tid=0x00c180e8 nid=0x30 in Object.wait() [cf701000..cf7019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0212a50> (a com.octetstring.vde.backend.standard.TransactionProcessor)
	at java.lang.Object.wait(Object.java:429)
	at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:356)
	- locked <0xe0212a50> (a com.octetstring.vde.backend.standard.TransactionProcessor)
	at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x008acf18 nid=0x2f in Object.wait() [cf801000..cf8019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe01ad370> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x0186af78 nid=0x2e in Object.wait() [cf901000..cf9019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe01ace50> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x00632a80 nid=0x2d in Object.wait() [cfa01000..cfa019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe01ac910> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"weblogic.security.SpinnerRandomSource" daemon prio=5 tid=0x007fa9e8 nid=0x2c in Object.wait() [cfb01000..cfb019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe00a2818> (a java.lang.Object)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
	- locked <0xe00a2818> (a java.lang.Object)
	at java.lang.Thread.run(Thread.java:534)

"weblogic.time.TimeEventGenerator" daemon prio=5 tid=0x00280150 nid=0x2b in Object.wait() [cfc01000..cfc019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe00a0b90> (a weblogic.time.common.internal.TimeTable)
	at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:271)
	- locked <0xe00a0b90> (a weblogic.time.common.internal.TimeTable)
	at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:118)
	at java.lang.Thread.run(Thread.java:534)

"ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x003a0fd8 nid=0x2a in Object.wait() [cfd01000..cfd019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe00a0330> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x003a03e8 nid=0x29 in Object.wait() [cfe01000..cfe019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009fe88> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x0039f860 nid=0x28 in Object.wait() [cff01000..cff019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009f9e0> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x0065ef28 nid=0x27 in Object.wait() [d0001000..d00019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009f538> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x0065e338 nid=0x26 in Object.wait() [d0101000..d01019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009f090> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0065d7b0 nid=0x25 in Object.wait() [d0201000..d02019c0]
	at java.lang.Object.wait(Native Method)
	at weblogic.common.internal.DynaQueue.getW(DynaQueue.java:228)
	- locked <0xe00b2680> (a weblogic.common.internal.DynaQueue)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:277)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '23' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00eda688 nid=0x24 runnable [d0301000..d03019c0]
	at java.net.PlainDatagramSocketImpl.receive(Native Method)
	- waiting to lock <0xe046cf78> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:711)
	- locked <0xe4c860f8> (a java.net.DatagramPacket)
	- locked <0xe046ce88> (a java.net.MulticastSocket)
	at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:169)
	at weblogic.cluster.MulticastManager.execute(MulticastManager.java:377)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00ed9b00 nid=0x23 runnable [d0401000..d04019c0]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:651)
	- locked <0xe00b27a8> (a java.net.Socket)
	at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:617)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:282)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '21' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00ed9020 nid=0x22 in Object.wait() [d0501000..d05019c0]
	at java.lang.Object.wait(Native Method)
	at weblogic.common.internal.DynaQueue.getW(DynaQueue.java:228)
	- locked <0xe00b2680> (a weblogic.common.internal.DynaQueue)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:277)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003f8b50 nid=0x21 in Object.wait() [d0601000..d06019c0]
	at java.lang.Object.wait(Native Method)
	at weblogic.common.internal.DynaQueue.getW(DynaQueue.java:228)
	- locked <0xe00b2680> (a weblogic.common.internal.DynaQueue)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:277)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003f7f60 nid=0x20 in Object.wait() [d0701000..d07019c0]
	at java.lang.Object.wait(Native Method)
	at weblogic.common.internal.DynaQueue.getW(DynaQueue.java:228)
	- locked <0xe00b2680> (a weblogic.common.internal.DynaQueue)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:277)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003f73d8 nid=0x1f waiting for monitor entry [d0800000..d08019c0]
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:438)
	- waiting to lock <0xe0601de8> (a com.waveset.util.XmlUtil)
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:456)
	at com.waveset.object.Subject.<init>(Subject.java:131)
	at com.waveset.ui.LoginHelper.getSession(LoginHelper.java:146)
	at jsp_servlet._task.__tasklaunch._jspService(__tasklaunch.java:446)
	at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:971)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:402)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6370)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3635)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2585)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '17' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00219b78 nid=0x1e waiting for monitor entry [d08ff000..d09019c0]
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:438)
	- waiting to lock <0xe0601de8> (a com.waveset.util.XmlUtil)
	at com.waveset.util.XmlUtil.parseString(XmlUtil.java:456)
	at com.waveset.repository.AbstractDataStore.deserializeItemXml(AbstractDataStore.java:6090)
	at com.waveset.repository.AbstractDataStore.deserializeItem(AbstractDataStore.java:5889)
	at com.waveset.repository.AbstractDataStore$Item.getObject(AbstractDataStore.java:707)
	at com.waveset.repository.AbstractDataStore.checkoutIfExists(AbstractDataStore.java:2461)
	at com.waveset.repository.AbstractDataStore.getIfExists(AbstractDataStore.java:2414)
	at com.waveset.repository.ServerRepository.getIfExists(ServerRepository.java:2836)
	at com.waveset.server.Server.loadObject(Server.java:958)
	at com.waveset.object.ObjectCache.loadObject(ObjectCache.java:772)
	at com.waveset.object.ObjectCache.getHandle(ObjectCache.java:663)
	- locked <0xe0986790> (a com.waveset.object.ObjectCache)
	at com.waveset.object.ObjectCache.getObjectIfExists(ObjectCache.java:572)
	at com.waveset.object.ObjectCache.getObjectIfExists(ObjectCache.java:507)
	at com.waveset.server.InternalSession.getObjectIfExists(InternalSession.java:413)
	at com.waveset.object.LighthouseContextWrapper.getObjectIfExists(LighthouseContextWrapper.java:101)
	at com.waveset.object.ViewUtil.getForm(ViewUtil.java:69)
	at com.waveset.object.ViewUtil.getForm(ViewUtil.java:54)
	at com.waveset.view.UserViewer.getForm(UserViewer.java:3581)
	at com.waveset.view.UserViewer.createView(UserViewer.java:630)
	at com.waveset.object.ViewMaster.createView(ViewMaster.java:467)
	at com.waveset.session.WorkflowServices.createView(WorkflowServices.java:1204)
	at com.waveset.session.WorkflowServices.call(WorkflowServices.java:400)
	at com.waveset.workflow.WorkflowEngine.callApplication(WorkflowEngine.java:4089)
	at com.waveset.workflow.WorkflowEngine.callAction(WorkflowEngine.java:3916)
	at com.waveset.workflow.WorkflowEngine.callAction(WorkflowEngine.java:3229)
	at com.waveset.workflow.WorkflowEngine.execute(WorkflowEngine.java:3070)
	at com.waveset.workflow.WorkflowEngine.makeTransition(WorkflowEngine.java:2809)
	at com.waveset.workflow.WorkflowEngine.checkExplicitTransitions(WorkflowEngine.java:2700)
	at com.waveset.workflow.WorkflowEngine.checkTransitions(WorkflowEngine.java:2486)
	at com.waveset.workflow.WorkflowEngine.processSteps(WorkflowEngine.java:1891)
	at com.waveset.workflow.WorkflowEngine.walkCases(WorkflowEngine.java:1748)
	at com.waveset.workflow.WorkflowEngine.walkCases(WorkflowEngine.java:1656)
	at com.waveset.workflow.WorkflowEngine.execute(WorkflowEngine.java:816)
	at com.waveset.workflow.WorkflowEngine.execute(WorkflowEngine.java:478)
	at com.waveset.workflow.WorkflowExecutor.execute(WorkflowExecutor.java:235)
	at com.waveset.task.Scheduler.execute(Scheduler.java:2393)
	at com.waveset.task.Scheduler.launchTask(Scheduler.java:1564)
	at com.waveset.task.Scheduler.launchTask(Scheduler.java:1251)
	at com.waveset.task.TaskManager.launchTask(TaskManager.java:267)
	at com.waveset.server.InternalSession.runTask(InternalSession.java:2627)
	at com.waveset.server.ViewMaster.runTask(ViewMaster.java:909)
	at com.waveset.view.ProcessViewer.checkinView(ProcessViewer.java:656)
	at com.waveset.object.ViewMaster.checkinView(ViewMaster.java:708)
	at com.waveset.session.LocalSession.checkinView(LocalSession.java:664)
	at com.waveset.ui.util.GenericViewSource.checkinView(GenericViewSource.java:522)
	at com.waveset.ui.util.GenericEditForm.process(GenericEditForm.java:560)
	at jsp_servlet._task.__tasklaunch._jspService(__tasklaunch.java:674)
	at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:971)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:402)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:446)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6370)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3635)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2585)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '16' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00218980 nid=0x1d in Object.wait() [d0a01000..d0a019c0]
	at java.lang.Object.wait(Native Method)
	at weblogic.common.internal.DynaQueue.getW(DynaQueue.java:228)
	- locked <0xe00b2680> (a weblogic.common.internal.DynaQueue)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:277)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x008f75c0 nid=0x1c in Object.wait() [d0b01000..d0b019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009b830> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x008f6a00 nid=0x1b in Object.wait() [d0c01000..d0c019c0]
	at java.lang.Object.wait(Native Method)
	at weblogic.common.internal.DynaQueue.getW(DynaQueue.java:228)
	- locked <0xe00b2680> (a weblogic.common.internal.DynaQueue)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:277)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x008f5f20 nid=0x1a runnable [d0d00000..d0d019c0]
	at java.lang.Throwable.fillInStackTrace(Native Method)
	at java.lang.Throwable.<init>(Throwable.java:195)
	at java.lang.Exception.<init>(Exception.java:41)
	at java.io.IOException.<init>(IOException.java:40)
	at java.io.InterruptedIOException.<init>(InterruptedIOException.java:45)
	at java.net.SocketTimeoutException.<init>(SocketTimeoutException.java:24)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:651)
	- locked <0xe0a3e1a8> (a weblogic.socket.WeblogicSocket)
	at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:617)
	at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:282)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

"ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00c6e850 nid=0x19 in Object.wait() [d0e01000..d0e019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009aab0> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00c6dcf8 nid=0x18 in Object.wait() [d0f01000..d0f019c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009a600> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00ad30c0 nid=0x17 in Object.wait() [d1001000..d10019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe009a138> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe009a138> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00b6f408 nid=0x16 in Object.wait() [d1101000..d11019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0099c70> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0099c70> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00b8fb58 nid=0x15 in Object.wait() [d1201000..d12019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe00997a8> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe00997a8> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00aa3298 nid=0x14 in Object.wait() [d1301000..d13019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe00992e0> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe00992e0> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00aa2cf8 nid=0x13 in Object.wait() [d1401000..d14019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0098e18> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0098e18> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x009f9c60 nid=0x12 in Object.wait() [d1501000..d15019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0098950> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0098950> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x009f9ac0 nid=0x11 in Object.wait() [d1601000..d16019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0098488> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0098488> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x009f9920 nid=0x10 in Object.wait() [d1701000..d17019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0097fc0> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0097fc0> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x002a5240 nid=0xf in Object.wait() [d1801000..d18019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0097af8> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0097af8> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x002a50a0 nid=0xe in Object.wait() [d1901000..d19019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0097630> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0097630> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x00a7a7a8 nid=0xd in Object.wait() [d1a01000..d1a019c0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe0097050> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:145)
	- locked <0xe0097050> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:164)

"Signal Dispatcher" daemon prio=10 tid=0x000f1248 nid=0x8 waiting on condition [0..0]

"Finalizer" daemon prio=8 tid=0x000eddd8 nid=0x6 in Object.wait() [fb981000..fb9819c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
	- locked <0xdfea0378> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000ed550 nid=0x5 in Object.wait() [fba81000..fba819c0]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
	- locked <0xdfea02a0> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x00037958 nid=0x1 in Object.wait() [ffbee000..ffbee444]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xdff304d0> (a weblogic.t3.srvr.T3Srvr)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1170)
	- locked <0xdff304d0> (a weblogic.t3.srvr.T3Srvr)
	at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:384)
	at weblogic.Server.main(Server.java:32)

"VM Thread" prio=5 tid=0x000ecf00 nid=0x4 runnable 

"VM Periodic Task Thread" prio=10 tid=0x000f6280 nid=0xc waiting on condition 
"Suspend Checker Thread" prio=10 tid=0x000efff8 nid=0x7 runnable 
<Aug 12, 2006 12:04:09 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:11 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:14 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:16 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:19 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:21 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:24 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:26 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:29 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:31 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:34 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:36 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:39 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:41 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:44 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:46 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:49 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:51 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:54 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:56 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:04:59 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:05:01 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:05:04 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:05:06 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<Aug 12, 2006 12:05:09 AM PDT> <Debug> <HTTP> <BEA-101147> <HttpServer(21011496,null default ctx,identity_m2) Found no context for "/". This request does not match the context path for any installed Web applications, and there is no default Web application configured.> 
<A

Guys you were so helpful.
Thanks,
pandu