Solaris 10. Getting a boatload of "Resource unavailable" on read errors for a Java process which is taking 350% of CPU. No I/O issues and dtrace shows lots of lwp calls. There are only 54 threads (or so, it fluxuates) but certainly not a lot. App is WebLogic.
App has been up since the third and we don't know if the app guys did the Java DST patch since then. System is a recent upgrade to Solaris 10 so perhaps there's a /etc/system setting that needed to be increased.
I used pstack to examine the worst thread but nothing popped out. I also used the dtrace command to example the stack. Lots of libjvm.so calls which is to be expected. Again, nothing about a missing resource is popping up.
iotop doesn't show anything there that's an issue and netstat -i doesn't show much network activity.
One DTrace agent library is called the "dvmti" agent. This can be run inside the Java Virtual Machine (JVM) to provide DTrace probes that are specific to the JVM and the Java platform. This agent is available for download from the Java.net project solaris10-dtrace-vm-agents. The dvmti agent uses the Java Virtual Machine Tool Interface (JVM TI). Specifically, the agent library utilizes the JVM TI interface to request various VM events and provide DTrace probes in the callback code for those events. To use these DTrace probes for the JVM and Java code, the Java application has to be started as shown here:
java -Xrundvmti[:options] <MainClass>
Where options are:
all same as: unloads,allocs,stats,methods
help print help message
unloads track class unloads
allocs track object allocations and frees
stats generate heap stats after each GC
methods generate method entry exit events
exclude=name exclude class names
default is none of unloads, allocs, stats, or methods.
The dvmti agent will utilize byte code instrumentation (BCI) where needed and will need a jar file called dvmti.jar, which should be located in the same directory as the shared library libdvmti.so. The JDK 5.0 release has an environment variable that can be used to include options in the startup of any Java application:
export JAVA_TOOL_OPTIONS="-Xrundvmti:all"
You could also try jstat & jstatd to check the operations..
Thanks for the info. That sounds like something the app guys would be more likely to use. I was looking for something on the system side. I think we went as far as we could as sysadmins and needed to kick it back to the app guys to resolve. I just wanted to make sure there wasn't an additional tool I could use to further identify the problem from my point of view or if there was a configuration change that should have been done.
/etc/system only had the couple of lines related to SDS and nothing configuring the system for the use of WebLogic. There are 4 CPUs and 16 gigs of ram. It seems the system is plenty powerful enough to handle the load. It just seems odd that it's having such trouble.