We have a user where a process is behaving abnormally and crashing. It would be great if a core file was generated, but I don't see one.
user101671:/apps/ -$lscore -d
compression: on
path specification: on
corefile location: /opt/core
naming specification: off
There is no core file being created in the /opt/core directory. Now, I have asked the user to submit the output of errpt -a and to check for errors with the following:
LABEL: CORE_DUMP_FAILED .
There were none, unfortunately.
This resource from IBM contains some valuable information:
IBM Technical support search - United States
The user does not have root access to the AIX box, which means they cannot modify the location where core files are being generated, which I don't think would matter.
My question is whether having root access could be responsible for the core files not being generated or could it be some other cause?
A couple of things come to mind:
is /opt/core (rather /opt as directory) writeable by the process (could be permissions, could be /opt is mounted read-only).
The full errpt output gives a hint (errno) as to why it says it fails (i.e., get the number and than check /usr/include/error.h for an explanation).
From IBM Technical support search - United States - same as you provided!!!
Failure Causes
CORE DUMP FAILED - SEE A REASON CODE BELOW
...
REASON CODE
11
USER ID
232
...
Hope this helps!
Hi Michael,
Thanks for your input.
No, the user does not have write permissions to /opt/core , but only read and execute.
However, we are expecting the core dump to appear in the working directory of the process which is crashing. The user does have write permission to this directory.
I decided to collect some permission data:
nzapap671:/apps/iwstu/appserver/jboss/server/IDMDomain/log
-$aclget /opt/core
*
* ACL_type AIXC
*
attributes: SVTX
base permissions
owner(root): rwx
group(system): r-x
others: r-x
extended permissions
disabled
nzapap671:/apps/iwstu/appserver/jboss/server/IDMDomain/log -$aclget /opt
*
* ACL_type AIXC
*
attributes:
base permissions
owner(root): rwx
group(system): r-x
others: r-x
extended permissions
disabled
nzapap671:/apps/iwstu/appserver/jboss/server/IDMDomain/log -$ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user) unlimited
nzapap671:/apps/iwstu/appserver/jboss/server/IDMDomain/log -$ulimit -c
unlimited
nzapap671:/apps/iwstu/appserver/jboss/server/IDMDomain/log -$ulimit -f
unlimited
The output of errpt -a is as follows, which does not display any Core Dump related failures.
nzapap671:/apps/ptadm2/tower/vidm/home/runtime -$errpt -a
---------------------------------------------------------------------------
LABEL: J2_FS_FULL
IDENTIFIER: F7FA22C9
Date/Time: Mon Mar 21 00:03:26 2016
Sequence Number: 4767
Machine Id: 00F646944C00
Node Id: nzapap671
Class: O
Type: INFO
WPAR: Global
Resource Name: SYSJ2
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FULL
Recommended Actions
INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
REMOVE UNNECESSARY DATA FROM FILE SYSTEM
USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED
Detail Data
JFS2 MAJOR/MINOR DEVICE NUMBER
0026 0001
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/lv_fgl, /apps/foglight
---------------------------------------------------------------------------
LABEL: J2_FS_FULL
IDENTIFIER: F7FA22C9
Date/Time: Fri Mar 18 00:56:41 2016
Sequence Number: 4766
Machine Id: 00F646944C00
Node Id: nzapap671
Class: O
Type: INFO
WPAR: Global
Resource Name: SYSJ2
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FULL
Recommended Actions
INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
REMOVE UNNECESSARY DATA FROM FILE SYSTEM
USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED
Detail Data
JFS2 MAJOR/MINOR DEVICE NUMBER
000A 0009
FILE SYSTEM DEVICE AND MOUNT POINT
Since the failing process is in a directory with the path /apps/ptadm2/tower/vidm/home/runtime/is/ , I assume this is where the core dump would be. The output of the lscore -d is what confuses me. I am unsure which directory the file should appear in -- however due to output of errpt -a , I am unsure whether the system is attempting to generate the core file at all.
The literal error message is "File System Full", and the filesystem to focus on is:
Detail Data JFS2 MAJOR/MINOR DEVICE NUMBER 0026 0001 FILE SYSTEM DEVICE AND MOUNT POINT /dev/lv_fgl, /apps/foglightand
Detail Data JFS2 MAJOR/MINOR DEVICE NUMBER 000A 0009 FILE SYSTEM DEVICE AND MOUNT POINT
From memory, 000A 0009 is rootvg and /opt.
Maybe the message is wrong (FileSystem Full) - but I would check.
In any case, I would consider making /opt/core a separate filesystem rather than piggyback on /opt.
Hope this helps!