Unable to remove package in Solaris-10

I am installing a IBM package (ILMT) on Solaris-10 box, which is having few non global zones. It is not allowing me to remove old package and not allowing me to install new one.

root@pqdb_s50:/root# pkgrm ILMT-TAD4D-agent
## Waiting for up to <300> seconds for package administration commands to become available (another user is administering packages on zone <pqdb_s50-zrpp10>)
pkgrm: ERROR: ERROR: Unable to acquire package administration lock for zone <pqdb_s50-zrpp10>; please try again later
pkgrm: ERROR: ERROR: Unable to acquire lock on non-global zone <pqdb_s50-zrpp10>: releasing all locks
pkgrm: ERROR: unable to lock zones to perform operations

1 package was not processed!
pqdb_s50-zrpp10 is one of its non global zone. I have already deleted /var/sadm/install/.lockfile and /tmp/.ai.pkg.zone.lock-afdb66cf-1dd1-11b2-a049-000d560ddc3e files. But /tmp/.ai.pkg.zone.lock-afdb66cf-1dd1-11b2-a049-000d560ddc3e keeps coming back, whenever I am running pkgrm.
How should I remove this ?

make sure that there is nobody in the non-global zone actually doing something with that package then rerun pkgrm -A ILMT-TAD4D-agent

No luck. This instance is installed in non-global zone also. I tried stopping services (old) and removing it from non-global zone first. There also, it is giving same error.

unless somebody here has a better alternative ...

if you could reboot the non-global zone without consequence, reboot it and then retry the pkgrm ...

if still an issue, you may have to remove the package the hard way by actually deleting the ILMT-TAD4D-agent directory under /var/sadm/pkg and then rerunning the pkgrm -A command if pkginfo still lists the package ...

I just noticed, when I was trying to remove/install in few more servers. I am facing this issue when ITLM is installed in global as well as any of its zone also.
on other servers, where it is installed only on global or only on non-global, there is no problem.

Try pkgrm -G (remove on the current zone only).

It doesn't recognize switch

/# pkgrm -G ILMT-TAD4D-agent
pkgrm: illegal option -- G
usage:
        pkgrm [-a admin] [-n] [[-M|-A] -R host_path] [-V fs_file] [-v] [-Y category[,category ...] | pkg [pkg ...]]
        pkgrm -s spool [-Y category[,category ...] | pkg [pkg ...]]

Have you tried removing it from the non-globals first then from the global?

Yes. Stopped services on both, deleted lock files from both, tried removing it from non-global zone first, but no luck.
This is production box, so I want to try all possible things before going with downtime for reboot.

in that case, do ...

cd /var/sadm/pkg
ls -d stubborn_pkg
rm -rf stubborn_pkg

if unsure of the process, rename the stubborn_pkg directory to something else and recheck with pkginfo -l stubborn_pkg to confirm that the process works ...

It did not helped. It is still looking for some instance

root@pqdb_s50:# mv /var/sadm/pkg/ILMT-TAD4D-agent /var/sadm/pkg/ILMT-TAD4D-agent.old
root@pqdb_s50:# pkgrm ILMT-TAD4D-agent
pkgrm: ERROR: no package associated with <ILMT-TAD4D-agent>
root@pqdb_s50:# pkgadd -d ILMT-TAD4D-agent-7.5.0.111-solaris-sparc64
The following packages are available:
  1  ILMT-TAD4D-agent     ILMT-TAD4D-agent
                          (sparc64) 7.5.0.111
Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:
## Waiting for up to <300> seconds for package administration commands to become available (another user is administering packages)
pkgadd: ERROR: ERROR: Unable to acquire package administration lock for this system; try again later
pkgadd: ERROR: Unable to lock this zone for administration
1 package was not processed!
root@pqdb_s50:# 

run kill -9 $(ps -ef | grep pkgadd | grep -v grep | awk '{print $2}') prior to running pkgadd as well as remove any lockfiles ...

check for any other pkg processes and then retry pkgadd again if none found ...

There is no pkg process alive, I checked with "ps -ef". But I can see a lock file in /tmp

.ai.pkg.zone.lock-afdb66cf-1dd1-11b2-a049-000d560ddc3e

I remove it and then try pkgrm. It again fails and when I checked /tmp, a new lock file gets created.

run truss on the pkgadd (see code below) in a different window to see if there is anything there that can give you a hint on what is going on ...

truss -p $(ps -ef | grep pkgadd | grep -v grep | awk '{print $2}')

if still nothing, you may have to actually reboot both the non-global zone and then the host before you retry the pkgadd again ...

if still nothing, then i would suggest giving oracle support a call as your issue may be os-related ...

Am I missing some syntax ?

:/# truss -p $(ps -ef | grep pkgadd | grep -v grep | awk '{print $2}')
usage:  truss [-fcaeildD] [-[tTvx] [!]syscalls] [-[sS] [!]signals] \
        [-[mM] [!]faults] [-[rw] [!]fds] [-[uU] [!]libs:[:][!]funcs] \
        [-o outfile]  command | -p pid[/lwps] ...

Ensure that each zone has its own /var/sadm/

zonename
df /var/sadm
ls -li /var/sadm

It is sparse root zone

root@pqdb_s50:/# zonename
pqdb_s50
root@pqdb_s50:/# df /var/sadm
Filesystem            kbytes    used   avail capacity  Mounted on
/                    8178892  891753 7287139    11%    /
root@pqdb_s50:/# ls -li /var/sadm
total 117
      4329 -r--r--r--   1 root     sys         1092 Jan  9  2005 README
      4330 dr-xr-xr-x   4 root     bin            9 Aug  7 10:15 install
      4723 drwxr-xr-x   2 root     sys            2 Aug 30  2011 install_data
     16513 drwxr-xr-x  48 root     root          48 Dec  2  2011 patch
      4898 dr-xr-xr-x 1219 root     sys         1219 Aug  8 09:42 pkg
      4788 dr-xr-xr-x   2 root     sys            2 Aug 30  2011 security
      4671 drwxr-xr-x   3 root     sys            3 Aug 30  2011 servicetag
      4333 drwxr-xr-x   2 root     sys            3 Aug 30  2011 softinfo
      4648 drwxr-xr-x   4 root     sys            4 Sep  1  2011 spool
      4335 drwxr-xr-x   2 root     sys            3 Aug 30  2011 svm3
        13 drwxr-xr-x   5 root     sys            6 Dec  2  2011 system
      4339 drwxr-xr-x   5 root     sys            5 Aug 30  2011 wbem
root@pqdb_s50:/# 

you probably started the truss run before you started pkgadd ... the command line i gave would have given you the pid of pkgadd without looking for it separately through ps ... however, the pkgadd process needs to be running before you can find its pid which is why you were getting the truss error because there was no pid found ...

You are right. This time, if I run pkgrm, it immediatelly comes back to prompt, failing on below error message

:/# pkgrm ILMT-TAD4D-agent
## Waiting for up to <300> seconds for package administration commands to become available (another user is administering packages)
pkgrm: ERROR: ERROR: Unable to acquire package administration lock for this system; try again later
pkgrm: ERROR: Unable to lock this zone for administration
1 package was not processed!

Then without removing it, I run pkgadd command and was waiting for

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:

On another terminal, I ran truss command and pressed enter for default on first terminal. Here is truss output

/# truss -p $(ps -ef | grep pkgadd | grep -v grep | awk '{print $2}')
read(0, 0xFEEB9D10, 1024)       (sleeping...)
read(0, "\n", 1024)                             = 1
open64("/var/tmp/dstreAAAbBayew/signature", O_RDONLY|O_NONBLOCK) Err#2 ENOENT
zone_lookup("")                                 = 2
zone_getattr(2, ZONE_ATTR_NAME, 0xFFBFF098, 64) = 14
getpid()                                        = 11299 [10262]
getpid()                                        = 11299 [10262]
zone_lookup("")                                 = 2
zone_lookup("")                                 = 2
zone_getattr(2, ZONE_ATTR_NAME, 0xFFBFEAB8, 64) = 14
access("/usr/bin/pkgadm", X_OK)                 = 0
open("/dev/null", O_RDONLY)                     = 4
pipe()                                          = 5 [6]
lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0x0000FFFF) = 0x00000000 [0x00000000]
vfork()                                         = 11308
lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
close(4)                                        = 0
close(6)                                        = 0
read(5, " E R R O R", 256)                      = 5
read(5, " :   c a n n o t   c r e".., 251)      = 102
read(5, 0x000741DB, 149)                        = 0
close(5)                                        = 0
waitid(P_PID, 11308, 0xFFBFEA18, WEXITED|WTRAPPED) = 0
write(2, " # #   W a i t i n g   f".., 131)     = 131
write(2, "\n", 1)                               = 1
getpid()                                        = 11299 [10262]
zone_lookup("")                                 = 2
zone_lookup("")                                 = 2
zone_getattr(2, ZONE_ATTR_NAME, 0xFFBFEAB8, 64) = 14
access("/usr/bin/pkgadm", X_OK)                 = 0
open("/dev/null", O_RDONLY)                     = 4
pipe()                                          = 5 [6]
lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0x0000FFFF) = 0x00000000 [0x00000000]
vfork()                                         = 11310
lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
close(4)                                        = 0
close(6)                                        = 0
read(5, " E R R O R", 256)                      = 5
read(5, " :   c a n n o t   c r e".., 251)      = 102
read(5, 0x000741DB, 149)                        = 0
close(5)                                        = 0
waitid(P_PID, 11310, 0xFFBFEA18, WEXITED|WTRAPPED) = 0
write(2, " p k g a d d", 6)                     = 6
write(2, " :   E R R O R :  ", 9)               = 9
write(2, " E R R O R :  ", 7)                   = 7
write(2, " U n a b l e   t o   a c".., 78)      = 78
write(2, "\n", 1)                               = 1
write(2, " p k g a d d", 6)                     = 6
write(2, " :   E R R O R :  ", 9)               = 9
write(2, " U n a b l e   t o   l o".., 43)      = 43
write(2, "\n", 1)                               = 1
close(3)                                        = 0
access("/bin/rm", X_OK)                         = 0
open64("/dev/null", O_RDONLY)                   = 3
pipe()                                          = 4 [5]
lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0x0000FFFF) = 0x00000000 [0x00000000]
vfork()                                         = 11312
lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
close(3)                                        = 0
close(5)                                        = 0
read(4, " r m :   C a n n o t   r".., 256)      = 101
read(4, "\n", 155)                              = 1
read(4, 0x00073F16, 154)                        = 0
close(4)                                        = 0
waitid(P_PID, 11312, 0xFFBFE9A0, WEXITED|WTRAPPED) = 0
access("/var/tmp/dstreAAAbBayew", F_OK)         = 0
access("/bin/rmdir", X_OK)                      = 0
open64("/dev/null", O_RDONLY)                   = 3
pipe()                                          = 4 [5]
lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0x0000FFFF) = 0x00000000 [0x00000000]
vfork()                                         = 11313
lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
close(3)                                        = 0
close(5)                                        = 0
read(4, " r m d i r", 256)                      = 5
read(4, " :   d i r e c t o r y  ".., 251)      = 76
read(4, 0x00073F01, 175)                        = 0
close(4)                                        = 0
waitid(P_PID, 11313, 0xFFBFE9A0, WEXITED|WTRAPPED) = 0
schedctl()                                      = 0xFF3F4000
sigaction(SIGINT, 0xFFBFF058, 0xFFBFF0F8)       = 0
sigaction(SIGHUP, 0xFFBFF058, 0xFFBFF0F8)       = 0
chdir("/")                                      = 0
write(2, "\n 1   p a c k a g e   w".., 30)      = 30
write(2, "\n", 1)                               = 1
_exit(1)

And then it fails.