Trying to debug truss command of PID in Solaris 10.

I'm getting the following output from a truss operation of a PID from an application that will not start properly, something with the writing to an archive. Here is the output:

open("arch/chkpt.arch", O_RDWR)                 Err#13 EACCES
open("arch/chkpt.arch", O_RDWR|O_CREAT|O_TRUNC, 0666) Err#13 EACCES
rename("", "./error/")                          Err#2 ENOENT
open("/usr/local/gentran61/prod/./temp//arch.l", O_WRONLY|O_APPEND|O_CREAT, 0666) = 4
llseek(4, 0, SEEK_END)                          = 0x137016BB
time()                                          = 1552504635
fstat64(4, 0xFFBFD738)                          = 0
fstat64(4, 0xFFBFD5E0)                          = 0
ioctl(4, TCGETA, 0xFFBFD6C4)                    Err#25 ENOTTY
write(4, " a r c h : 3 1 5 4 : 0 3".., 85)      = 85
close(4)                                        = 0
unlink("")                                      Err#2 ENOENT
unlink("arch/chkpt.arch")                       Err#13 EACCES
open("arch/chkpt.arch", O_RDWR)                 Err#13 EACCES
open("arch/chkpt.arch", O_RDWR|O_CREAT|O_TRUNC, 0666) Err#13 EACCES
rename("", "./error/")                          Err#2 ENOENT
open("/usr/local/gentran61/prod/./temp//arch.l", O_WRONLY|O_APPEND|O_CREAT, 0666) = 4
llseek(4, 0, SEEK_END)                          = 0x13701710
time()                                          = 1552504635
fstat64(4, 0xFFBFD738)                          = 0
fstat64(4, 0xFFBFD5E0)                          = 0
ioctl(4, TCGETA, 0xFFBFD6C4)                    Err#25 ENOTTY
write(4, " a r c h : 3 1 5 4 : 0 3".., 85)      = 85
close(4)                                        = 0
unlink("")                                      Err#2 ENOENT
unlink("arch/chkpt.arch")                       Err#13 EACCES
open("arch/chkpt.arch", O_RDWR)                 Err#13 EACCES
open("arch/chkpt.arch", O_RDWR|O_CREAT|O_TRUNC, 0666) Err#13 EACCES
rename("", "./error/")                          Err#2 ENOENT
open("/usr/local/gentran61/prod/./temp//arch.l", O_WRONLY|O_APPEND|O_CREAT, 0666) = 4
llseek(4, 0, SEEK_END)                          = 0x13701765
time()                                          = 1552504635
fstat64(4, 0xFFBFD738)                          = 0
fstat64(4, 0xFFBFD5E0)                          = 0
ioctl(4, TCGETA, 0xFFBFD6C4)                    Err#25 ENOTTY
 write(4, " a r c h : 3 1 5 4 : 0 3".., 85)      = 85

It can't open, create, or delete arch/chkpt.arch. Can you ls -l that file and ls -ld its folder and post the permissions please?

I can manually create the file with the same username/UID with no errors. permissions are wide open at 777 on the arch folder as well.

ls -l | grep arch
drwxrwxrwx 3 ediprod edi 1945600 Mar 13 19:49 arch

That file does not exist but I believe it's created as a checkpoint for the application to rename archive files. All the files in the folder are 777 as well just to rule out permissions. I went from an NFS share on an IBM v7000 to just a straight up Ubuntu 16 Linux server running NFS. All the mount/export options are the same from both storage devices.

So the file is on an NFS mount. Correct?
How is the failing process run (executed).... using batch or crontab?
Can you run the problem job successfully when logged in interactively as that user and at the command line?? ... I think that is what you were trying to say. Correct me, please.

Unfortunately I inherited an old version of Gentran EDI software that runs it's application on an NFS share. It was on an IBM v7000 and we migrated to a regular Ubuntu Linux server running NFS server and I've migrated several successful shares with no issues. However on this particular share with the application it will not start that application as version 3 NFS on Solaris client. If I go to version 4 it will start the application but in degraded mode and having all kinds of ACL issues.

Process is started manually but within the NFS share just running an application command that's compiled for Solaris. You start it running a shell program and it calls this executable. I traced it down to the archive portion I started this thread with. I can write to that folder and everything but I keep getting this error in the truss output.

Can you unmount the NFS mountpoint (I guess it is /arch) on the solaris box? At that point what does ubuntu think? You do have an NFS problem. I have had this issue as well. It was caused in NFS shutdown by moving directories without knowing what zone had an open NFS file, as in cwd for example. The global zone kept track of NFS mounts. Get the inode from inside the zone and in the global zone. If they do not match you have this problem. ls -di /path/to/dirname will give you inode for given directory

As an aside - a workaround: why can you not simply move the entire /arch directory tree to Solaris, set up NFS to serve the disk from Solaris to elsewhere if required.

You say you manually can create files. Did you do this on the Solaris box??
arch/chkpt.arch is relative to the current work directory. Please peek the process with

pwdx <PID of the running process>

and verify.