Sol11 - Error at prompt (no space left on device)

UNIX Solaris
1 
user1@:/$
-ksh: line 3: write to 1 failed [No space left on device]
user11@:/$
-ksh: line 3: write to 1 failed [No space left on device]
user1@:/$
-ksh: line 3: write to 1 failed [No space left on device]
user1@:/$
-ksh: line 3: write to 1 failed [No space left on device]
user1@:/$
-ksh: line 3: write to 1 failed [No space left on device]

But theres plenty of space :-

user1@:/$ df -kh
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris      59G    14G        21G    41%    /
/devices                 0K     0K         0K     0%    /devices
/dev                     0K     0K         0K     0%    /dev
ctfs                     0K     0K         0K     0%    /system/contract
proc                     0K     0K         0K     0%    /proc
mnttab                   0K     0K         0K     0%    /etc/mnttab
swap                   148M   3.0M       145M     3%    /system/volatile
objfs                    0K     0K         0K     0%    /system/object
sharefs                  0K     0K         0K     0%    /etc/dfs/sharetab
fd                       0K     0K         0K     0%    /dev/fd
rpool/ROOT/solaris/var
                        59G    18G        21G    46%    /var
swap                   151M   6.4M       145M     5%    /tmp
rpool/VARSHARE          59G   3.1M        21G     1%    /var/share
rpool/export            59G    32K        21G     1%    /export
rpool/export/home       59G   438K        21G     1%    /export/home
rpool                   59G    73K        21G     1%    /rpool
rpool/VARSHARE/zones    59G    31K        21G     1%    /system/zones
rpool/VARSHARE/pkg      59G    32K        21G     1%    /var/share/pkg
rpool/VARSHARE/pkg/repositories
                        59G    31K        21G     1%    /var/share/pkg/repositories
2

So what is line 3 of the script?

Is it definitely trying to write locally or to some remote volume?

You can also run out of inodes rather than disk space but I would expect the error to be different.

More info please.

3

Sorry - there is no script. I thought that at first.

This is just hitting return at the command line..... Weird thing is it seems a bit intermittent too.

Hmm, inodes?

4

You might think the error is from a command line but, if you are on the system console, it's just the OS reporting an error. Is another user on the system running the script?

5

The device with no space left might be any device on the system (eg, a USB stick) or simply a scripting syntax error. Until we know more we cannot say.

6

Hi,

Have you checked for snapshots? Try:-

zfs list -t snapshot

Regards

Gull04

2 Likes
7

/tmp seems awfully small in the output above.

Is this machine under heavy load memory wise ?

Please post the output of :

echo "::memstat" | mdb -k

Check dmesg for unable to fork errors perhaps..

Also the exact version of Solaris :

pkg info entire

Regards
Peasant

8

Just an observation - in my own experience I only ever see this when /tmp is full.

Non researched suggestion: add a (bigger) swap volume!

You have a *tiny* /tmp so it wouldn't be unusual to see a scheduled activity fill it. If that process then dies and cleans up you will have your 145MB back again. However there are also situations where a Virtual Memory (swap) reservation fails. [See Note2]

Note: In Solaris /tmp is backed off onto the virtual memory subsystem (swap). Available swap is the sum of physical swap space + (some?) free RAM (the freelist?).
This is not visible from the output you have given.
Use swap -l to see physical swap space and swap -s for the summary.

Note2: Solaris refuses to overcommit virtual memory and a large parent process calling a fork may try to reserve the same memory footprint of the forking process. Search "Minimizing Memory Usage for Creating Application Subprocesses" in oracle.com for an article covering this.

9

I would look where descriptor 1 goes, with

pfiles $$

Or, if this is not possible, display the pid

echo $$

and run the pfiles <pid> in another shell.

10

I've seen this alot. there are a few causes. I'll list them in order of occurrence for me.

  1. Boot Environments; see
beadm list
  1. Snapshots; see
zfs list -t snapshot
  1. extraneous files in /; see
du -akd /|sort -nr|more