I have a Solaris Ultra 25 that has run out of space on the root partition. When I try to delete files it says there is not enough space. I have tried 'echo "" >> {a file}' and can overwrite it (zero it out) but the space seems to immediately go away. It could be a hardware problem but I don't think so. The system is using zfs and has mirrored system disks. It's possible it is trying to write a crash dump but I don't know. I am a new Solaris admin, but have been a Linux admin for quite a while.
Hello,
Welcome to the forum ! We hope you find this to be a friendly and helpful place, and that you enjoy your time here.
You say that you get an error saying there is not enough space when you actually try to delete files. That's a bit odd - the rm command itself does not (and indeed, can not) ever consume disk space, so you wouldn't normally expect to get an error to that effect from the rm command itself. You can in certain situations find that removing files does not immediately return disk space, certainly - is that what's going on ? In any case, if you are genuinely getting an error returned by the rm command, then if you could copy-and-paste the exact command you're typing, and the full output you get returned, that would be helpful in figuring out what may be going on.
If instead it is the case that you're removing files and never getting space back, the most classic example of why that is happening is that a running process was writing to the file, and you have removed that file while the process is still running. That causes the file handle to remain open, still linked to the same inode, and thus the space is not freed up until such time as the process actually exits and closes the file handle. So if you're trying to clear out your syslog file, or the log file of an application such as a database or Java process, you need to re-start that service to get it to free up the space.
Lastly, can you tell us what version of Solaris you're running ? I'm guessing it's likely to be Solaris 10, but if you could just confirm that 100% either way that would be great.
Hope this helps !
Kevin,
Strange is it might seem, I can't remove a file. I can sometimes overwrite a file but even that is somewhat intermittent (I think the command 'yes > {the file}, wait a few seconds and then press ^c) worked to overwrite it so it had zero length. I have gotten rid/zeroed out of most of the larger files that I can find that are safe to remove (/var/log/, /var/tmp, /tmp, /var/crash). I understand that files that are still open will not return the space until after the controlling process releases the file handles. A lot of the files I zeroed or removed should not have had active processes attached to them and I have rebooted the system a few times.
Another strange thing is that I did an init 0, and at the boot prompt I said "boot -s" to go into single user mode, but after it booted (entered root's password) I could ping the system, and from the system I could ping the gateway, but ssh would not start due to insufficient space. Currently the only way to access it is from the serial port. I am not a Solaris guy, but I would think single user mode would not enable networking at all.
Please see responses below.
Thank you,
Carl
bash-3.2# rm vmdump.0
rm: vmdump.0 not removed: No space left on device
bash-3.2# df -h
Filesystem size used avail capacity Mounted on
rpool/ROOT/s10s_u11.202111
146G 84G 0K 100% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 220M 1.5M 219M 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
fd 0K 0K 0K 0% /dev/fd
rpool/ROOT/s10s_u11.202111/var
146G 6.0G 0K 100% /var
swap 219M 0K 219M 0% /tmp
swap 219M 8K 219M 1% /var/run
bash-3.2# uname -a
SunOS sekhmet 5.10 Generic_153153-01 sun4u sparc SUNW,A70
bash-3.2# cat /etc/rel
Oracle Solaris 10 1/13 s10s_u11wos_24a SPARC
Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.
Assembled 17 January 2013
This is normal behaviour and can happen on a ZFS filesystem. Deleted files are still open so the disk space is not reclaimed.
Deleted file descriptors that are still open can be identified with:
# find /proc/*/fd -type f -links 0 !-size 0 -ls
Thanks for the help. After poking around some more we found a backup script that was trying to create a flar and that was consuming all the space. I was able to zero out that file and then comment out the backup script and all seems good now.
You could have also (temporarily) significantly reduced your swap space do a minimal amount and that would have given you a lot of "head room" to work.
swap 220M 1.5M 219M 1% /etc/svc/volatile
swap 219M 0K 219M 0% /tmp
swap 219M 8K 219M 1% /var/run
Remember, swap space is "a good thing" but it's not "so good" as it is more important than actually disk space when the server is running up against a major disk space issue.
It's kinda like wearing your best running shoes when you have a broken leg and cannot run; the running shoes look "cool" but they are not necessary. When you are running out of disk space, as an interim measure, you can just reduce that space while working on the underlying issue(s) caused the problem.
HTH
Neo, you have shown file systems that are mounted on "swap" that stands for "virtual memory".
The on-disk swap space is shown with swap -l
And it could be on a raw disk partition or a zfs pool that is not the root pool.
The problem that deletion of a file needs temporary space is an old bug in zfs. Should have been fixed meanwhile.
Truncation of a file is done with
true > filename
If it works on an open file depends on the open mode.