No space left on device when using rm

Hello people

I have a small fileserver running busybox (very small linux distro with most essential stuff on it) and I am trying to remove some unused directories on it.

When I try this:

rm -R test/

I get:

rm: cannot remove 'test': No space left on device

df shows:

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock2           32768     15868     16900  48% /
none                    128004        12    127992   0% /tmp
/tmp/.cemnt/sdb1     369386932  54555641 290412091  16% /tmp/.cemnt/mnt_sdb1
/tmp/.cemnt/sda5     1915824128 1186595444 631553488  65% /tmp/.cemnt/mnt_sda5
/dev/mtdblock3           93184     17384     75800  19% /opt

The drive I am trying to remove this folder from is /mnt_sdb1. There is still plenty of space left there, so I am at a loss here what the cause could be. I can still copy files to the drive as well, so I believe it cannot be an inode-problem as well.

When trying to remove the folder by referring to its inode number:

-bash-3.2# find . -inum 7888609 -exec rm -r -i {} \;
rm: descend into directory './test'? y
rm: remove directory './test'? y
rm: cannot remove './test': No space left on device

So what could be the cause and how can I remove obsolete folder or files from the filesystem?

Thanks in advance!

truncate each file in the directory

find /path/to/directory -type f -exec cat /dev/null > {} \;

Then use rm on the whole directory tree you want to remove

rm -R /path/to/directory

Looks like a bug in the file system (or the file system driver).
What file system is it?

mount | grep sdb1

 mount | grep sdb1
/tmp/.cemnt/sdb1 on /tmp/.cemnt/mnt_sdb1 type ext3 (rw,nosuid,nodev,noexec,noatime,data=ordered)

It's an ext3 filesystem.

I see nothing out of the ordinary. Is there something I can use to check/repair the filesystem?

---------- Post updated at 04:38 PM ---------- Previous update was at 04:38 PM ----------

I tried this, but no avail.

Something is not Kosher here. /tmp is often not a physical device in Linux, it is the equivalent of a RAMDISK, or a filesystem in memory not on a disk. It is interesting that your problem filesystem seems to be mounted there.

Let's clarify:

As root - what is the exact output of this command?

df -k

df -k is the same as the df already posted.

df -ki

would give information about inodes.

Any chance to umount it (for fsck)?

fuser /tmp/.cemnt/sdb1

shows the processes that use it.

I have been able to solve the problem. I checked the drive with fsck and it seemed that a superblock of the drive was broken. Fsck needed a few hours to fix things, and now everything is working again.

So basically the Filesystem was all messed up, which caused not being able to delete stuff.

Thanks for your help people. This topic can be closed now.

Thank you for reporting back.

I read this thread a couple days ago and took a look at the busybox code. Before I could post, something more pressing came up and I forgot about it.

Only three paths in the Busybox rm implementation lead to a "cannot remove" message, and the gateway to each is a system call which should not return ENOSPC: lstat, rmdir, and unlink.

Now that you've gone and fixed the issue, we'll never know where the system's error handling went off the rails. Damn you.

By the way, Busybox isn't a Linux distro. It's just a single binary executable. If in the future you have another issue, please be more specific. At the very least, identify the distro, kernel, and busybox versions.

It's also useful to know when we're dealing with "ancient" software. While I was looking at busybox's code, I realized that you must be using quite an old version. Your "cannot remove" error message was changed to "can't remove" over 3.5 years ago (Nov 2009), and the last release to not include that change (1.15.3) goes back to Dec 2009.

Regards,
Alister

Hello Alister

Thank you for your helpful info. I'm still quite a newbie on this subject, learning as I go.

I do know that the busybox version on my server (a pogoplug V2) is ancient. I might update it one day, once I know how and if it is possible.

Thank you for your remarks.