[solved] memory not freed by deleting files

[mimosa]

This seems to be a common problem, based on my searches:

Code: Select all

root[var]# du -hs /data
89G     /data
root[var]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        16G  8.3G  6.6G  56% /
/dev/sda3        94G   89G  100M 100% /data
tmpfs          1000M     0 1000M   0% /dev/shm

However, from what I found, this might be caused by a running process using the deleted files, yet rebooting -F to force a filesystem check didn't work. I also tried detaching screen from everything and wiping dead sessions.

It happened because my bittorrent client apparently filled the partition where it keeps its files (though this was unexpected and I have my doubts) so I burned some of them to DVD with Brasero and deleted those. Two reboots later df still thinks 94G - 89G = 100M!

I also read (probably with reference to some other distro) something about a five percent reserve space only accessible to root, so if root filled it even deleting 5GB from 100GB wouldn't be enough for normal users to see any free space. But even if Salix does something like this, I wasn't doing anything as root. As far as I know, the only process writing data to the partition in question was rtorrent. Brasero's temporary files must go somewhere under /

[Shador]

Maybe some sort of fragmentation? What fs is this?

[mimosa]

According to /etc/fstab, it's ext4 on that partition.

I also checked for files ending foo~ in the directory I deleted from - these sometimes seem to appear, but not in this case. The deleted files were about twenty of 130MB or so each, which also doesn't account for the full shortfall in available memory.

EDIT

Deleting an old VM freed up the corresponding space, but still leaving the same shortfall:

Code: Select all

vanilla[VBox]$ rm -r Openbox/
vanilla[VBox]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        16G  8.3G  6.6G  56% /
/dev/sda3        94G   87G  2.3G  98% /data
tmpfs          1000M     0 1000M   0% /dev/shm

[Shador]

Gotta be some sort of fs metadata, because that usually doesn't show up as used space but isn't counted as available either. Somehow you must have caused some sort of fragmentation or similar by filling the fs up, which can easily happen on full filesystems. What does e2fsck say? You could backup ypur whole data recreate the fs and replay it. That's probably the easiest solution.

[mimosa]

It's been running fairly full for ages, so what you say makes sense - and I like the simplicity of your solution. Of course, there's no rush!

However, I'm curious as to whether this might be the result of mismanagement on my part. Surely a modern Linux fs should be able to run pretty much indefinitely without accumulating five gigs of misfiled paperwork? And *suddenly*, too - whatever was there didn't stop rtorrent till it ran out of space, and then deleting 1-2 GB failed to free it up. Salix is not Windows.

[Shador]

Surely a modern Linux fs should be able to run pretty much indefinitely without accumulating five gigs of misfiled paperwork?

Unfortunately this is not always true. A filled up filesystem is always a bad filesystem and easily prone to cause bad performance and space usage. There's not much a fs dev can do here anyway except than automatically cleaning up the mess afterwards when there's enough space again. But that behaviour has major disadvantages too (eats io/ressources, can break bootloaders which make assumptions like lilo, ...) Different fs might perform in such corner cases different though.
The problem when the filesystem gets very full can be the fragmentation. Only few space is still available and it's likely not continous. So you have very many, very small gaps of free space where to put files. So the only choice left for the fs is to split the files in very many parts which fit into all those small gaps, which understandably creates some overhead and might create even smaller gaps.

Anyway it would be interesting to see some numbers regarding fragmentation e.g. as provided by e2fsck. To see whether this is the actual cause.
The output of this command could be interesting too:

Code: Select all

dumpe2fs /dev/root | egrep 'reserved|size|block count' -i

Anyway by default 5% are reserved so that could very well explain the numbers you're seeing too (0.05*94 = 4.7). But without more numbers there's no making sense of this numbers, just guessing.

[mimosa]

Thanks Shador! What you say about the five percent sort of makes sense, except that I *was* able to use that space until I wasn't any more, and also, I don't think df normally shows the discrepancy.

I didn't quite believe it about the five percent when I came across this information while researching the problem.

It's perhaps worth mentioning that the way my bittorrent client is set up, it writes the data to the data partition via a symlink in ~/

Code: Select all

root[vanilla]# ls -l | grep torrents
drwxr-xr-x  2 vanilla users    4096 Dec 11 16:31 new_torrents/
drwxr-xr-x  2 vanilla users    4096 Oct 19 18:29 old_torrents/
lrwxrwxrwx  1 vanilla users      33 Dec  8 14:21 torrents -> /data/mimosa/incomplete_torrents/

Here's some output from e2fsck:

Code: Select all

root[vanilla]# umount /data
root[vanilla]# e2fsck -fv /dev/sda3
e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

   28364 inodes used (0.45%)
     339 non-contiguous files (1.2%)
      13 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
         Extent depth histogram: 28186/155
23473140 blocks used (94.18%)
       0 bad blocks
       7 large files

   26071 regular files
    2271 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
      12 symbolic links (12 fast symbolic links)
       1 socket
--------
   28355 files

Here's what you asked for:

Code: Select all

root[vanilla]# dumpe2fs /dev/root | egrep 'reserved|size|block count' -i
dumpe2fs 1.41.14 (22-Dec-2010)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Block count:              4160835
Reserved block count:     208041
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      443
Flex block group size:    16
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal size:             128M
  Reserved GDT blocks at 2-444
  Reserved GDT blocks at 32770-33212

  Reserved GDT blocks at 98306-98748
  Reserved GDT blocks at 163842-164284
  Reserved GDT blocks at 229378-229820
  Reserved GDT blocks at 294914-295356
  Reserved GDT blocks at 819202-819644
  Reserved GDT blocks at 884738-885180
  Reserved GDT blocks at 1605634-1606076
  Reserved GDT blocks at 2654210-2654652
  Reserved GDT blocks at 4096002-4096444

And this is the same command but run on the data partition in question:

Code: Select all

root[vanilla]# dumpe2fs /dev/sda3 | egrep 'reserved|size|block count' -i
dumpe2fs 1.41.14 (22-Dec-2010)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Block count:              24922839
Reserved block count:     1246140
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1018
Flex block group size:    16
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal size:             128M
  Reserved GDT blocks at 7-1024
  Reserved GDT blocks at 32775-33792
  Reserved GDT blocks at 98311-99328
  Reserved GDT blocks at 163847-164864
  Reserved GDT blocks at 229383-230400
  Reserved GDT blocks at 294919-295936
  Reserved GDT blocks at 819207-820224
  Reserved GDT blocks at 884743-885760
  Reserved GDT blocks at 1605639-1606656
  Reserved GDT blocks at 2654215-2655232
  Reserved GDT blocks at 4096007-4097024
  Reserved GDT blocks at 7962631-7963648
  Reserved GDT blocks at 11239431-11240448
  Reserved GDT blocks at 20480007-20481024
  Reserved GDT blocks at 23887879-23888896
root[vanilla]# 

[Shador]

Thanks Shador! What you say about the five percent sort of makes sense, except that I *was* able to use that space until I wasn't any more, and also, I don't think df normally shows the discrepancy.

I didn't quite believe it about the five percent when I came across this information while researching the problem.

Yes df does.
Here I'm getting this for example:

Code: Select all

/dev/sda6              297G    214G   69G   76% /home

So exactly the same:

Code: Select all

>>> 0.05*297
14.850000000000001
>>> 214+69+_
297.85000000000002
>>> 214+69
283

Some calculation for your data partition:

Code: Select all

>>> 1246140/24922839.0
0.049999921758512346    # --> i.e. 5% reserved
>>> 1246140*4096.0/1024**3
4.7536468505859375    # --> given the blocksize this are ~4.75GB

so this is quite exactly what you're seeing and fragmentation is not high either. You can change the percentage of reserved blocks with tune2fs:

Code: Select all

tune2fs -m 1 /dev/sda3

On a non-root/data-only partition settings this value low or to zero should be no serious problem. The man page of tune2fs also includes an explanation as to why this feature is there. Also have a look here: https://www.walkernews.net/2007/02/28/t ... isk-space/

[mimosa]

So the puzzling thing is why I was allowed to "overfill" the space ... but it doesn't really matter. As they say, it's an academic question (though I hate it when they say that).

The lesson seems to be - don't run filesystems bursting at the seams. I will bear this in mind during ongoing rationalisation and backups.

Thanks Shador!

EDIT

For the record:

Code: Select all

-m reserved-blocks-percentage
              Set  the  percentage  of  the filesystem which may only be allocated by privileged processes.   Reserving some number of filesystem blocks for use by privileged processes is done to avoid filesystem fragmentation, and to allow system daemons, such as syslogd(8), to continue to function correctly after  non-privileged  processes are prevented from writing to the filesystem.  Normally, the default percentage of reserved blocks is 5%.

(From the man page Shador mentioned)

And indeed, also for the record, it works:

Code: Select all

root[vanilla]# umount /data
root[vanilla]# tune2fs -m 0 /dev/sda3
tune2fs 1.41.14 (22-Dec-2010)
Setting reserved blocks percentage to 0% (0 blocks)
root[vanilla]# mount -a
mount: devpts already mounted or /dev/pts busy
root[vanilla]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        16G  8.3G  6.6G  56% /
tmpfs          1000M     0 1000M   0% /dev/shm
/dev/sda3        94G   89G  5.6G  95% /data
root[vanilla]#