Disk space monitoring with Zabbix while file descriptor/handle still open

Question

Recently I've faced situation, when "standard" disk space monitoring failed - Zabbix uses vfs.fs.size item for checking disk usage, which is documented and preconfigured in bundled templates.

The Problem:

When files are deleted, but file descriptor/handle is still open - disk might be filled, but Zabbix will report no usage in this case.

How can one monitor disk space in this case?

Answer 1

Here is a test i've done :

df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 15350768 14367976 809124 95% / 

I create a 600Mb file and open it with less :

dd if=/dev/zero of=test bs=1024k count=600 & less test [1] 1952 "test" may be a binary file. See it anyway? 600+0 records in 600+0 records out 629145600 bytes (629 MB) copied, 11.895 s, 52.9 MB/s [1]+ Done dd if=/dev/zero of=test bs=1024k count=600 

Now df show that more space is used :

df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 15350768 14981664 195436 99% / 

now if I remove my test file :

rm test 

df says :

df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 15350768 14981712 195388 99% / 

so the space is still in use, we can see it with lsof |grep deleted :

lsof |grep deleted|grep test less 1953 mick 4r REG 8,1 629145600 5407 /tmp/test (deleted) 

now if I stop less :

pkill less 

df show that the disk usage is free :

df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 15350768 14367584 809516 95% / 

I think your problem comes more from a bad intepretation of df due to percent reserved for root.

From the last df, total minus used minus available equals :

15350768-14367584-809516=173668 

So maybe your user has a disk full error, but root can still write to disk, and Zabbix will not see disk full due to percent reserved for root.

You can check and change this with tune2fs