"No space left on device" error when writing certain session filenames

Question

I have some filenames (session files) that can't be written ("No space left on device"). Other filenames (same directory) are fine. Disk is not full. The filesystem is ext3

PHP is giving an error when trying to create the files, but the error can be reproduced on the command line:

# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1 /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory # touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1 touch: cannot touch `/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1': No space left on device # touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0 # less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0 # ls -al /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1 ls: /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory 

Note the different filenames. There doesn't seem to be any pattern, but it does affect certain filenames only, and they don't seem to be ever able to be written to.

To possibly complicate things further, this is an OpenVZ server, but you can't write to/create those filenames from either the virtual server or the hardware node.

There are a lot of files in that directory (18,456,002 at the time of writing), but there is no inode problem, and the disk is definitely not full.

As requested, df outputs:

[root@web1 session_data]# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 2.7T 2.1T 495G 81% / /dev/root 2.7T 2.1T 495G 81% /path/to/session_data [root@web1 session_data]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/simfs 726761472 22843560 703917912 4% / /dev/root 726761472 22843560 703917912 4% /path/to/session_data 

(I've edited the path to the session data, but it really is mounted separately like that, to enable sharing of session data between multiple virtual servers)

Answer 1

Ext3 does hashing on file/dir-names.
With so many many files in one directory I wouldn't be surprised if you are running in hash-collision issues.
Having all the filenames of the same length and have the same structure makes the likelihood of a collision even worse.

If there is a collision the filename is effectively unusable, as you noticed.

I have no further experience with this sort of thing so I can't give you any more detailed advice.
(We ran into a similar problem in the office once and the resident Linux guru used some tunefs magic to make the problem go away. I never got the exact details what he did. Just that it had to do with hash-collisions on the filenames.)

You can apparently set the hashing behavior via tunefs. There is also a ext3 extension called "dirindex" that really helps dealing with directories containing so many files.