8

My PHP sessions on my Debian webserver using Apache2 with mod_php seem to be failing randomly, saying that there’s no space to write them:

sudo tail -60 /var/log/apache2/error.log [Fri Jan 30 15:55:35 2015] [error] [client xxx.xxx.xxx.xxx] PHP Warning: session_start() [<a href='function.session-start'>function.session-start</a>]: open(/tmp/sess_555555555555555555, O_RDWR) failed: No space left on device (28) in /path/to-first-session-use/core/bootstrap.php on line 18

When I try to:

ls /tmp

It just hangs forever, so that’s bad.

But when I check free space, and check that inode usage is reasonable...

$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 150G 121G 22G 85% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw udev 10M 16K 10M 1% /dev tmpfs 2.0G 4.0K 2.0G 1% /dev/shm $ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 19922944 11143605 8779339 56% / tmpfs 513524 4 513520 1% /lib/init/rw udev 513524 135 513389 1% /dev tmpfs 513524 3 513521 1% /dev/shm

The numbers seem fine. Sure, 85% is more than I’d like, but it's not 99% or anything.

I was suspecting that it was a problem due to not rebooting the machine for 5 years and maybe the creation of a lot of small files but the inode info that I’m getting kinda contradicts this. Where should I investigate instead?

Edit:

ls -l /

drwxrwxrwt 4 root root 692M Feb 1 11:09 tmp/ drwxr-xr-x 10 root root 4.0K Jan 1 2013 usr/ drwxr-xr-x 14 root root 4.0K Oct 7 2010 var/ ...etc
Improve this question
8
  • Do you use Plesk? Commented Jan 30, 2015 at 21:35
  • What webserver do you use? Commented Jan 30, 2015 at 21:37
  • @utrecht This is on a Debian server. Commented Jan 30, 2015 at 21:42
  • Ok, but what webserver, e.g., apache2? Commented Jan 30, 2015 at 21:45
  • @utrecht Ah, yes, apache2 on this server. Commented Jan 30, 2015 at 21:46

3 Answers 3

Reset to default
8

It could be that the /tmp/ directory itself is filled with stale PHP sessions that are not getting cleaned up; meaning the source of the issues might be isolated to the /tmp/ directory itself. If that is the case I would just remove all /tmp/sess_* files. First, list all of the sess_* files like this:

ls -la /tmp/sess_*

Or you can get count with wc like this:

ls -la /tmp/sess_* | wc -l

Now once you get some confirmation there is some insane number of files there, go ahead and run this command to delete the /tmp/sess_* files:

sudo rm -rf /tmp/sess_*

And the ephemeral session files will be blown away.

But another brute force—but relatively safe—way to deal wit this is to blow away the /tmp directory itself, recreate the /tmp directory and reboot the server.

Since the /tmp directory is basically a coding holding pen for cached material, there is nothing valid that should be in there. So my best advice is to run the following command to remove & rebuidl the /tmp directory.

rm -rf /tmp && mkdir /tmp/ && chown root:root /tmp && chmod 1777 /tmp

Now that one liner is basically a list of shell commands connected by && that will first delete /tmp, recreate /tmp, change the ownership of /tmp back to root:root and then set proper permissions to the /tmp directory. If you wish you can run each command one by one if you feel safer doing it that way.

sudo rm -rf /tmp sudo mkdir /tmp sudo chown root:root /tmp sudo chmod 1777 /tmp

Once that is done, I would recommend rebooting the server. Things should be calm cleared up again.

Improve this answer
2
  • nods I'll try this approach sometime next week. Commented Feb 1, 2015 at 20:55
  • 1
    I wasn't able to delete /tmp directly, same as I wasn't able to ls /tmp, but I was able to mount the filesystem temporarily elsewhere and -then- it allowed me to delete & remake the /tmp directory. I just mv'ed the old directory to /tmp_damaged since I actually wasn't able to rm & delete it either. Commented Feb 11, 2015 at 23:33
3

Sometimes damaged file system can do effects like it - for example when directory /tmp is damaged. Or - when there is to much files.

For "quick" fix:

mv /tmp /tmp.xxx mkdir /tmp chmod a+rwxt /tmp

If that help - try reboot system and fsck root file system. If it's ok - just remove /tmp.xxx directory.

Another possibility is - when /tmp is "other" partition or tmpfs (seen on linux vservers) - but it's not show by df (because df get list of partitions from /etc/mtab file which sometimes is not correct). Try check disk space directly on tmp by command:

df /tmp df -i /tmp

Other option which usually helps with sessions - it using other session mechanism. If you have a lot of temporary sessions, which doesn't need to be very persistent - i would you recommend using memcache for session storing. Configuration it very simple - you must install php-memcache, memcached and then in php.conf configure:

session.save_handler = memcache session.save_path="tcp://server:port?persistent=1&weight=1&timeout=1&retry_interval=15"

Then - sessions will be stored in memcache up to defined size. over it - oldest will be automaticly removed.

Improve this answer
1

For me, changing fs.inotify.max_user_watches did the trick.

root@grostruc:/# service ssh restart Error: No space left on device root@grostruc:/# sysctl fs.inotify.max_user_watches fs.inotify.max_user_watches = 65536 root@grostruc:/# sysctl fs.inotify.max_user_watches=262144 fs.inotify.max_user_watches = 262144 root@grostruc:/# service ssh restart

Fix changed value in /etc/sysctl.conf

Improve this answer
3
  • 1
    That sounds pretty peculiar any clue why that happened? Commented Jul 17, 2015 at 15:45
  • Huh, crashplan started this issue for me and this solved it. Thanks! Does this have a significant performance change? I had just over 8k before and going up to 262k seems drastic but good enough for me Commented Aug 11, 2017 at 15:44
  • Update this did not fix it. Commented Sep 18, 2017 at 3:39

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.