3

I had a folder , with 4+ million files in it. I managed to delete all files , in course of a week , using "find . -delete" and "kill -9" after a minute as it locks the system otherwise.

now the folder is empty. When i go to that folder and do "ls" , all is fine. I just see that the folder is empty.

But when i do "find foo/" , it locks the system again , just like the time when i used to delete those millions of files.

What can be the reason ?

Improve this question
5
  • 3
    The directory is probably huge, even though it doesn't contain any entries. Why ls is fast and find is not, however, is a mystery to me. Anyway it may be advisable to delete and recreate the directory so that it's small again. Commented Oct 13, 2014 at 9:46
  • Are there any dot files or dirs? ls -a foo Commented Oct 13, 2014 at 10:14
  • @Cyrus no. there is only . and .. , as expected Commented Oct 13, 2014 at 11:05
  • I would take a look at dmesg, because such long disk operations are usually caused by faulty disks. Commented Oct 15, 2014 at 15:16
  • @wurtel the directory was huge yeah , it head 89 mb of metadata even 1-2 weeks after all the delete process. There is anotehr file that i create and delete around 30.000 files every day , and it has a meta data of 2.9 mb Commented Oct 15, 2014 at 18:54

3 Answers 3

Reset to default
2

Your file system may have some corruption on it. Try running fsck -f over it (when it is unmounted), see if it finds any problems.

Improve this answer
1
  • it's all about the huge amount of metadata of the folder as it seems Commented Oct 15, 2014 at 7:16
1

Curious why you didn't just delete the whole directory and then recreate it:

rm -rf foo_dir mkdir foo_dir

Or delete the files without using find:

cd [path]/foo_dir rm -rf *

Perhaps deleting and recreating the directory would still help you.

Improve this answer
2
  • 1
    the * expansion , thus the "rm -rf *" or "rm -rf foo/" doesn't work when you have 4 million of files in a folder Commented Oct 14, 2014 at 15:21
  • Some more detailed information: When you do a rm -rf * the shell will see the asterix (*) and feed it to rm. In effect you do not start an rm -rf * but rm -rf file1 file2 file3 file4 .... This runs into problems when there is not enough space in buffers to list all those expanded filenames. Hence find with -delete or find with -exec {} + Commented Oct 14, 2014 at 16:30
1

I have found the answer by sharing this question on Google Plus. The answer by Axel Engeland:

That really depends on the file system used. Directories are supposed to store inode tables. If inode tables grow too large, another one is created and the tables are linked. So even if you removed all the files, there may still be a big amout of interlinked inode tables. if you rmdir and mkdir the directory again, performance should be better.

As you may have deleted all files but the last, the inode tables are still in place. ls -ld foo should show you the size of directory meta information. Consider:

# mkdir some_dir # ls -ld some_dir drwxr-xr-x 2 root root 4096 Oct 14 21:16 some_dir

So, the directory has 4096 bytes for meta data. ok. Let's make some files.

# pushd some_dir ; for i in {1..1000} ; do mktemp test.XXXXXXXXXXXXXX ; done ; popd

We just created 1000 files in some_dir.

# ls -ld some_dir drwxr-xr-x 2 root root 49152 Oct 14 21:16 some_dir

Oh wow, the directory now needs 49k für meta data. Let's remove the files.

# find some_dir/ -type f -delete # ls -ld some_dir drwxr-xr-x 2 root root 49152 Oct 14 21:19 some_dir

The files are gone. But still 48k meta data. A find will still have to go through all the meta data in the directory to... well... find something.

Improve this answer
6
  • This still doesn't explain why find is slower than ls Commented Oct 15, 2014 at 7:29
  • @artistoex How come ? find is slower because it is going through the inode metadata of the folder , which is huge , even if the folder is emptied Commented Oct 15, 2014 at 8:00
  • 1
    I don't think so. I've just skimmed over the gnu find and ls sources, and it basically does the same as ls does: opendir() followed by readdir() (see function xsavedir() in savedirinfo.c) Commented Oct 15, 2014 at 10:36
  • Besides that, even a couple of Million bytes of inode processing barely explain a weeks worth of computation. Commented Oct 15, 2014 at 10:38
  • there is no "a week worth of computation" . There is me "find . -delete" ing like 50k files in a minute and killing the process , whenever i find the server load low enough , in course of a week . Commented Oct 15, 2014 at 10:51

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.