Behemoths periodically run out of inodes #1633
Description
Spinning off the discussion from scala/scala-dev#732 (comment) into a new ticket
Indeed it looks like inodes is more likely the issue than actual disk space. On behemoth-1:
admin@ip-172-31-2-3:~$ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvdj 25M 25M 816K 97% /home/jenkins while disk space looks fine
/dev/xvdj 393G 244G 130G 66% /home/jenkins The community build workspaces have huge numbers of files and directories. For example, for "scala-2.13.x-jdk11-integrate-community-build" there are currently 103 extraction directories
admin@ip-172-31-2-3:~$ ls /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction | wc -l 103 A single one of those has > 200k inodes:
admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 | wc -l 207593 Looking at things a bit, it seems we could save > 40% of inodes by not pulling in all the git refs to pull requests. They look similar to this:
/home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658/projects/93deaed81507c97b97bdf01b44a6723b14827dc1/.git/refs/pull/110 Some directory counting
admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type d | wc -l 80892 admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type d | grep '\/pull\/' | wc -l 43463 Looking at files in the extraction, again a large number of git refs corresponding to pull requests
admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type f | wc -l 126693 admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type f | grep -E 'pull\/[0-9]+\/head' | wc -l 43463 @SethTisue do you think we can do something about these git refs to pull requests?