Skip to content

HDFS-17853. Support to make dfs.namenode.fs-limits.max-directory-items reconfigurable #8064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from
Open

HDFS-17853. Support to make dfs.namenode.fs-limits.max-directory-items reconfigurable #8064

wants to merge 2 commits into from
+64 −8

Conversation

@lfxy
Copy link
Contributor

@lfxy lfxy commented Nov 7, 2025

Sometimes, certain directories—such as temporary library directories—contain too many subdirectories and files, exceeding the limit defined by the dfs.namenode.fs-limits.max-directory-items configuration. This causes many jobs to fail.

To quickly restore job execution, we need to temporarily adjust this configuration. However, since it currently requires to restart NameNode to take effect, we need to make it dynamically reconfigurable without restarting the NameNode.
@hadoop-yetus
Copy link

hadoop-yetus commented Nov 10, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 18s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 43m 44s trunk passed
+1 💚 compile 1m 24s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 1m 28s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 57s trunk passed
+1 💚 mvnsite 1m 36s trunk passed
+1 💚 javadoc 1m 11s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 1m 11s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 4m 8s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html hadoop-hdfs-project/hadoop-hdfs in trunk has 291 extant spotbugs warnings.
+1 💚 shadedclient 34m 41s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 24s the patch passed
+1 💚 compile 1m 16s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javac 1m 16s the patch passed
+1 💚 compile 1m 21s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 javac 1m 21s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 52s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 109 unchanged - 1 fixed = 110 total (was 110)
+1 💚 mvnsite 1m 27s the patch passed
-1 ❌ javadoc 1m 0s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 2 new + 9998 unchanged - 2 fixed = 10000 total (was 10000)
-1 ❌ javadoc 1m 4s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 2 new + 9671 unchanged - 0 fixed = 9673 total (was 9671)
-1 ❌ spotbugs 4m 9s /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html hadoop-hdfs-project/hadoop-hdfs generated 2 new + 291 unchanged - 0 fixed = 293 total (was 291)
+1 💚 shadedclient 34m 4s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 256m 59s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 52s The patch does not generate ASF License warnings.
393m 13s
Reason Tests
SpotBugs module:hadoop-hdfs-project/hadoop-hdfs
Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.hdfs.server.namenode.FSDirectory.setMaxDirItems(int) At FSDirectory.java:org.apache.hadoop.hdfs.server.namenode.FSDirectory.setMaxDirItems(int) At FSDirectory.java:[line 589]
Unread field:field be static? At FSDirectory.java:[line 220]
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/2/artifact/out/Dockerfile
GITHUB PR #8064
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3c18d8640dd3 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2925348
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/2/testReport/
Max. process+thread count 2421 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/2/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.
@lfxy lfxy force-pushed the feature/HDFS-17853 branch from 2925348 to da91c0d Compare November 10, 2025 10:01
@hadoop-yetus
Copy link

hadoop-yetus commented Nov 10, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 20m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 41m 24s trunk passed
+1 💚 compile 1m 23s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 1m 27s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 57s trunk passed
+1 💚 mvnsite 1m 38s trunk passed
+1 💚 javadoc 1m 9s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 1m 10s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 4m 8s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html hadoop-hdfs-project/hadoop-hdfs in trunk has 291 extant spotbugs warnings.
+1 💚 shadedclient 35m 2s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 48s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 49s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 49s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ compile 0m 50s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 50s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 0m 53s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 109 unchanged - 1 fixed = 109 total (was 110)
-1 ❌ mvnsite 0m 52s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ javadoc 0m 37s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javadoc 0m 38s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ spotbugs 0m 49s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ shadedclient 14m 5s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 51s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
125m 50s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/3/artifact/out/Dockerfile
GITHUB PR #8064
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 41356d1dd994 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / da91c0d
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/3/testReport/
Max. process+thread count 568 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/3/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.
@lfxy lfxy force-pushed the feature/HDFS-17853 branch from da91c0d to 4ec30a8 Compare November 10, 2025 12:35
@hadoop-yetus
Copy link

hadoop-yetus commented Nov 10, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 39s trunk passed
+1 💚 compile 1m 24s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 1m 26s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 58s trunk passed
+1 💚 mvnsite 1m 38s trunk passed
+1 💚 javadoc 1m 10s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 1m 9s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 4m 3s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html hadoop-hdfs-project/hadoop-hdfs in trunk has 291 extant spotbugs warnings.
+1 💚 shadedclient 33m 53s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 21s the patch passed
+1 💚 compile 1m 17s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javac 1m 17s the patch passed
+1 💚 compile 1m 21s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 javac 1m 21s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 52s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 109 unchanged - 1 fixed = 109 total (was 110)
+1 💚 mvnsite 1m 27s the patch passed
-1 ❌ javadoc 0m 59s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 2 new + 9998 unchanged - 2 fixed = 10000 total (was 10000)
-1 ❌ javadoc 1m 3s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 2 new + 9671 unchanged - 0 fixed = 9673 total (was 9671)
-1 ❌ spotbugs 4m 8s /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html hadoop-hdfs-project/hadoop-hdfs generated 2 new + 291 unchanged - 0 fixed = 293 total (was 291)
+1 💚 shadedclient 34m 20s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 259m 43s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 53s The patch does not generate ASF License warnings.
391m 50s
Reason Tests
SpotBugs module:hadoop-hdfs-project/hadoop-hdfs
Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.hdfs.server.namenode.FSDirectory.setMaxDirItems(int) At FSDirectory.java:org.apache.hadoop.hdfs.server.namenode.FSDirectory.setMaxDirItems(int) At FSDirectory.java:[line 589]
Unread field:field be static? At FSDirectory.java:[line 220]
Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/4/artifact/out/Dockerfile
GITHUB PR #8064
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux f8e3b3182b07 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4ec30a8
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/4/testReport/
Max. process+thread count 2337 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8064/4/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.
@lfxy lfxy force-pushed the feature/HDFS-17853 branch from 4ec30a8 to 405e7f3 Compare November 11, 2025 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

HDFS trunk

2 participants

@lfxy @hadoop-yetus