JsonAnalyzer is taking several hours to add/index large JSON files. #2118
Description
OS : Ubuntu 12.04.5 LTS Opengrok Version: 1.1-rc26 Tomcat Version : 8.5.31 Java : Oracle JRE/JDK 8u172 Ctags : Universal Ctags 0.0.0(6418097) - Based on Exuberant Ctags 5.8 Hardware : Intel(R) Xeon(R) CPU E5-1603 0 @ 2.80GHz with 16GB RAM Project : AOSP JsonAnalyzer is taking several hours to add/index large JSON files.
From Opengrok index log:
INFO: Add: /external/chromium-trace/trace-viewer/tracing/test_data/tall_trace.json (JsonAnalyzer) **May 19, 2018 7:38:19 PM** org.opensolaris.opengrok.index.DefaultIndexChangedListener fileAdd INFO: Add: /external/chromium-trace/trace-viewer/tracing/test_data/tcmalloc_multi_renderer.json (JsonAnalyzer) **May 19, 2018 9:22:03 PM** org.opensolaris.opengrok.index.DefaultIndexChangedListener fileAdd There are several large JSON files in this directory and indexing stalls for several hours.
There are around 6 threads running for indexing at this stage (using 26.9% memory):
2 threads are using around 80% CPU.
4 threads mostly stay near 20% CPU usage.
Around 22-26 threads under indexing parent process are sleeping.