add parallel phpmd processing and polish #2
Conversation
- use forking to process files in parallel (need for large code bases) - add support for specifing rulesets in the config
There was a problem hiding this comment.
Where do result files get stored? (We'll need to ensure it's within /tmp)
There was a problem hiding this comment.
Just saw the reference to sys_get_temp_dir, so this should be OK.
| Hi @ablyler -- Thanks for this. I think this is going to be a big improvement. I added some line notes for questions/suggestions, but this seems really close. |
- we can use it now that it has php-pcntl support
| @brynary everything should be all fixed up. also just changed the base back to alpine linux, now that they accepted my php-pcntl addition. :-) |
| @ablyler looks great. Ready to merge, just one more thing... 20 forks processing files simultaneously is probably too many (which would cause a slowdown due to unnecessary CPU switching). For our CodeClimate.com system, we'll allocate 2 CPUs to this, so we should probably run with a concurrency of about 2 or 3. (We can allow this to be overridden with an environment variable for local usage and on-premise deployments with different resource constraints as well, in the future.) |
- did some performance testing after @brynary comment in GitHub and found that 3 performs best on my large code bases
| @brynary good call, did some testing, 3 appears to be a good setting. |
add parallel phpmd processing and polish
| Merged. Thanks for all the help on this, @ablyler! |
2 participants
Note: I switched to from Alpine to Ubuntu for now because Alpine doesn't have php-pcntl support, which was needed for forking. I submitted a patch to Alpine to get php-pcntl support added.