DEV Community

Guixing Bai
Guixing Bai

Posted on

Speed up Jenkins with npm build

#jenkins #speedup #ci

Our team have many projects delivered by Jenkins. Every component need to be assembled by webpack which write in React.js or Vue.js.

Each build will spend 5 or 10 minutes to finish, and most time spend on npm install. But we all know that if package.json were not change, node_modules directory should not change as well.

So that we checksum package.json with MD5, and package whole node_modules directory as cache. If the hash string does not changed, which implied nothing changed, just unpack cached node_modules directory and then execute npm install, all things done.

Here is a simple example

PKG_SUM=$(md5sum package.json | cut -d\  -f 1) NPM_TARBALL=node_modules-${PKG_SUM}.tgz NPM_TARBALL_MD5SUM=${NPM_TARBALL}.md5sum NPM_TARBALL_CACHE=${HOME}/.cache/npmtarball S3_NPM_TARBALL=s3://deployment/npmtarball [[ ! -e $NPM_TARBALL_CACHE ]] && mkdir -p $NPM_TARBALL_CACHE function downloadNpmTarball(){ pushd $NPM_TARBALL_CACHE if [ ! -f ${NPM_TARBALL} ];then s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL} ${NPM_TARBALL} s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL_MD5SUM} ${NPM_TARBALL_MD5SUM} md5sum -c ${NPM_TARBALL_MD5SUM} || rm -f ${NPM_TARBALL} ${NPM_TARBALL_MD5SUM} fi popd } function checkNpmMod() { downloadNpmTarball TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL} [[ -f $TARBALL ]] && tar xzf $TARBALL } function uploadNpmMod() { TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL} if [ ! -f ${TARBALL} ];then if [ -d node_modules ];then tar zcf ${TARBALL} node_modules || return 1 fi pushd $NPM_TARBALL_CACHE md5sum $NPM_TARBALL > ${NPM_TARBALL_MD5SUM} s3cmd put $NPM_TARBALL $S3_NPM_TARBALL/$NPM_TARBALL s3cmd put $NPM_TARBALL_MD5SUM $S3_NPM_TARBALL/$NPM_TARBALL_MD5SUM popd fi } checkNpmMod npm install uploadNpmMod
Enter fullscreen mode Exit fullscreen mode

Top comments (6)

Collapse
 
vitaliidasaev profile image
Vitalii Dasaev •

I created such script to check md5sum of package.json in Jenkins:

stage('NPM Build') { steps { sh ''' node -v && npm -v ''' // rm -rf node_modules sh ''' CACHE_FOLDER=${HOME}/.cache/md5 echo "EXECUTOR_NUMBER: ${EXECUTOR_NUMBER}" MD5_FILE_NAME=package-json_${EXECUTOR_NUMBER}.md5sum [ -d ${CACHE_FOLDER} ] || mkdir -p ${CACHE_FOLDER} ls ${CACHE_FOLDER} if [ -f ${CACHE_FOLDER}/${MD5_FILE_NAME} ];then cp ${CACHE_FOLDER}/${MD5_FILE_NAME} ${MD5_FILE_NAME} md5sum package.json cat ${MD5_FILE_NAME} md5sum -c ${MD5_FILE_NAME} || npm ci else echo "No md5sum backup" npm ci fi echo "create new md5sum backup" md5sum package.json md5sum package.json > ${MD5_FILE_NAME} cp ${MD5_FILE_NAME} ${CACHE_FOLDER} ''' sh ''' npm run ngcc ''' sh ''' npm run build ''' } }
Enter fullscreen mode Exit fullscreen mode
Collapse
 
ttodua profile image
T. Todua •

For the sake of whatever, please create a Jenkins plugin which will facilitate life of hundreds of others. it's untold overkill everytime doing npm install such unnecessarily

Collapse
 
guilhermejcgois profile image
Gois •

How much fast is to cache on S3? Download, unzip, zip and upload time... My team have a similar problem due to npm install time

Collapse
 
khsing profile image
Guixing Bai •

If you are using aws ec2, it will be very fast.

Collapse
 
guilhermejcgois profile image
Gois •

I wrote a script to get MD5 only from dependencies and devDependencies, cause we do a patch bump on every merge to develop branch (we work with master and develop in a fork flow). Here is the gist.

Thread Thread
 
khsing profile image
Guixing Bai •

Great!