Where is the "Link step" cached?

[pongells]

I am setting up a BitBucket Pipeline with the latest Yarn 3.x and a NX monorepo setup.

I am installing with global cache:

yarn config set enableGlobalCache true yarn install --immutable 

and in the Pipeline, I am storing the cache as follows:

definitions: caches: yarn3: /root/.yarn/berry 

When the Pipeline runs, the cache is resolved and the "Fetch step" takes few seconds. The "Link step" however, takes 20 seconds:

➤ YN0000: ┌ Link step ➤ YN0007: │ core-js@npm:3.25.5 must be built because it never has been before or the last one failed ➤ YN0007: │ cypress@npm:10.10.0 must be built because it never has been before or the last one failed ➤ YN0007: │ @parcel/watcher@npm:2.0.4 must be built because it never has been before or the last one failed ➤ YN0007: │ core-js-pure@npm:3.25.5 must be built because it never has been before or the last one failed ➤ YN0000: │ cypress@npm:10.10.0 STDOUT Installing Cypress (version: 10.10.0) ➤ YN0000: │ cypress@npm:10.10.0 STDOUT ➤ YN0000: │ cypress@npm:10.10.0 STDOUT [STARTED] Task without title. ➤ YN0000: │ cypress@npm:10.10.0 STDOUT [SUCCESS] Task without title. ➤ YN0000: │ cypress@npm:10.10.0 STDOUT [STARTED] Task without title. ➤ YN0000: │ cypress@npm:10.10.0 STDOUT [SUCCESS] Task without title. ➤ YN0000: │ cypress@npm:10.10.0 STDOUT [STARTED] Task without title. ➤ YN0000: │ cypress@npm:10.10.0 STDOUT [SUCCESS] Task without title. ➤ YN0000: │ cypress@npm:10.10.0 STDOUT ➤ YN0000: │ cypress@npm:10.10.0 STDOUT You can now open Cypress by running: node_modules/.bin/cypress open ➤ YN0000: │ cypress@npm:10.10.0 STDOUT ➤ YN0000: │ cypress@npm:10.10.0 STDOUT https://on.cypress.io/installing-cypress ➤ YN0000: │ cypress@npm:10.10.0 STDOUT ➤ YN0007: │ nx@npm:14.8.5 [440b5] must be built because it never has been before or the last one failed ➤ YN0000: │ nx@npm:14.8.5 [440b5] STDOUT ➤ YN0000: │ nx@npm:14.8.5 [440b5] STDOUT > NX Daemon Server - Stopped ➤ YN0000: │ nx@npm:14.8.5 [440b5] STDOUT ➤ YN0007: │ REPO@workspace:. must be built because it never has been before or the last one failed ➤ YN0000: │ REPO@workspace:. STDOUT husky - Git hooks installed ➤ YN0000: └ Completed in 30s 525ms ➤ YN0000: Done with warnings in 34s 719ms 

is it possible to cache this step?

[av]

When debugging, it appears that .yarn/install-state.gz is used as an index of what was built and what wasn't:

package.json

{ "name": "yarn-link-cache", "version": "1.0.0", "main": "index.js", "license": "MIT", "packageManager": "yarn@3.6.4", "dependencies": { "cpu-features": "^0.0.9" } }

Initial yarn install

➤ YN0000: ┌ Resolution step ➤ YN0032: │ cpu-features@npm:0.0.9: Implicit dependencies on node-gyp are discouraged ➤ YN0032: │ nan@npm:2.18.0: Implicit dependencies on node-gyp are discouraged ➤ YN0000: └ Completed in 3s 352ms ➤ YN0000: ┌ Fetch step ➤ YN0013: │ wide-align@npm:1.1.5 can't be found in the cache and will be fetched from the remote registry ➤ YN0013: │ wrap-ansi@npm:7.0.0 can't be found in the cache and will be fetched from the remote registry ➤ YN0013: │ wrap-ansi@npm:8.1.0 can't be found in the cache and will be fetched from the remote registry ➤ YN0013: │ wrappy@npm:1.0.2 can't be found in the cache and will be fetched from the remote registry ➤ YN0013: │ yallist@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry ➤ YN0000: └ Completed in 0s 519ms ➤ YN0000: ┌ Link step ➤ YN0007: │ cpu-features@npm:0.0.9 must be built because it never has been before or the last one failed ➤ YN0000: └ Completed in 1s 930ms ➤ YN0000: Done with warnings in 5s 821ms

Subsequent yarn install:

➤ YN0000: ┌ Resolution step ➤ YN0000: └ Completed ➤ YN0000: ┌ Fetch step ➤ YN0000: └ Completed ➤ YN0000: ┌ Link step ➤ YN0000: └ Completed ➤ YN0000: Done in 0s 24ms

After rm -rf ./.yarn/install-state.gz:

➤ YN0000: ┌ Resolution step ➤ YN0000: └ Completed ➤ YN0000: ┌ Fetch step ➤ YN0000: └ Completed ➤ YN0000: ┌ Link step ➤ YN0007: │ cpu-features@npm:0.0.9 must be built because it never has been before or the last one failed ➤ YN0000: └ Completed in 1s 887ms ➤ YN0000: Done in 1s 950ms

However, it doesn't look like the are assets cached in that file, it's way too small.

[hasan-aa]

I've been dealing with the same issue on aws amplify for canvas package
@av's answer helped me resolve the issue.

Just want to add that, I don't think yarn caches the actual result of the build. The package itself has the build script and determines where the build output goes. In my case it's node_modules/canvas/build folder.

[shinebayar-g]

Removing .yarn/install-state.gz makes very little difference. It seems node_modules/.yarn-state.yml is also relevant.

• With existing .yarn/install-state.gz and node_modules

❯ yarn ➤ YN0000: · Yarn 4.7.0 ➤ YN0000: ┌ Resolution step ➤ YN0000: └ Completed in 0s 259ms ➤ YN0000: ┌ Post-resolution validation ➤ YN0000: └ Completed ➤ YN0000: ┌ Fetch step ➤ YN0000: └ Completed in 1s 263ms ➤ YN0000: ┌ Link step ➤ YN0000: └ Completed in 0s 218ms ➤ YN0000: · Done with warnings in 1s 849ms

• Removed .yarn/install-state.gz and kept node_modules

❯ yarn ➤ YN0000: · Yarn 4.7.0 ➤ YN0000: ┌ Resolution step ➤ YN0000: └ Completed in 0s 258ms ➤ YN0000: ┌ Post-resolution validation ➤ YN0000: └ Completed ➤ YN0000: ┌ Fetch step ➤ YN0000: └ Completed in 1s 274ms ➤ YN0000: ┌ Link step ➤ YN0007: │ nx@npm:20.6.1 [5c7ae] must be built because it never has been before or the last one failed ➤ YN0007: │ esbuild@npm:0.25.1 must be built because it never has been before or the last one failed ➤ YN0007: │ @cdktf/node-pty-prebuilt-multiarch@npm:0.10.2 must be built because it never has been before or the last one failed ➤ YN0007: │ highlight.js@npm:9.18.5 must be built because it never has been before or the last one failed ➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed ➤ YN0000: └ Completed in 3s 158ms ➤ YN0000: · Done with warnings in 4s 802ms

• Removed node_modules/.yarn-state.yml and kept node_modules and .yarn/install-state.gz

❯ yarn ➤ YN0000: · Yarn 4.7.0 ➤ YN0000: ┌ Resolution step ➤ YN0000: └ Completed in 0s 264ms ➤ YN0000: ┌ Post-resolution validation ➤ YN0000: └ Completed ➤ YN0000: ┌ Fetch step ➤ YN0000: └ Completed in 1s 256ms ➤ YN0000: ┌ Link step ➤ YN0007: │ nx@npm:20.6.1 [5c7ae] must be built because it never has been before or the last one failed ➤ YN0007: │ esbuild@npm:0.25.1 must be built because it never has been before or the last one failed ➤ YN0007: │ @cdktf/node-pty-prebuilt-multiarch@npm:0.10.2 must be built because it never has been before or the last one failed ➤ YN0007: │ highlight.js@npm:9.18.5 must be built because it never has been before or the last one failed ➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed ➤ YN0000: └ Completed in 19s 195ms ➤ YN0000: · Done with warnings in 20s 831ms

Without node_modules/.yarn-state.yml it adds 20 seconds. Should I cache this file in CI?