Check registry status code for 500 errors

[jmorrell]

Summary

Over the last week I received several reports from Heroku users using Yarn to install dependency where the installation would fail. Generally the errors would be one of these two:

Error: Received malformed response from registry for undefined. The registry may be down. 

Error: https://registry.yarnpkg.com/tmpl/-/tmpl-1.0.4.tgz: Integrity check failed for "tmpl" (computed integrity doesn't match our records, got "sha1-JRR4x9Dq/iPzyx1+yQY+TRa9x6c=") 

The next build would generally succeed, but then a different module would fail an integrity check. I deployed a modified version of Yarn that would console log the response when one of these errors would occur and found that in these cases, CloudFlare was returning a 500 response with an HTML error message as the body. This is not common, but I was able to recreate it by curling a particular tarball in the registry in a loop.

Looking at the malformed response code, it would receive the HTML error page, check for parameters, and then fail with the error above.

This PR adds an explicit check for the case that the returned status code is 500 or greater without relying on a JSON response with an error field.

Test plan

yarn test 

[jmorrell]

Just realized this PR is functionally similar to #6383

[jmorrell]

This likely resolves these issues:

#6352
#6347

[Ojisama]

Hey @jmorrell ! I'm the author of #6383 and I was wondering why we should stop at status >= 500? 😮

Surely a 4xx should make any command fail, right?

[jmorrell]

@arcanis It looks like one of the CI tests failed with one of these exact errors? Maybe this doesn't fix the integrity issues 🤔

 warning There appears to be trouble with our server. Retrying... error https://registry.yarnpkg.com/url-loader/-/url-loader-0.5.9.tgz: Integrity check failed for "url-loader" (computed integrity doesn't match our records, got "sha512-uGYPY/MiYqyLMx8iUF2wK9s73uh5JEMnoVz94bV9UhYtTA9Jz8WA8TpngOk0C51HcuyakwiA7vRc3AqUB6xleQ==") 

[jmorrell]

@Ojisama agreed, I'll see about adding it

[arcanis]

So the thing is that NPM apparently retries the requests on 408 and 5xx until they work. That's why they break with Yarn: we don't seem to correctly retry them in every case.

@jmorrell Do you think it would be possible to update your PR to ensure we retry as well on these errors? I'm not fan of retrying, but since it breaks so many installs... I wonder what's the reason behind the server errors, though 😞

[arcanis]

I refactored part of the code to support this retry logic - please check #6413 if you have some time. I'm going to close this issue and the other to keep discussions on a single thread 👍