Skip to content

Conversation

@andershermansen
Copy link
Contributor

@andershermansen andershermansen commented Mar 16, 2025

What?

The s3 storage adapter returns a 500 internal server error when a file is not found.
It's expected that it will return 404 when a file is not found.

Why?

The getObject function from aws s3 sdk does not return undefined when a blob is not found, but throws a NoSuchKey error: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-client-s3/Class/NoSuchKey/

How?

Check if exception thrown is of type NoSuchKey and return a 404 in that case.

Related discord discussion:
https://discord.com/channels/967097582721572934/1350826594062696539/1350826594062696539

@andershermansen
Copy link
Contributor Author

@denolfe Tagging you as you previously reviewed #10327 for storage-vercel-blob
This is the same fix for storage-s3

There is also PR with same fix for storage-azure #11734
And a PR with fix to storage-gcs #11746 thanks to @akhrarovsaid

@andershermansen
Copy link
Contributor Author

@denolfe I have now updated this PR with latest upstream changes and done proper testing of this PR. Would you please consider it?

To test I built the storage-s3 package using pack_to_dest.sh script and installed that tgz package in a repo of mine. Here are the results.

With stock 3.42.0

HTTP error response code is 500:
Skjermbilde 2025-06-10 kl 08 11 41

Log in console:

[08:11:34] ERROR: UnknownError err: { "type": "NoSuchKey", "message": "UnknownError", "stack": NoSuchKey: UnknownError at de_NoSuchKeyRes (webpack-internal:///(rsc)/./node_modules/.pnpm/@aws-sdk+client-s3@3.826.0/node_modules/@aws-sdk/client-s3/dist-es/protocols/Aws_restXml.js:4089:23) at de_CommandError (webpack-internal:///(rsc)/./node_modules/.pnpm/@aws-sdk+client-s3@3.826.0/node_modules/@aws-sdk/client-s3/dist-es/protocols/Aws_restXml.js:3988:25) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@smithy+middleware-serde@4.0.8/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.826.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@smithy+middleware-retry@4.1.12/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.826.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.826.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:137:14) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@aws-sdk+middleware-logger@3.821.0/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@payloadcms+storage-s3@3.42.0_@types+react@19.1.7_monaco-editor@0.52.2_next@15.3.3_@babel+cor_6bihwnozjgtq26sloalbr4fsp4/node_modules/@payloadcms/storage-s3/dist/staticHandler.js:53:22) at async getFileHandler (webpack-internal:///(rsc)/./node_modules/.pnpm/payload@3.42.0_graphql@16.11.0_typescript@5.8.3/node_modules/payload/dist/uploads/endpoints/getFile.js:43:30) at async handleEndpoints (webpack-internal:///(rsc)/./node_modules/.pnpm/payload@3.42.0_graphql@16.11.0_typescript@5.8.3/node_modules/payload/dist/utilities/handleEndpoints.js:179:26) at async eval (webpack-internal:///(rsc)/./node_modules/.pnpm/@payloadcms+next@3.42.0_@types+react@19.1.7_graphql@16.11.0_monaco-editor@0.52.2_next@15.3.3__cqhsyoiffs6scpjwy7n2nq4dkq/node_modules/@payloadcms/next/dist/routes/rest/index.js:27:20) at async AppRouteRouteModule.do (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:26:34112) at async AppRouteRouteModule.handle (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:26:41338) at async doRender (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/base-server.js:1518:42) at async DevServer.renderToResponseWithComponentsImpl (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/base-server.js:1920:28) at async DevServer.renderPageComponent (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/base-server.js:2408:24) at async DevServer.renderToResponseImpl (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/base-server.js:2445:32) at async DevServer.pipeImpl (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/base-server.js:1008:25) at async NextNodeServer.handleCatchallRenderRequest (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/next-server.js:305:17) at async DevServer.handleRequestImpl (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/base-server.js:900:17) at async /Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/dev/next-dev-server.js:371:20 at async Span.traceAsyncFn (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/trace/trace.js:157:20) at async DevServer.handleRequest (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/dev/next-dev-server.js:368:24) at async invokeRender (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/lib/router-server.js:237:21) at async handleRequest (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/lib/router-server.js:428:24) at async requestHandlerImpl (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/lib/router-server.js:452:13) at async Server.requestListener (/Users/anders/code/adsign/modena-cms/node_modules/.pnpm/next@15.3.3_@babel+core@7.27.4_react-dom@19.1.0_react@19.1.0__react@19.1.0_sass@1.77.4/node_modules/next/dist/server/lib/start-server.js:158:13) "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000d8c16b57a1da7e43-006847cc96-ebe0ec19-default", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "modena-cms-images", "RequestId": "tx00000d8c16b57a1da7e43-006847cc96-ebe0ec19-default", "HostId": "ebe0ec19-default-default" } GET /api/images/file/no_such_file.png 500 in 81ms 

With local storage-s3 package with this change

HTTP error response code is 404:
Skjermbilde 2025-06-10 kl 08 19 02

Log in console:

 GET /api/images/file/no_such_file.png 404 in 73ms 
@andershermansen
Copy link
Contributor Author

I of course testing the regular case of downloading images that did exist as well, and it's also working as expected with the change.

@paulpopus paulpopus changed the title fix(storage-s3): return 404 when file is not found fix(storage-s3): return 404 when file is not found instead of 500 Jun 10, 2025
Copy link
Contributor

@paulpopus paulpopus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's expand the int tests to check the returned error code and to make sure a file that does exist returns as expected

@andershermansen
Copy link
Contributor Author

Let's expand the int tests to check the returned error code and to make sure a file that does exist returns as expected

Test case added.

Here is the test case running on main showing it fails with 500 on current main:
Skjermbilde 2025-06-11 kl 07 40 14

@paulpopus paulpopus changed the title fix(storage-s3): return 404 when file is not found instead of 500 fix(storage-s3): return error status 404 when file is not found instead of 500 Jun 11, 2025
@paulpopus paulpopus enabled auto-merge (squash) June 11, 2025 10:57
@paulpopus paulpopus merged commit a19921d into payloadcms:main Jun 11, 2025
154 of 156 checks passed
@andershermansen andershermansen deleted the storage-s3-404 branch June 11, 2025 12:23
paulpopus pushed a commit that referenced this pull request Jun 11, 2025
<!-- Thank you for the PR! Please go through the checklist below and make sure you've completed all the steps. Please review the [CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md) document in this repository if you haven't already. The following items will ensure that your PR is handled as smoothly as possible: - PR Title must follow conventional commits format. For example, `feat: my new feature`, `fix(plugin-seo): my fix`. - Minimal description explained as if explained to someone not immediately familiar with the code. - Provide before/after screenshots or code diffs if applicable. - Link any related issues/discussions from GitHub or Discord. - Add review comments if necessary to explain to the reviewer the logic behind a change ### What? ### Why? ### How? Fixes # --> ### What? In a similar vein to #11734, #11733, #10327 - this PR returns a 404 in the response when a file is not found while using the `storage-gcs` adapter. Currently a 500 is returned. ### Why? To return the correct error level in the response when a file is not found when using `storage-gcs`. ### How? The GCS nodejs library exposes the `ApiError` as a general error - these changes check that the caught error is an instance of this class and if the provided code is a `404`.
@github-actions
Copy link
Contributor

🚀 This is included in version v3.43.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

2 participants