0

I'm trying to configure Nginx + nginx-auth-jwt (from alpine registry) with the multiple locations:

  • / serving React application
  • 3 REST APIs each with
 auth_jwt "closed site"; auth_jwt_key_request /_jwks_uri;
  • 1 exact location (almost identical configuration, without trailing slash, and $username in the path)
 location = /auth/users { auth_jwt "closed site"; auth_jwt_key_request /_jwks_uri; proxy_pass ${AUTH_HTTPS_URL}/some/path/$username/info; proxy_set_header api-key ${SOME_API_KEY}; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout ${SOME_TIME}; }

Before the locations I have the following:

auth_jwt_claim_set $username ${JWT_USERNAME_CLAIM}; resolver ${NGINX_DNS_RESOLVER};

All of the variables are populated by envsubst-on-templates.sh as in the official Nginx docker image.

With the same values (even the same .conf file with different resolver) everything works locally, but when deployed in k8s the /auth/users returns 403 without response body.

The only way I was able to reproduce 403 locally is when change the ${JWT_USERNAME_CLAIM} value, so that $username is an empty string, but this way I receive back the response from the AUTH_HTTPS_URL upstream. From the k8s deployment the response body is empty

  • If the token is invalid/expired the endpoint returns 401 Unauthorized
  • The responses from the 3 APIs work just fine with the very same token
Improve this question

2 Answers 2

Reset to default
1

This usually happens because Nginx doesn’t expand variables in proxy_pass when the URL includes a scheme (like https://). So $username isn’t being resolved at runtime in Kubernetes, even though it works locally.

Try changing this:

proxy_pass ${AUTH_HTTPS_URL}/some/path/$username/info;

to:

set $auth_upstream "${AUTH_HTTPS_URL}/some/path/$username/info"; proxy_pass $auth_upstream;

That forces Nginx to evaluate $username before making the request.

If it still returns 403, enable debug logs and check if $username is empty. In Kubernetes, it can happen if ${JWT_USERNAME_CLAIM} isn’t passed correctly to the container or the JWT claim name is different in production.

You can check that by running:

kubectl exec -it <pod> -- printenv | grep JWT

Also make sure your resolver in k8s points to a valid internal DNS server. A bad resolver can also cause 403s with empty bodies.

Most of the time, fixing the proxy_pass line and confirming the env vars are properly injected resolves this issue.

Improve this answer
New contributor
Steve is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1
  • Nginx doesn’t expand variables in proxy_pass when the URL includes a scheme - This is complete nonsense and absolutely untrue. Was it suggested by some LLM? Commented yesterday
0

As I was expecting the behaviour of the application (entire container) was the same on Kubernetes and locally, as I was using the very same image.

It turned out the request was blocked by the upstream, due to allow list filtering by X-Forwarded-For value.

If I had changed the configuration to:

proxy_set_header X-Forwarded-For localhost;

it would have worked.

TL;DR: I know that my question is specific to my environment/upstream, but I hope this answer could eventually be helpful to someone.

Improve this answer
2
  • Why changing the X-Forwarded-For header value to something meaningless, leaving it in the payload to the upstream, instead of not passing it at all? Commented yesterday
  • @IvanShatsky In my case I didn't left it as "something meaningless", it was whitelisted by the upstream, I've shared it as a way to replicate the working flow, as this was the value sent when running the app on localhost. If the header is expected form the upstream I presume this could eventually help someone struggling in similar situation Commented 18 hours ago

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.