Azure Application Gateway: Backend health returns HTTP 463, users get circular loop of 301s

Question

I have an application gateway for which my backend settings are configured with health probes allowing status 200-499 as healthy (I've included the 4xx series codes as some solutions have site roots which return 404 whilst have content under other paths, and some sites return a 403 automatically for an unauthorised user / I don't want those to impact the backend health status / I wanted my solution to be as generic as possible so I can reuse the same settings across a series of sites to minimise effort when onboarding new sites).

However, my backend health shows one of my sites as returning HTTP 463 status codes (so the backend is still healthy, but this response is unexpected). Further, if I navigate to the URI of the listener associated with this backend, I get stuck in a 301 redirect to/from the same site. I do have redirects configured:

• The backend redirects from / to /login.aspx for an unauthenticated user. This uses relative paths, so isn't impacted by the AppGW's hostname or frontend vs backend protocol.
• The AppGW has HTTP and HTTPS listeners, the HTTP rule redirects to the HTTPS listener, whilst the HTTPS rule is associated with the backend.

However, even if I remove those features, I get the same behaviour.

Other sites on the same AppGW are setup in the same way, but don't have this issue.

I have a PowerShell script for testing redirects: Get-HttpUrlRedirects. If we assume the configuration was for example.com, the EXPECTED output of running Get-HttpUrlRedirects -Url 'http://example.com' -Verbose would be:

VERBOSE: Redirecting to [https://example.com/] VERBOSE: Redirecting to [https://example.com/login.aspx] Url StatusCode --- ---------- http://example.com/ 301 https://example.com/ 302 https://example/login.aspx 200 

Whilst the ACTUAL output is:

 VERBOSE: Redirecting to [https://example.com/] VERBOSE: Redirecting to [https://example.com/] VERBOSE: Redirecting to [https://example.com/] VERBOSE: Redirecting to [https://example.com/] Url StatusCode --- ---------- http://example.com/ 301 https://example.com/ 301 VERBOSE: Redirecting to [https://example.com/] https://example.com/ 301 VERBOSE: Redirecting to [https://example.com/] https://example.com/ 301 VERBOSE: Redirecting to [https://example.com/] # etc - I hit ctrl + C to terminate when I see this loop occurring 

Answer 1

The issue here was our firewall blocking traffic between the AppGW and the related backend. Adding an allow rule resolved it.

Detail

Our application gateway is setup on a subnet which has a route table associated such that 0.0.0.0/0 is directed to next hop: internet, whilst the various private ip ranges (e.g. 172.16.0.0/12) have their next hop type as virtual appliance and the target as our Firewall's internal IP. I.e. this ensures that traffic hitting the AppGW's public IP has responses returned from the same public IP (rather than those responses going out via the FW's public IP), but all internal traffic goes via our firewall before hitting the backend servers.

The firewall had a rule restricting access from our AppGW's CIDR to the backend's VNet's CIDR (or rather, it didn't have a rule allowing this; and denies by default).

When checking the firewall's logs in case that had been the issue nothing showed up (i.e. the following Kusto returned only backends that were working, without showing an Action=Deny for the failing backend: AZFWNetworkLog | where ipv4_is_in_range(SourceIp, '172.12.0.0/28') /* CIDR for our AppGW Subnet */ and DestinationPort == 80 | distinct DestinationIp), so it looked as if the traffic wasn't reaching the firewall, as it wasn't getting logged. Since HTTP codes 463 and 301 were being returned, that implied that something was responding, rather than something getting blocked (as we may expect if we saw a 503); so it took a while to uncover as the evidence felt like it was pointing away from the FW.