1

I've setup a Load Balancer in AWS which routes traffic to an Apache Ubuntu server.

When I browse to my URL, sometimes everything goes well. Then I hit the Refresh button and I start getting 502 errors when downloading some files. It's different files each time.

Then I hit Refresh again and everything goes fine.

I'm currently the only one using the server.

My Load balancer is setup like this:

  • Availability Zones: us-east-1a and us-east-1b
  • Security groups: port 80 and 443 opened
  • Listeners: Port 80 redirect to port 443, port 443 forwards to my Target

My target group is configured like this:

  • Target type: instance
  • Protocol: HTTP: 80
  • Load balancer: My load balancer name
  • Register targets: I have 1 instance running in each zone.

My Auto Scaling Group is configured like this:

  • Desired capasity: 2
  • Minimum capacity: 2
  • Maximum capacity: 5
  • Availability zones: us-east-1a, us-east-1b

My apache server is configured with the default install values, plus:

  • KeepAlive On
  • Timeout 120
  • KeepAliveTimeout 120
  • MaxKeepAliveRequests 100
  • AcceptFilter http none
  • AcceptFilter https none

I'm not a server guy, I'm a programmer so I really don't know what to check. I thought configuring a Load Balancer was straight forward...

Thanks for your help

EDIT 2020-08-27 15h39

I enabled the logs on the VPC and on the Load Balancer and I refreshed my page. I had an 502 error on a specific file.

In the Load Balancer logs, I see the 502 entry for the specific file. Here`s the entry - JSON formatted - in which I replaced any sentive data

[ { "type": "h2", "time": "2020-08-27T19:20:52.547402Z", "elb": "app/web-server-load-balancer/**my-load-balancer-id**", "client:port": "*my-ip-address*:60831", "target:port": "10.0.0.106:80", "request_processing_time": 0, "target_processing_time": 0.013, "response_processing_time": -1, "elb_status_code": 502, "target_status_code": "-", "received_bytes": 45, "sent_bytes": 610, "request": "GET https://**my-url** HTTP/2.0", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36", "ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256", "ssl_protocol": "TLSv1.2", "target_group_arn": "**my-target-group-arn**", "trace_id": "Root=1-5f480794-150df3bcad7fd0ac6bdfc41c", "domain_name": "**my-domain-name**", "chosen_cert_arn": "session-reused", "matched_rule_priority": 0, "request_creation_time": "2020-08-27T19:20:52.534000Z", "actions_executed": "forward", "redirect_url": "-", "error_reason": "-", "target:port_list": "10.0.0.106:80", "target_status_code_list": "-", "classification": "-", "classification_reason": "-" } ]

I don`t know if those data can help you helping me!

EDIT 2020-08-31 (1)

I opened the two VMs in the Firewall to access them directly. When I access each server directly, I do not get the 502 errors at all. When I access via the Load balancer, I get the 502.

EDIT 2020-08-31 (2)

I changed my load balancer to route the HTTP traffic to the instance instead of redirect it to the 443 port.

On port 80, I do not get 502 errors. I hard-refreshed many many times and did not get 502. As soon as access the load balancer with https, I start to get 502 errors. I remember you that when I directly access the VM with https, I do not get 502.

Improve this question
10
  • When I have a 502 on a file, the request does not reach Apache. I do not see the request neither in the access.log and error.log file Commented Aug 26, 2020 at 22:34
  • OK, if the 502 came from the load balancer, then you get to make wild guesses. Commented Aug 26, 2020 at 22:58
  • You could enable VPC flow logs to see what's happening on the network. They capture network flow metadata, not packet contents, but they could give you some clues. docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html Commented Aug 27, 2020 at 9:17
  • I'll create a new VPC, setup a new load balancer and enable the logs on my VPC. I'll keep you in touch. Commented Aug 27, 2020 at 15:58
  • The log entry doesn't specify the reason for the 502 error. But it does say that it received 45 bytes. That's probably not enough for full response headers. This suggests your app may be crashing in the process of sending a response. Commented Aug 27, 2020 at 20:44

2 Answers 2

Reset to default
4

I came across this forum post: AWS Developer Forum

The last entry actually gave me the answer:

Ensure you're not using Apache's event MPM module (default) when behind a ALB/ELB. It dynamically closes connections. Try worker MPM.

I changed my Apache config to use worker MPM and as far as it goes, I did not had any 502 errors.

Improve this answer
2
  • If running PHP, the worker MPM isn't compatible. Use PHP-FPM instead. This solved the issue for me, where PHP was crashing without a response to ELB which resulted in a 502 and no entries in access log nor error log. Switching to PHP-FPM (from the default prefork) turned the same underlying bug (infinite recursion!) in a 500 error that did get logged to access and error logs, and did send a 500 response back to ELB. No more 502s! Then from there it was easy to identify and solve the underlying bug. Commented Jan 26, 2024 at 15:08
  • Also note the AWS Developer Forum link is dead. I couldn't find an archived version of it. Commented Jan 26, 2024 at 15:16
0

I tried both worker and event MPMs and neither of them completely eliminated the random 502. I finally switched to prefork MPM and this fixed the issue for me. This is on Apache in the middle between ALB and Tomcat.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.