2

I have an instance group with 2 instances behind a HTTP Load Balancer. one instance is up and functioning normally (returning http 200), the other is crashed (HTTP requests timeout). I am not sure what I'm doing wrong, but according to the documentation the failed instance should automatically be removed from the load balancer.

Here is the related docs: https://cloud.google.com/compute/docs/load-balancing/health-checks with the related paragraph:

For a health check to be deemed successful, the backend must return a valid HTTP response with code 200 and close the connection normally within the timeoutSec period. If an instance fails its health check, it is removed from the group or pool without any notification being sent. If it later passes a heath check, it is returned to the group or pool, again without any notification.

Here is what I currently see on my google cloud console page for the HTTP Load Balancer's Backend.

enter image description here

When visiting my site (http://stage2.phantomjscloud.com) Rougly half the time I get

Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

The HTTP Load Balancer (and health check) clearly detects the failed instance, but traffic is still being served to it regardless.

How can I resolve the issue?

Improve this question
3
  • Do you still have the same issue ? The issue could be because of the session affinity "An HTTP(S) session stays on the same instance for the duration of the session, even if the instance begins to register as unhealthy." This information is documented here (cloud.google.com/compute/docs/load-balancing/http/…) Commented Oct 9, 2015 at 16:28
  • Thanks for that detail, I didn't know about the session affinity, but I don't think that's it. I did recreate the failed instance and kept getting errors (???) until I modified the backend priorities, and now it's working. I will let it run a few days and see if the issue reoccurs (as it has in the past) Commented Oct 10, 2015 at 18:33
  • To troubleshoot this kind of issues it can be helpful to install the stackdriver agent in the backend instances. The load balancer logs could also help. Having a large HC interval could also affect Commented Nov 3, 2016 at 20:43

2 Answers 2

Reset to default
3

health checks managed instance groups VS health checks load balancing

The health checks used by managed instance groups are the same health checks used by load balancing, with some differences in behavior. Health checks that you apply to load balancing services help a load balancer determine where to direct network traffic. These health checks do not cause Compute Engine to recreate instances. Health checks that you apply to managed instance groups will proactively signal to the managed instance group to delete and recreate instances if they become UNHEALTHY.

For the majority of scenarios, use separate health checks for load balancing and for monitoring managed instance groups. Health checking for load balancing can and should be more aggressive since these health checks determine whether an instance receives user traffic. Since customers might rely on your services, you want to catch non-responsive instances quickly so you can redirect traffic if necessary. In contrast, health checking for instance groups will cause Compute Engine proactively replace failing instances so you could create health checks that are more conservative than health checks for a load balancer.

https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances#monitoring_groups

Improve this answer
0

I haven't seen this kind of bug in a while (6 months or so) so I think it was a bug with Google Cloud, and they fixed it.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.