HTTP load balancer for API with concurrent requests

Question

We have a single server that communicates with another internal server via API calls. It takes a long time to handle the request on the API server, so we have 3 endpoints.

//Send all the information to the API to start the background process /start //Request the current status /progress //Get the final result /result 

We want to share the load over multiple API servers with a load balancer in front. In an ideal situation we would send a request to the load balancer (/start) and it would use round-robin to select an available server and pass back a parameter that it used server X. Then we send request to /progress and /result with parameter X to make sure it goes to the same server.

So basically we receive information back which back end server was used and we can control which server we want to use with a request.

I searched online but I can only find routing to the same back end based on IP or cookies, but that won't work because all request are from the same single client.

Is there a load balancer that can be setup like this on ubuntu?

Answer 1

all request are from the same single client.

That's an unusual scenario, but ultimately sticky routing boils down to:

1. creating a map of SOME PARAMETER to a specific origin host which might be in the form of an algorithm or a lookup table

2. when a request arrives, extract that parameter and resolve the relevant origin host

That parameter is usually a cookie or IP address but it can equally be part of the URL. Any of hostname, port, path, query.

The specifics of the best way to implement this depends on whether you control the code running on the backend and the choice of proxy. It can also be anything else in the request but manipulating other headers is complicated if the client is a browser.

Answer 2

This is all assuming you have the ability to modify the API.

A: convert steps 2 and 3 into callbacks

or

B: have the server uniquely identify itself in its response to stet one and use that identification to bypass the load-balancer for steps 2 and 3

Answer 3

What you're looking for is called "stickiness", "sticky sessions" or "session persistence".

The simplest and probably fastest-to-implement solution is to keep the TCP connection alive (TCP and HTTP can do keep-alives) between /start, /progress and /result. This will ensure the LB sends the traffic to the same backend server as the LB balances connections and not packets (otherwise your whole infra would be broken!).

Alternatively, a typical and more scalable solution is for backend servers to send a header with a unique value in the response to /start (i.e. a session cookie), and subsequent client requests should include said header. This header could also be just managed by the LB without the backend servers being aware of it. The LB will route the requests for the session to the right backend server based on the header's/cookie's value. You have mentioned cookie would not be viable for you because of the single client, but that's not true as cookies will be unique per session, and the client should know which session is querying during /progress and /result. Single client in this scenario is only relevant to IP based stickiness / persistence.

I'd say most if not all LBs support cookie based stickiness. A couple of well known open source LBs:

Traefik: https://doc.traefik.io/traefik/v2.11/routing/services/#sticky-sessions

HAProxy: https://www.haproxy.com/documentation/haproxy-configuration-tutorials/proxying-essentials/session-persistence/ (make sure you do cookie based persistence and not IP based!)