Systems & infrastructure ↔ Interactive essay
Understanding load balancing
How a request finds its way to the right server. A practical explanation, with a few things to try along the way.
A request arrives. Somewhere behind a single address, several servers are ready to answer it. Which one should do the work?
That’s the question a load balancer answers. The basic idea is simple, but the decision gets more interesting when servers have different capacities, requests take different amounts of time, or something stops responding.
Start with taking turns
The simplest approach is round robin. Send the first request to the first server, the next to the second, and keep going. When you reach the end, start again.
Try it below. Send a few requests and watch which server receives each one. Then take a server offline and send another request.
A tiny system. Yours to play with.
The address your users talk to hasn’t changed. The set of servers doing the work has. That separation is one of the most useful things a load balancer gives you: a stable way into a system whose parts keep changing.
Notice that the offline server gets skipped. In a real system, a health check helps decide whether a server belongs in that rotation. Until the check detects a failure, some requests can still reach the unhealthy server.
Equal turns aren’t always equal work
Round robin works well when servers are similar and requests are short. But an equal number of requests doesn’t necessarily mean an equal amount of work.
One request might return a cached value. Another might wait on a slow database query. A third might open a connection that stays alive for an hour. Treating those as identical can leave one server busy while another has room to spare.
Different algorithms answer different versions of the question:
- Weighted balancing gives a larger share of requests to servers with a higher assigned weight. It’s useful when capacities differ or you’re gradually shifting traffic.
- Least connections chooses the server with the fewest active connections. It helps when requests last for different amounts of time, although connection count is still an imperfect measure of work.
- Power of two choices samples two servers and picks the less loaded one. It avoids having to compare every server for each decision.
- IP hash uses a client key to pick a server consistently. That can help with session affinity, but a large group of clients sharing a key can make the distribution uneven.
There isn’t one technique that wins everywhere. You need to know what work looks like before deciding how to share it out.
The decision happens at more than one layer
A request can pass through several routing decisions before it reaches your application. DNS might choose a region. An edge proxy might choose an origin. An internal service might choose a pod.
Each layer has different information. DNS doesn’t know how long an individual HTTP request will take. An application proxy can inspect a path or header. A service-level balancer usually knows about a smaller, changing set of endpoints.
It’s useful to ask which layer is making a decision, what it can see, and how quickly it reacts when that information changes.
Try a larger system
The simple illustration above leaves out capacity and request duration. This more detailed model lets you compare algorithms, increase traffic, and fail a backend. Expand it when you’re ready to explore those trade-offs.
Explore algorithms, health checks, and capacity
Load balancing is one of those ideas that sounds simple until a real system starts bending around it. It is not just "send requests to servers evenly". It is where health, capacity, failure, locality, rollouts, and product promises meet the network.
Where it fits
A load balancer gives callers one stable place to send traffic while the system behind it keeps changing. Instances scale up, pods restart, regions degrade, deploys move from old code to new code, and the caller should not need to know the exact machine that will answer.
There is rarely just one load balancer. There can be DNS or global traffic management before the request reaches a region, CDN and WAF rules at the edge, an HTTP gateway in front of the app, internal service balancing between services, and connection pooling near databases.
The trick is to ask what decision is being made at each layer. Are we choosing a region, a cluster, a service version, a pod, or a read replica? Same family of problem, different consequences.
Current layer
Service LB
Once inside the platform, load balancing becomes service discovery plus routing.
The common techniques
Every technique is a guess about what "fair" means. Round robin says fair means equal turns. Weighted balancing says stronger machines should carry more. Least connections says current load matters more than history. Hashing says the same client should keep landing in the same place.
None of these is universally best. A good system usually combines a routing technique with health checks, timeouts, retries, circuit breakers, autoscaling, and observability. The balancer can steer traffic, but it cannot make a slow dependency fast.
api-a
az-1 / weight 4 / capacity 90
api-b
az-1 / weight 3 / capacity 72
api-c
az-2 / weight 2 / capacity 54
api-d
az-2 / weight 1 / capacity 28
What the simulator is trying to show
Turn health checks off and a dead backend still receives requests. That is the classic load-balancing footgun: distribution without health is just a very tidy way to send users into a hole.
Push traffic higher and the algorithm starts to matter less than capacity. Once every healthy backend is saturated, the fix is not a cleverer rotation strategy. You need more capacity, less work per request, better caching, backpressure, or a product decision about what to shed first.
Enable sticky load with IP hash and you can see why session affinity is useful but dangerous. It protects local state and warm caches, but a few heavy clients can make the system look unfair even when the balancer is doing exactly what you asked.
The system design version
In an interview or architecture review, load balancing is not a box you draw once and move past. It is a set of routing decisions. Start from the user and walk inward: global entry, edge policy, application gateway, internal service routing, and dependencies.
For each step, explain the signal you use to route traffic, the failure mode you expect, and how you know the balancer is helping. That is usually more useful than naming one algorithm and hoping it sounds fancy.
The numbers in the model are illustrative. What matters is the relationship between the routing policy, the available capacity, and the work already in flight.
What I take away from this
Load balancing is a policy decision about where work should go. The algorithm is one part of it. Health checks, timeouts, retries, session affinity, and capacity all affect the result.
When a system struggles, I find these questions more useful than asking which algorithm is best:
- What counts as healthy, and how quickly do we detect a failure?
- Does the routing decision reflect the work servers are actually doing?
- What happens to the remaining servers when one goes away?
- Could retries turn a small failure into a bigger one?
The goal is to make those choices deliberately, and to know what your system will do when the assumptions stop holding.