What is Load Balancer?

A load balancer evenly distributes incoming traffic among web servers or workers defined in a load-balanced pool. Users connect to the public IP / VIP of the load balancer; backend servers are reachable over private IP from the load balancer only. Examples: Nginx, HAProxy, Amazon ELB/ALB/NLB, F5, Cloudflare.

Block Diagram — Load Balancer Placement

The client never talks to individual web servers directly. DNS resolves the domain to the load balancer's virtual IP (VIP). The LB terminates or forwards the connection, picks a healthy backend from its pool, and rewrites the destination address before forwarding on the private network.

flowchart TB
    subgraph Internet
      C[Client / Browser
DNS: example.com → 1.2.3.4]
    end

    C -->|HTTP/TCP to VIP 1.2.3.4| LB

    subgraph DC["Data Center / VPC — private network"]
      LB[Load Balancer
VIP: 1.2.3.4 public
10.0.0.1 private]

      subgraph Pool["Backend pool"]
        WS1[Web Server 1
10.0.1.10:8080]
        WS2[Web Server 2
10.0.1.11:8080]
      end

      LB -->|forward to chosen backend| WS1
      LB -->|forward to chosen backend| WS2

      WS1 --> Cache[(Cache cluster)]
      WS2 --> Cache
      Cache --> DB[(DB farm)]
    end
        

Advantages of LB

1. Failover

If Web-Server1 goes offline, all the traffic will be routed to Web-Server2. This prevents the website from going offline. If traffic grows rapidly and two servers are not enough, new web servers can be added to the pool without changing the public VIP the clients use.

Places where LB can be kept

Scheduling Algorithms for LB

Types of LB

Layer 3 Load Balancer (NLB, VPN concentrators, anycast edge routers)

A Layer-3 (network-layer) load balancer makes routing decisions using IP headers only. It does not inspect TCP ports in depth (that is Layer 4) and never parses HTTP (Layer 7). It is commonly used for NLB, VPN concentrators, and anycast edge routers that steer flows before transport termination.

How L3 Load Balancer Routes Traffic (Decision Path)

Consider an HTTP request arriving at load balancer with two web servers(WS1, WS2) behind it. The LB must pick WS1 or WS2 in well under a microsecond. LB does not query a database on every request — it keeps hot-path state in memory.
At starting LB keeps on checking health of Web servers and maintains the state of Web servers

struct BackendServer {
    int backendId;
    string ip;          // Real IP of web server
    int port;           // Real Port of web server
    int weight;
    int maxConnections;
    string availabilityZone;
    vector <string> tags;
    enum class HealthState {UP, DOWN, DRAIN} health_state;
};
unordered_map <int, BackendServer> backendPool;

Steps:
1. Client completes TCP handshake with the LB VIP (SYN → SYN-ACK → ACK). LB stores every HTTP Client information in Connection Table.

struct FiveTuple {    //
    string srcIP;                   // Src IP of HTTPS Client which connects to LB
    string dstIP;                   // Dst IP to which HTTPS Client want to connect. LB IP
    uint16_t srcPort;               // Src Port
    uint16_t dstPort;               // Dst Port
    uint8_t protocol;
    bool operator==(const FiveTuple& other) const {
        return srcIP == other.srcIP &&
               dstIP == other.dstIP &&
               srcPort == other.srcPort &&
               dstPort == other.dstPort &&
               protocol == other.protocol;
    }
};
struct ConnectionState {
    int backendId;
    string translatedIP;
    uint16_t translatedPort;
    chrono::steady_clock::time_point lastAccess;
};
unordered_map <FiveTuple, ConnectionState> connectionTable;

HashTable
            Key                       |   Value       
 src=1.1.1.1,dst=LBIP,                |   backendId=1           // WS1=1, WS2=2
 sport=2012,dport=2323,               |   translatedIP=2.2.2.2  
 Protocol=TCP/UDP                     |   translatedPort=5000
                                      |   lastAccess=
            

2. TLS termination (if HTTPS): LB decrypts using session keys cached in an SSL session cache
3. LB opens (or reuses from a connection pool) a TCP connection to WS2 10.0.1.11:8080 and forwards the HTTP request.
4. Response travels back through the LB to the client; connection table entry is updated or removed when the connection closes.

Why modern load balancers are fast

No disk on the hot path — every lookup is RAM-only.
Kernel bypass — DPDK, XDP/eBPF, or hardware ASICs process packets in user space or NIC firmware, avoiding syscalls per packet.
Pre-warmed backend connections — HTTP keep-alive and TLS session resumption remove repeated handshakes.
Health checks off the request path — probe results are cached; the forwarding thread only reads a boolean flag.

Capacity — Hardware Specs and Load Metrics

Load metrics to watch

Scaling the Load Balancer Itself

Important distinction: scaling backend web servers (adding WS3, WS4 to the pool) is separate from scaling the load balancer tier. When LB capacity is exhausted, you do not magically spawn a new LB instance from a single hardware box — you add redundancy and capacity through architecture.

What gets provisioned when load grows?

Backends scale out independently: auto-scaling groups or Kubernetes HPA add application pods/VMs and register them in the LB's backend pool via service discovery (Consul, Kubernetes Endpoints, cloud target groups). The LB configuration picks up new backends from its pool table — it does not "birth" them itself.

Primary and Secondary Load Balancer — Communication

Production systems almost never run a single LB. Two (or more) instances share a Virtual IP (VIP) or anycast address so clients see one entry point while operators get redundancy.

Active / Standby (Primary + Secondary)

flowchart LR
    C[Clients] --> VIP[VIP 1.2.3.4]

    VIP --> P[Primary LB
ACTIVE
holds VIP via VRRP]
    VIP -.->|VIP floats on failover| S[Secondary LB
STANDBY
syncs config]

    P --> B1[Backend pool]
    S -.->|takes over on failure| B1

    P <-->|VRRP/keepalived heartbeats
config sync| S
        

• VIP ownership — protocols like VRRP (keepalived) or cloud failover APIs assign the VIP to the primary. Heartbeats flow over a dedicated link; if the primary stops responding, the secondary gratuitously announces the VIP via ARP/NDP.
• Config sync — primary pushes nginx/HAProxy config, certificates, and routing rules to secondary (rsync, git ops, or vendor cluster sync). Both must have identical backend pools.
• Connection state — by default the standby has an empty connection table. Existing TCP sessions break on failover unless connection sync (HAProxy peers, F5 mirroring) replicates the 5-tuple table in real time — expensive and often limited to short flows.
• Split-brain prevention — quorum witness, STONITH, or cloud API ensures only one node holds the VIP.

Active / Active

Both LBs serve traffic simultaneously. Clients reach either node via:

• DNS round-robin / anycast — same VIP announced from multiple PoPs; BGP anycast routes each client to the nearest healthy LB.
• ECMP — top-of-rack switches hash flows across multiple LB NICs.

Each node maintains its own connection table. Stickiness must use a deterministic algorithm (source-IP hash or cookie) so the same client tends to hit the same node, or stickiness is abandoned in favor of stateless backends.

Session Stickiness (Persistence)

Session stickiness forces requests from the same client (or session) to reach the same backend across multiple requests, even when the LB would otherwise round-robin.

When stickiness helps

When stickiness hurts or is unnecessary

• Stateless REST / gRPC APIs with JWT and shared DB — stickiness adds imbalance when one client hogs a hot server.
• Mobile clients behind carrier NAT — thousands of users share one public IP; source-IP stickiness overloads one backend.
• Backend drain / deploy — sticky sessions resist graceful shutdown unless the LB supports slow drain and cookie refresh.

Internal logic — common mechanisms

Stickiness + multiple LB nodes

In active/active, an inserted cookie must be honored by all LB instances (same cookie name → same backend mapping algorithm). If only the primary knows the table, failover breaks sessions. Best practice: make backends stateless with a shared session store (Redis) and treat stickiness as an optimization, not correctness.