Distributed Systems Architecture

Q: Why are virtual nodes necessary in Consistent Hashing?

With few physical nodes, hash positions can clump together on the ring, causing one server to receive 80% of all requests (Hotspot). Virtual nodes (e.g. 100-250 virtual tokens per physical node) ensure uniform key distribution across the ring.

Architect high-scale architectures: stateless application tiers, consistent hashing, distributed caching (Redis), database sharding, and message queues.

Horizontal Scaling, L4 vs L7 Load Balancing & Consistent Hashing

Designing systems to serve hundreds of millions of users requires scaling out horizontally across stateless compute nodes, with intelligent layer-4 and layer-7 traffic distribution and distributed caching rings.

Layer 4 Load Balancing (Transport):Routes raw TCP/UDP packets without terminating TLS or inspecting HTTP headers (e.g. AWS Network Load Balancer, Linux LVS/IPVS) for millions of connections per second.
Layer 7 Load Balancing (Application):Inspects HTTP headers, cookies, URL paths, and JSON payloads (e.g. NGINX, HAProxy, Envoy, AWS ALB) for intelligent path routing and SSL termination.
Consistent Hashing with Virtual Nodes:Maps both servers and cache keys to a 360-degree integer ring. When a server node is added or removed, only K/N keys are remapped on average, preventing cache stampedes.

Consistent Hashing Ring Placement: Hash each server ID onto the ring: pos = Hash(Server_ID) mod 2^32. To avoid hot spots, generate V virtual nodes per physical machine (e.g. Server1#1, Server1#2).
Key Lookup via Binary Search: Hash the data key Hash(Key). Locate the first server whose ring position is >= Hash(Key) using binary search (std::lower_bound or bisect).
CAP Theorem Partition Tolerance: In a distributed network partition, you must choose between Consistency (CP - reject writes) or Availability (AP - allow split-brain writes and reconcile via Vector Clocks/CRDTs later).

Definition: Consistent Hashing

A distributed hashing scheme where the addition or removal of one cache server node causes only minimal data migration (K/N keys) rather than remapping all keys from scratch.

Rebalancing Footprint Formula

Keys Remapped on Node Change = TotalKeys / TotalServers (K / N)

Variable Definitions & Term Breakdown:

Traditional Hashing (Key mod N)Changing N forces almost 100% of keys to move, crashing backend databases

Consistent HashingOnly 1/N-th of all keys migrate to the neighboring node on the ring

Layer 4 vs Layer 7 Load Balancer Comparison

Feature	Layer 4 (L4 - Transport)	Layer 7 (L7 - Application)
Routing Criteria	IP & TCP/UDP Port only	HTTP Headers, Cookies, URL Paths, JSON Body
Throughput & Latency	Ultra High (>1M req/s, <0.1ms)	High (~100k req/s, 1-2ms)
SSL/TLS Termination	Pass-through or simple offload	Full TLS Handshake, Certificate Inspection & Decryption
Use Cases	Gaming servers, VoIP, edge gateways	Microservices routing, rate limiting, A/B canary testing

Throughput, inspection depth, and TLS handling

Tip

Always remember the PACELC theorem: If there is a Partition (P), how does your system trade Availability (A) vs Consistency (C)? Else (E) when normal, how does it trade Latency (L) vs Consistency (C)?

PythonConsistent Hashing Ring Implementation Sandbox

1import hashlib
2import bisect
3 
4class ConsistentHashRing:
5    def __init__(self, replicas=3):
6        self.replicas = replicas
7        self.ring = []
8        self.ring_map = {}
9 
10    def _hash(self, key: str) -> int:
11        return int(hashlib.md5(key.encode()).hexdigest(), 16)
12 
13    def add_node(self, node: str):
14        for i in range(self.replicas):
15            vnode = f'{node}#v{i}'
16            pos = self._hash(vnode)
17            bisect.insort(self.ring, pos)
18            self.ring_map[pos] = node
19 
20    def get_node(self, key: str) -> str:
21        if not self.ring:
22            return None
23        pos = self._hash(key)
24        idx = bisect.bisect_right(self.ring, pos)
25        if idx == len(self.ring):
26            idx = 0
27        return self.ring_map[self.ring[idx]]
28 
29ring = ConsistentHashRing(replicas=5)
30ring.add_node('cache-node-1.db')
31ring.add_node('cache-node-2.db')
32ring.add_node('cache-node-3.db')
33 
34keys = ['user:1001', 'session:abc928', 'order:5920', 'product:laptop']
35for k in keys:
36    print(f'{k} -> Assigned to: {ring.get_node(k)}')

Output

user:1001 -> Assigned to: cache-node-3.db
session:abc928 -> Assigned to: cache-node-2.db
order:5920 -> Assigned to: cache-node-1.db
product:laptop -> Assigned to: cache-node-3.db

NGINXNGINX Layer 7 Reverse Proxy & Load Balancer Config

1upstream backend_cluster {
2    least_conn;
3    server backend1.prod.internal:8080 weight=3;
4    server backend2.prod.internal:8080 weight=2;
5    server backend3.prod.internal:8080 backup;
6}
7 
8server {
9    listen 443 ssl http2;
10    server_name api.programmersos.com;
11 
12    location / {
13        proxy_pass http://backend_cluster;
14        proxy_set_header Host $host;
15        proxy_set_header X-Real-IP $remote_addr;
16    }
17}

Output

[NGINX Active - Load Balancing across 3 upstream servers]

Quick Syntax & Reference Sheet

Concept / Rule	Syntax / Usage
Consistent Hashing Ring	`pos = hash(node_id); bisect.insort(ring, pos); node = ring[bisect(ring, hash(key))]`
CAP Theorem Rule	`P (Network Partition) is inevitable in distributed systems. You MUST choose between CP or AP.`

Common Pitfalls & Developer Mistakes

✕ Mistake 1:

Using simple modulo hashing `hash(key) % N` for distributed caches

Why it happens: Adding or removing a single node changes N, causing almost 100% of keys to map to the wrong server, crashing the primary database with cache misses.

Correction: Always use Consistent Hashing with virtual nodes for distributed key distribution.

# Incorrect:
node = servers[hash(key) % len(servers)]

# Correct:
node = consistent_hash_ring.get_node(key)

Frequently Asked Questions

Why are virtual nodes necessary in Consistent Hashing?

With few physical nodes, hash positions can clump together on the ring, causing one server to receive 80% of all requests (Hotspot). Virtual nodes (e.g. 100-250 virtual tokens per physical node) ensure uniform key distribution across the ring.