Design a Distributed Rate Limiter w/ a Ex-Meta Staff Engineer: System Design Breakdown
Summary AI-generated
Summarised by an AI assistant when this source was saved. Double-check against the original.
In summary, the most interesting aspect of the design is the trade-off between latency and accuracy, specifically how using a Redis Lua script ensures atomicity to prevent race conditions in a distributed environment while maintaining sub-10ms performance.
- Rate limiters protect backend services from abuse and spam by controlling request frequency per client.
- Ideal placement is at the edge (API Gateway/Load Balancer) to act as a 'bouncer' before requests hit microservices.
- Client identification can be based on User ID, IP address, or API keys, often using a combination for different tiers (e.g., premium users).
- Token Bucket is the preferred algorithm for handling both steady-state load and temporary bursts.
- Redis is used for shared global state, utilizing sharding and Redis Cluster to scale to millions of requests per second.
- To ensure high availability, the system can 'fail open' or 'fail close,' with read replicas mitigating node failures.
- Latency is minimized via connection pooling and geographic colocation of gateways and cache instances.
- Dynamic rule updates are best handled through a push-based configuration system like Zookeeper or etcd to avoid polling overhead.
Notes this inspired
- Adding a dependency to a service adds network overheadGrowing
Server operation time + network overhead. For example, Redis operations are quite fast (couple of ms), there will always be network overhead.
- Distributed configuration managementGrowing
Distributed configuration management, zookeeper is a bit outdated but good for this. Example: The API Gateway keeps an open connection to config server and pushes changes straight away. Other not as good…
- Connection PoolingGrowing
Connection pooling allows avoiding TCP handshake overhead. Keep some connections ahead.
- Database ReplicationSeedling
Synchronous is slower, but ensures stronger consistency.
- Database ShardingEstablished
Write-heavy: Too much data for 1 instance (storage or write per second) Read-heavy: when read replicas or caching are not enough. Global queries will be slower. Complexity to operate and keep data…
- API Rate limitingGrowing
Django rate limiter? In memory per server. You don't get a global picture tho. Or separate service but extra latency. Most common is to have it with the api gateway. Rate limiting…