How I Scaled an API to 100K Requests/Second with P99.99 Latency of 150ms

The Scale Problem

Ximalaya's Album Service was the backbone of one of China's largest audio platforms -- 80M+ albums, billions of plays, and growing fast. When I took ownership, the service was struggling to keep up with demand during peak hours.

The goal was clear: 100K+ API requests per second with P99.99 latency under 150ms.

Layer 1: Application-Level Caching

The first optimization was a local cache layer using Caffeine (a high-performance Java caching library):

LoadingCache<Long, AlbumDTO> localCache = Caffeine.newBuilder()
    .maximumSize(50_000)
    .expireAfterWrite(30, TimeUnit.SECONDS)
    .refreshAfterWrite(10, TimeUnit.SECONDS)
    .build(albumId -> fetchFromRedis(albumId));

This eliminated ~60% of requests from ever hitting the network. The 30-second TTL with 10-second refresh meant data was usually fresh, and the 50K entry limit kept memory bounded.

Layer 2: Distributed Cache (Redis Cluster)

For cache misses, we hit a Redis cluster. But not just any Redis -- we used a three-tier approach:

1. Codis proxy layer: Handles connection pooling and request routing
2. Redis Cluster: The primary distributed cache with 6 master nodes
3. Pika: A Redis-compatible persistent store for cold data that didn't fit in memory

Solving the Hot Key Problem

Popular albums could receive millions of requests per minute, creating hot key issues on individual Redis nodes. Our solution:

• Key replication: Hot keys replicated across multiple slots with a suffix
• Local cache promotion: Albums trending above a threshold were pinned in local cache
• Read replicas: Added Redis read replicas for the hottest data partitions

Layer 3: Database Sharding

With 80M+ albums, a single PostgreSQL instance was never going to cut it. We implemented sharding using Apache ShardingSphere:

• Shard key: Album ID with consistent hashing across 64 shards
• Routing: ShardingSphere intercepted queries and routed to the correct shard
• Cross-shard queries: Avoided where possible; used Elasticsearch for search-based access patterns

RPC Optimization with Netty and Apache Thrift

The final piece was optimizing the RPC layer:

• Replaced HTTP/JSON with Apache Thrift binary protocol (40% smaller payloads)
• Custom Netty server with connection pooling and multiplexing
• Batch APIs: Allowed fetching multiple albums in a single RPC call, reducing round trips

Monitoring and Validation

We validated every optimization with comprehensive load testing:

• JMeter for sustained load tests (100K RPS for 30 minutes)
• Custom P99.99 tracking in our metrics pipeline
• Flame graphs to identify CPU bottlenecks in serialization paths

Results

Metric	Before	After
Throughput	~30K RPS	100K+ RPS
P99.99 Latency	~800ms	150ms
Cache Hit Rate	45%	94%
Database Load	70% CPU	15% CPU

Lessons Learned

1. Cache close to the consumer: Local cache > distributed cache > database. Each layer reduces load on the next.
2. Measure before optimizing: Flame graphs and profiling told us exactly where time was spent.
3. Plan for hot spots: In any system with popular content, hot key management is critical.
4. Binary protocols matter: At 100K RPS, the difference between JSON and Thrift is significant.
5. Shard early, shard smart: Database sharding is easier to implement before you're drowning in data.