When to use sharding

• Write-heavy: Too much data for 1 instance (storage or write per second)
• Read-heavy: when read replicas or caching are not enough

Trade offs

• Global queries will be slower
• Complexity to operate and keep data consistent

Sharding methods

Hashing

Needs to be consistent (consistent hashing ).

How to choose a shard key

• high cardinality (lots of values to spread)
• even distribution so that every shard has the same amount of data
• when you query, you need only one shard

To avoid hot spots, perhaps you can shard a bit more granular, ie not just bye user id but by user id + post creation date

What not to do

Boolean is very low cardinality
Sharding by creation date where most people query the most recent year makes one shard hot, not evenly distributed
Range based sharding, like by user id might make hot shards, all users in one shard depending on how many

Directory based sharding

You can do directory based sharding where you stored somewhere in which shard the data is. Allows flexibility you can easily change in which shard something is ik case it's really hot, but it adds lots of latency as you need to look it up every time. Adds a single point of failure too.

You can do something hybrid where you manually store for hot keys in which shard they are stored, and use the normal hashing method for the rest.

Issues with sharding

Cross-shards requests

Some queries might need data across shards, and this could become really slow.

Remediation

You can cache expensive cross shard queries.
This is where de normalisation could have a use case, let the related data live in the same shard but multiple writes

Consistency

However, if you need to make atomic transactions across services or databases, then sharding makes it harder, but not impossible.
Splitting data across services breaks atomicity, creating distributed transaction challenges.
Distributed transactions
Easiest is to have all in one db that.

Resource

Youtube - Hello Interview – Sharding in System Design Interviews w Meta Staff Engineer