When to use system queues

• operations too long for a live server, get feedback to users quickly
• reduce load on a live server
• when you want the ability to scale differently for different tasks (ie retrievers need less memory than collectors in dogs)
• reliability: keep messages to be processed in queue until workers are healthy again

Difference between message queue and stream

Kafka can be used as a queue, but main use case alongside Kinesis is streaming.
For Kafka:

The key difference between the two lies in the consumption pattern. When used as a message queue, each message is processed by one consumer in a group and then effectively "consumed" (though Kafka still retains it based on retention policy). When used as a stream, consumers continuously process messages as they arrive in real-time, and the same data can be read by multiple independent consumer groups or replayed from any point in the log.

at least once vs at most once vs exactly once

Message queues can be at least once, at most once or exactly once (hard).

at least once

Default approach if you're ok with messages sometimes being processed more than once.
Messages need to be idempotent for this to work (no side effect if you process it twice).

• set balance + 1: not idempotent as it relies on the previous state
• set balance to 54: idempotent as it can be processed multiple times and achieve the same outcome

at most once

if a message cannot be processed more than once, then consumers can ack as soon as they get the message. there is no guarantee however that it will succeed at processing the message.
you have to be ok with messages being lost sometimes

exactly once

hard. don't promise you can do that unless you can explain how it works, apparently kafka has some way to get close to that

How to ensure that messages are consumed by only 1 consumer at a time

SQS makes messages invisible after a consumer acquires it. When it acknowledges it, it will delete it. If it doesn't after a certain time, it will become available for consumption again.
This is essentially how "at least once" works.

Kafka makes partitions so that messages will only ever be consumed by a certain consumer.
1 partition = 1 consumer.

Ordering

Ordering can be guaranteed within a partition, make sure you put messages that need to be processed in order in the same partition.
SQS offers different options of ordering.

Partitioning

Careful when you create your partition key that you're not creating hot partitions (same concept as Database Sharding).

How to deal with queues growing

Add more consumers.
Add back pressure (make producers produce less).

Kafka vs SQS vs Kinesis

Kafka is pub/sub (multiple subscribers).
SQS is queue (1 consumer for 1 queue).
SNS allows publishing messages and creating multiple unique queues out of it (fannout pattern).
SQS remembers messages for up to 4 days.
SQS has limited size per message (256kb).
Kinesis is more for real time streaming. Continuous stream, multiple subscribers.
Kinesis is basically Kafka but using more the AWS stuff.

Resource

Youtube - Hello Interview – Message Queues in System Design Interviews w Meta Staff Engineer
Youtube - ByteByteGo – 20 System Design Concepts You Must Know - Final Part