Batch processing vs stream processing
Inspired by 50 shades of system design
Batch processing stream processingin the trade-off explorer →
In old existing product data, it was stream processing, processing one spider item at a time.
whereas in dogs, it was batch processing, processing 'big' sqlite files
Sources
- 50 shades of system design inspired this note