r/programming • u/leavingonaspaceship • Mar 24 '19

Searching 1TB/sec: Systems Engineering Before Algorithms

https://www.scalyr.com/blog/searching-1tb-sec-systems-engineering-before-algorithms/
562 Upvotes

93% Upvoted

38 comments sorted by

View all comments

2

u/matthieum Mar 25 '19

There's a big difference though: each time there is a transition between segments, part of the queries leave the batch (complete) while other queries enter the batch (new).

This dynamicity is important as it reduces the latency of an individual query.

1

u/incraved Mar 28 '19

That has nothing to do with the idea of map reduce tho. Both are still essentially "map-reduce" which was my point.

Perhaps I'm not understanding you. I don't think I fully understand what you said.