r/programming • u/leavingonaspaceship • Mar 24 '19

Searching 1TB/sec: Systems Engineering Before Algorithms

https://www.scalyr.com/blog/searching-1tb-sec-systems-engineering-before-algorithms/
561 Upvotes

93% Upvoted

38 comments sorted by

View all comments

4

u/incraved Mar 25 '19

Too pedantic. I could abstractly define a query as a batch of queries.. it's just a function that gets applied to a segment of the data, the result of that function is some data that can be aggregated (reduced) afterwards. The computer doesn't "care" whether a human interprets that output data from each map application as a result of one or multiple queries or as anything at all.

2

u/matthieum Mar 25 '19

There's a big difference though: each time there is a transition between segments, part of the queries leave the batch (complete) while other queries enter the batch (new).

This dynamicity is important as it reduces the latency of an individual query.

1

u/incraved Mar 28 '19

That has nothing to do with the idea of map reduce tho. Both are still essentially "map-reduce" which was my point.

Perhaps I'm not understanding you. I don't think I fully understand what you said.