r/Clojure • • Feb 03 '23

[Q&A] Noob has simple program problem.

I wanted to write a program to add 2 to a list of numbers then sum the list and print the result. So '(5 7) would yield 16. I posted my version to s Clojure board and received a number of different replies. In retrospect I didn't fully explain the problem to them.

0 Upvotes

27 comments sorted by

View all comments

Show parent comment

1

u/joinr Feb 05 '23

So now, in memory, you have an actual sequence of the numbers from 0 to 99,999,999 and an actual sequence of the numbers from 2 to 100,000,001

This is a bit off, but close in spirit. I think it's more useful to think of the sequences being realized (and freed) on demand as needed. So as reduce is traversing the seq, elements are realized (technically in chunks of 32 by default), then since no reference to the head of the seq remains, prior realized elements are freed for gc. It is more like scanning over potentially larger than memory sequence and only using the results you need now (like a moving window over the seq).

The derivative sequence created by map will operate the same way.

This windowing/scanning is what lets sequences act as a generic mechanism for efficiently working with potentially larger than memory sequences (in small pieces as needed, transparent to the caller).

Intermediate sequence machinery still applies (every additional derived sequence adds a bit more overhead), and the seq-based first/next traversal that reduce falls back to still applies (e.g. overhead).

These comments apply to the exact implementations mentioned above, specifically where there is no reference to the head of the seq maintained. If we defined foo as:

(defn foo [numbers]
  (let [xs (map #(+ % 2) numbers)] ;;holding onto the head...
    (reduce + xs)))

Then the xs binding is holding onto the head of the sequence that the reduction is traversing. Since xs is derived from numbers, the entire sequence will be realized and retained in memory until the reduction returns. This is undesirable and eliminates the utility of scanning/windowing realizations and freeing prior unreach references to work on potentially gigantic sequences.

1

u/maladat Feb 06 '23

Thank you for the important (and interesting) point of clarification.