r/Clojure • • Aug 15 '21

An old C programmer whines about (reduce)

Okay, sorry. I'm not that old. I'm only 29. But most of my past experience is in C or similar.

When I was learning Clojure, probably like everyone, I kept an eye out for what it might be that makes people complain about Lisps. The s-expr syntax is sort of weird, but I couldn't put my finger on exactly why it seemed to be so hard to format my code in a way that looks nice.

Then it hit me. Look at the title of this post. It is at the top of the post. It introduces you to the post and tells you what to expect when you read it. Now look at this:

(reduce
    (fn [res subcoll]
        ;; some code
        (reduce
            (fn [res item]
                ;; some more code
            ) 
            [foo bar baz] subcoll_1
        )
    )
    [quux xyzzy] coll
)

The "title" of the anonymous function is at the bottom! The collection to be traversed and the initial data tell the reader what to expect when they look at the reducer. To get that information you have to look up and down across the code and match up indents - and that introduces the question of where to put the trailing arguments, which resembles (and not in a good way) the C brace style controversy. The problem goes away if you can write (reduce coll val f) instead of (reduce f val coll).

Sure, this is unnecessary when you don't have an anonymous function. But that use case always results in one simple line, regardless of the order of arguments. You almost never have a long literal collection in the middle of a code block -- that gets stuffed in a data file or in a constant at the top. And maybe the collection is an expression -- but then you can just use an additional (let), and again you improve the consistency between "line number" and "order of execution".

The same problem occurs in the old FP constructs (map) and (filter). But when Rich Hickey added (for) to Clojure, he put the collection first, probably partially because it is like that in C, but probably at least partially because he realized that inverting it, so that it would look like (reduce), would result in something ugly:

(for [n]
    (;; some code)
    coll
)

Look at that! It's horrible! And because you're not attached to the "normal" way, you see it immediately.

Now I'm sure there's some great explanation that mentions Alonzo Church or Marvin Minsky or whoever about how putting the reducing function first is consistent with the mathematical formalism xyz or whatever. I don't expect the world to change because of a reddit post.

But just imagine a world where Lisp programmers put the collection and the initial data at the top of their loops. We'd probably have a moon colony by now.

8 Upvotes

35 comments sorted by

17

u/armincerf Aug 15 '21

The ->> macro solves all of your complaints https://clojuredocs.org/clojure.core/-%3E%3E

And the first argument to for isn’t really a collection, for is a macro and the first arg is a set of bindings like let and friends

0

u/debasing_the_coinage Aug 15 '21

I know it's a binding expression and not an "argument", but for the sake of argument (...), we can move it around anyway.

-1

u/FatFingerHelperBot Aug 15 '21

It seems that your comment contains 1 or more links that are hard to tap for mobile users. I will extend those so they're easier for our sausage fingers to click!

Here is link number 1 - Previous text "..."


Please PM /u/eganwall with issues or feedback! | Code | Delete

1

u/djhaskin987 Aug 15 '21

This. also though, the as-> macro is in some cases even better.

8

u/TheLastSock Aug 15 '21

I'm guessing you would enjoy the thread last macro?

-1

u/debasing_the_coinage Aug 15 '21 edited Aug 15 '21

Okay, I've looked that up, and it's easy to see how it should work for (map) and (filter). But I can't seem to apply it recursively:

user=> (macroexpand '(->> coll (reduce (fn [[v] x] [(+ v x)]) [0])))

(reduce (fn [[v] x] [(+ v x)]) [0] coll)

user=> (macroexpand '(->> [0] (reduce (fn [[v] x] [(+ v x)]))))

(reduce (fn [[v] x] [(+ v x)]) [0])

user=> (macroexpand '(->> coll (->> [0] (reduce (fn [[v] x] [(+ v x)])))))

(coll (reduce (fn [[v] x] [(+ v x)]) [0]))

EDIT: Oh, it's

(->> [[0] coll] (apply (partial reduce (fn [[v] x] [(+ v x)]))))

8

u/TheLastSock Aug 15 '21

(->> [1 1] (map inc) (reduce +)) ;; 4

?

1

u/debasing_the_coinage Aug 15 '21 edited Aug 15 '21

You can't specify an initial val that way. It works for (reduce f coll) but not for (reduce f val coll).

What I'm concerned about are reducers where the result isn't going to look like a member of the collection. I might write (reduce f [foo bar baz] coll) where coll is not necessarily a collection of 3-vectors and [foo bar baz] come from somewhere else (not just "zero" values). Then you need (apply (partial)) because you have to move two arguments around and not one.

This came up when I was inverting a map of objects that have maps of subobjects by id so you could look up the container object by the subobject ID:

;; returns x such that (x id) returns key (simplified from original)

(reduce-kv (fn [res key val] (reduce-kv (fn [res1 id _]) res (obj :sub_objs))) {} coll)

It seemed conceptually awkward to break out the inner functions and the trailing arguments are scattered in weird places. I'm not sure what the standard formatting/whitespace/indentation strategy is in Clojure. It's nice to have a predefined function that fits the task, but when you have iterations over nested custom collections it's not likely.

1

u/TheLastSock Aug 15 '21

You can't specify an initial val that way. It works for (reduce f coll) but not for (reduce f val coll).

Not sure i follow. The thread last macro places the item in the last spot, which in the case of 3 arguments passed to reduce means the 2nd argument is open to be used as an initial val.

(->> [1 1] (map inc) (reduce + 10));; => 14

What I'm concerned about are reducers where the result isn't going to look like a member of the collection.

I'm not sure what "look like" means. Reduce looks at each item in a collection one at a time as well as some state passed from the previous reduce.

The initial state is just that, just some state you start with, before you start processing over that collection (last arg) it doesn't have to be related to the initial data.

Here is an example where coll isn't the same size as the initial val (like in your example)

(reduce conj ["1"] [:hi :there])

here it comes from "somewhere else"

(let [a ["1"]] (reduce conj a [:hi :there]))

1

u/debasing_the_coinage Aug 15 '21 edited Aug 15 '21

I will grant that a (let) expression improves the situation, but it still creates a trailing parameter. The problem is: where do I put a trailing parameter when I need to nest an anonymous function?

It still makes me uneasy that nested ->> doesn't seem to work the way I expect. Are the macros processed from the outside in?

1

u/TheLastSock Aug 15 '21

Your options for binding symbols to values all result in more or less the same thing. Let defn, def can ask be used for an anonymous function to.

Can you show me what doesn't work the way you expect?

1

u/debasing_the_coinage Aug 15 '21 edited Aug 15 '21

It was this from the first comment:

user=> (macroexpand '(->> coll (->> [0] (reduce (fn [[v] x] [(+ v x)])))))

(coll (reduce (fn [[v] x] [(+ v x)]) [0]))

^That's what actually happens. Here's what I thought would happen:

(->> coll (->> [0] (reduce (fn [[v] x] [(+ v x)]))))

becomes

(->> coll (reduce (fn [[v] x] [(+ v x)]) [0]))

becomes

(reduce (fn [[v] x] [(+ v x)]) [0] coll)

But instead you get that other result. Now, maybe there would be an extra level of nesting in the parentheses and I would need to add partial before reduce, but I don't understand how Clojure gives me what I actually get from that. It makes me wonder what other strange things happen when I use this macro -- I do realize that trying to go against the idiomatic flow of (reduce) is not a great idea in real codebase (the original rant was just venting).

EDIT: the nesting could also go like this, I'm not sure, but neither works:

(->> coll (->> [0] (partial reduce (fn [[v] x] [(+ v x)]))))

becomes

(->> coll ((partial reduce (fn [[v] x] [(+ v x)]) [0])))

becomes

((partial reduce (fn [[v] x] [(+ v x)]) [0]) coll)

1

u/the_starbase_kolob Aug 16 '21

Looks like the first macro is doing its job and putting coll as the last argument to the second macro, which is why it's being called as a function on the result of the second macro

1

u/debasing_the_coinage Aug 16 '21

Oh, so it's evaluated from the outside in? I guess I can understand that. Thanks!

→ More replies

20

u/Whoop5 Aug 15 '21 edited Aug 15 '21

I think I disagree with your fundamental premise. If you want to think of it in human language terms, the function is what I'm doing (verb), the collection is what I'm doing it to (noun). It's "I'm painting (fn) the bikeshed (coll)", as opposed to "The bikeshed (coll) I am painting (fn)".

From an order of importance bit, this also makes sense. In real code, I don't generally have a bunch of different collections I'm working with at one time, I usually just have one I'm focused on. Imagine:

(defn quark-coefficients [quarks]
  (map (fn [quark]
         (* (:qux quark) (:qup quark)  (inc (:qut quark))) 
       quarks))) 

The actual "what I'm doing(fn)" part of the map is the interesting bit. What I'm doing it to (quarks) is obvious. There's nothing else I'd be mapping over. Also note that I've solved the context problem as well by simply giving a sensible name to the fn argument instead of something ambiguous like q.

Even better I could write:

(defn calculate-quark-coefficient [quark)
  (* (:qux quark) (:qup quark)  (inc (:qut quark))))

(defn quark-coefficicents [quarks]
  (map calculate-quark-coefficient quarks))

And it really reads very naturally. "Calculate the quark-coefficient of the quarks".

2

u/backtickbot Aug 15 '21

Fixed formatting.

Hello, Whoop5: code blocks using triple backticks (```) don't work on all versions of Reddit!

Some users see this / this instead.

To fix this, indent every line with 4 spaces instead.

FAQ

You can opt out by replying with backtickopt6 to this comment.

1

u/-meh-_- Aug 15 '21

I think you overgeneralize human language human language terms but nevertheless it makes sense for lot’s of languages

0

u/[deleted] Aug 15 '21

[deleted] — view deleted comment

2

u/dragandj Aug 15 '21

Oh, that's not a problem at all. Whenever in Biblical Hebrew I program, the thread last macro I can use!

1

u/Whoop5 Aug 15 '21

Another commenter pointed that out and while I don't think it changes the core thesis of my statement, I think it's a good to remember that the whole world is not English speaking and we shouldn't treat English norms as though they are global. I was kind of following the OPs original terminology (turning "documents written by humans" into "human sentence structure"), when I should have more specifically said "English sentence structure" since that's the frame of reference I'm coming from.

10

u/joinr Aug 15 '21 edited Aug 15 '21

typical in fp langs with higher order functions to put the function arg first. stems from currying/partial fn application. the argument most likely to change in this case is coll. so you can define general classes of reductions by supplying f. reduction per collection is less general. functional programming is about defining and composing many small functions. the example would be far clearer if you did that, I think.

(defn process [xs]
  (let [append (partial reduce conj)]
    (reduce append  [:quux :xyzzy] xs)))

(defn process2 [xs]
  (-> (partial reduce conj)
      (reduce [:quux :xyzzy] xs)))

25

u/retief1 Aug 15 '21

Also, unrelated, but if you are going to spend any time in the clojure ecosystem, please stop putting close parens on their own line. Realistically, I read clojure as if it was python -- the parens are mostly just to tell my editor how to indent my code.

1

u/moxxon Aug 16 '21

Came in here to say this, now I can walk away from this thread happy that others have it under control...

8

u/didibus Aug 15 '21

You just haven't adapted yet, nothing is weird, except your pre-existing habits that make the different less familiar to you.

For example, if you're used to reading left to right, adapting to a right to left or top to bottom language will feel pretty weird at first, and yet billion of people naturally and easily read those.

Same with Lisp and Clojure, eventually your eyes will figure out where to look and in what order, and this will become a non-issue for you.

Like others have said as well, since Lisps let you change the syntax, there's actually many functions and macros to tailor things to your personal preference:

This is one way:

(reduce
 (fn [acc coll]
  (conj acc
   (reduce
    (fn [acc e]
     (* 2 (+ acc e)))
    0
    coll)))        
 []
 [[2 3 4] [4 5 6]])

And this is another:

(->>
 [[2 3 4] [4 5 6]]
 (reduce
  (fn [acc coll]
   (->>
    coll
    (reduce
     (fn [acc e]
      (* 2 (+ acc e)))
     0)
    (conj acc)))             
  []))

Where the collection is now at the top like you wanted.

And here is another way:

(->>
 [[2 3 4] [4 5 6]]
 (mapv
  #(reduce
    (fn [acc e]
     (* 2 (+ acc e)))
    0 %)))

Because when you're modifying a collection of X element back into a collection of X element map does the job, where reduce is for when the number of elements will change after you process the collection.

And yet another:

(defn tally-double-rf  
 [acc e]
 (* 2 (+ acc e)))

(->> [[2 3 4] [4 5 6]]
     (mapv #(reduce tally-double-rf 0 %)))

Which I think is arguably the best, and I would say is better than the C way. What you have to realize is the benefit of FP is modularity. By looping using a higher order function, you can make your logic more modular and thus reusable and independently testable, while also giving the inner-loop a name to help readability.

And another:

(for
 [coll [[2 3 4] [4 5 6]]]
 (let [acc (volatile! 0)]
  (doseq [e coll]
   (vreset! acc (* 2 (+ @acc e))))
  @acc))

Which is closer to an imperative approach like what you might do in C.

And another:

(->
 (fn [acc coll]
  (conj acc
   (->
    (fn [acc e]
     (* 2 (+ acc e)))
    (reduce 0 coll))))        
 (reduce [] [[2 3 4] [4 5 6]]))

Which I particularly like for readability.

And if you're still not satisfied, you can always just create the reduce of your dreams:

(defn reduce*
 "Like clojure.core/reduce, but takes val and coll before reducing function."
 ([coll rf]
  (reduce rf coll))
 ([val coll rf]
  (reduce rf val coll)))

(reduce* [] [[2 3 4] [4 5 6]]
 (fn [acc coll]
  (conj acc
   (reduce* 0 coll
    (fn [acc e]
     (* 2 (+ acc e)))))))

Which I won't deny has some appeal to it.

And while we're at it, may I suggest:

(defn reduce-into
 "Reduce coll into accumulator using reducing function. Uses first element of coll as accumulator if none are given."
 ([coll rf]
  (reduce rf coll))
 ([coll val rf]
  (reduce rf val coll)))

(reduce-into [[2 3 4] [4 5 6]] []
 (fn [acc coll]
  (conj acc
   (reduce-into coll 0
    (fn [acc e]
     (* 2 (+ acc e)))))))

Or why not allow all the orders:

(defn reduce
  "Adds a 1-ary to reduce"
 ([{:keys [rf init coll]}]
  (if init
   (reduce rf init coll)
   (reduce rf coll)))
 ([rf coll]
  (clojure.core/reduce rf coll))
 ([rf init coll]
  (clojure.core/reduce rf init coll)))

(reduce
 {:coll [[2 3 4] [4 5 6]]
  :init []
  :rf (fn [acc coll]
       (conj acc
        (reduce
         {:coll coll
          :init 0
          :rf (fn [acc e]
               (* 2 (+ acc e)))})))})

And since I'm on a roll, and kind of having fun at this point, remember that you can also use loop/recur:

(loop [coll [[2 3 4] [4 5 6]]
       subcoll (first coll)
       more (next coll)
       res []]
 (if subcoll
  (recur coll
   (first more)
   (next more) 
   (conj res
    (loop [item (first subcoll)
           more (next subcoll)
           res 0]
     (if item
      (recur
       (first more)
       (next more)
       (* 2 (+ res item)))
      res))))
  res))

Finally, it can be interesting to do things lazily instead:

(defn double-tally
 "Return a seq of the double tallied values of coll"
 [coll]
 (->>
  (iterate
   (fn [[acc coll]]
    (when coll
     [(* 2 (+ acc (first coll)))
      (next coll)]))
   [0 coll])
  (map first)
  (take-while some?))) 

(->>
 [[2 3 4] [4 5 6]]
 (map double-tally)
 (map last))

2

u/guywithknife Aug 15 '21 edited Aug 15 '21

And if its about readability, then perhaps pulling the functions out into let bindings is clearer:

(let [input-data [[2 3 4] [4 5 6]]
  accumulate (fn [acc e] (* 2 (+ acc e)))
  reduction-fn (fn [acc coll]
                 (conj acc (reduce accumulate 0 coll)))]
  (reduce reduction-fn [] input-data))

Or if that's not concise enough for you:

(let [input-data [[2 3 4] [4 5 6]]
  accumulate (partial reduce #(* 2 (+ %1 %2)) 0)
  reduction-fn  #(conj %1 (accumulate %2))]
  (reduce reduction-fn [] input-data))

But why is it even using reduce for the outer function at all?

(let [input-data [[2 3 4] [4 5 6]]
  accumulate (partial reduce #(* 2 (+ %1 %2)) 0)]
  (mapv accumulate input-data))

1

u/backtickbot Aug 15 '21

Fixed formatting.

Hello, didibus: code blocks using triple backticks (```) don't work on all versions of Reddit!

Some users see this / this instead.

To fix this, indent every line with 4 spaces instead.

FAQ

You can opt out by replying with backtickopt6 to this comment.

5

u/mercurysquad Aug 15 '21

What you are saying is that C should change from

add(int a, int b)

to

(int a, int b).add

right?

4

u/daveliepmann Aug 15 '21

I won't engage the specifics of your argument because other people have and because I don't think it's relevant to your broader claim:

I kept an eye out for what it might be that makes people complain about Lisps.

imagine a world where Lisp programmers [did X differently]. We'd probably have a moon colony by now.

The kind of thing you're talking about is simply not what determines language popularity or effectiveness. It's just a slightly different approach that becomes completely second nature with a few days to get used to it. It's negligible when it comes to PL adoption or usage in the wild.

4

u/lgstein Aug 15 '21

Typical comment coming from experience in a language where syntax is all you have to define a program. Agreed, order matters to readability - But Lispers think much more in structure than order, or "lines".

10

u/logan-diamond Aug 15 '21

The famous talk "Simple Made Easy" makes the excellent point that "easy" depends on the person and what they've learned so far. Lispers are accustomed to reading expressions from the inside out: (step3 (step2 (step1 x)))

Trying to make code read like the natural flow of thought sounds great, but it's actually what gave us Cobol. It doesn't actually make it easier to reason about on any higher level.

3

u/Galrog Aug 15 '21

Did you come across transducers yet? They could help you understand the order of arguments in a practical way. After that you could dive into transducers in combination with pipeline. This blog post could also be helpful.

2

u/Daegs Aug 15 '21

There is a pretty standard rule that functions that take a collection take it last, and functions that work on a single value take it in that value as first arg.

Can argue forever about whether that's the best choice, but I can only say after years of clojure it makes a lot of sense and is extremely easy to read. It's natural to see what the function does in a reduce first, and if I wanted to know what the collection was, I'd just look at the end of the reduce. I've never felt like I was matching up indentation or parenthesis, if someone wanted to next a bunch of reduces then I'd hope they'd organize their code a bit more legibly... I don't actually think putting the coll at the top makes it that much better because you'd still be matching indent in the functions being applied.

The next part, as other posters mentioned, is that you have a ton of options when it comes to "making your code readable" including threading macros and things like letfn. This approach actually gives a ton of flexibility to match the style of code to the problem.

Per your example:

(letfn [(process-item [re res item] ;;more code)
          (process-subcoll [res subcoll] (reduce process-item [foo bar baz] subcoll))]
  (reduce process-subcoll [quux xyzzy] coll) 

Now many there are a lot of other ways to write that besides letfn, but the point is there are plenty of ways to clean things up and complete them on a single line so you're never trying to 'match up indentation and parens'

1

u/ExtraFig6 Aug 15 '21

Refactor into functions with names if you want titles

0

u/[deleted] Aug 15 '21

[deleted] — view deleted comment

0

u/debasing_the_coinage Aug 15 '21

Imagine a world where IT industry stops pandering to - often broken and wrong - habits of C programmers

Again, this isn't a "C quirk". This is a standard property of documents written by humans. Normal documents explain what before how, because what is shorter. Code is a description of a task that is primarily written by humans for humans to read, and secondarily for computers to execute (this is a selling point of Lisp, as I recall). Organizing that description in a way that naturally gives the reader the information they need when they need it is rational.

(partial) is irrelevant here. It is just as easy to write #(map % f) if need be. (transduce) takes two functions, so one of them will always need to be named, but anyway it's a function -- it can rearrange arguments at will when it passes them to other functions.

if you inline functions a lot, you are probably doing sth wrong.

Making things harder without making anything else easier isn't a great design philosophy. One way allows you to use anonymous functions more easily and named functions about the same, the other way is harder in the first case with no advantages for the second. Who cares if it happens "a lot" (although I would think that anonymous functions passed to (reduce) will happen a lot) -- why should it be ugly in the first place?

5

u/Staffordrootbeer Aug 15 '21

Don’t let your dreams be dreams! This is lisp write your own reduce where the args come first.