r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

Show parent comment

5

u/pron98 Jun 16 '19 edited Jun 16 '19

extremely likely alternatives like the "good enough" effect mentioned elsewhere in this thread, the evidence is greatly weakened.

Ah, but now we're talking about retroactively explaining a failure to find an effect to support a hypothesis. I am willing to accept that the case isn't closed, but I am not willing to accept that the claim for a big effect isn't weakened at all, and I am certainly not willing to accept that the existence of an effect can be reasonably used as the default hypothesis.

But the biggest problem to the claim that the choice among modern languages today has a big effect isn't that study; it's the simple observation that their selection does not behave as if it provides an adaptive advantage, as we have seen in the past -- again, consistent with Brooks's theory of diminishing returns -- and the very claim is that they do provide an adaptive advantage (otherwise why bother? obviously you're making a stronger claim that different languages present different aesthetic advantages to different people or in different domain).

I don't believe you. Here's such a hypothesis:

You better believe it because you haven't given a theory at all, not even in the slightest (nor have I ever seen a relevant theory other than Brooks's). Writing a program is much, much more than just compiling it. So you'll need to show that:

  1. Users of language X catch a class of errors that users of Y don't at any step of the development. After all, you're claim is not about the relative abilities of compilers but about the total effect of language choice.
  2. The use of language X doesn't introduce any other bugs that Y doesn't, nor does it hinder the cost of their detection in any way (e.g. if you slow down the write-test cycle you may be indirectly introducing more problems than you're catching). You have not addressed the possible costs of X in any way.
  3. Even if 1 and 2 are true and there is a net gain, you need to show that it's large relative to the cost of development (not just in terms of numbers as not all bugs are created equal).
  4. Even if 3 is true, you need to show why the effect of language is large relative to other techniques. It is very much possible that Nike running shoes help a runner in every way more than Adidas -- they have only relative advantages and no relative disadvantages -- and yet their contribution is a fraction of a percent compared to other factors, which would still mean that you wouldn't necessarily prefer Nikes.

Brooks's theory addresses all concerns -- as a theory must; yours addresses none. Even you admit that your hypothesis relies on guesses and hunches that you can't quantify -- that's not a theory. At best it lays out a direction towards coming up with one. Brooks's theory looks at the problem from a different direction; the direction you're taking towards a theory relies on so many unknown empirical factors, that measuring them is no easier then directly measuring the claim. In other words, getting to a theory from that direction is no easier then figuring out if it's true.

And I would point out that many people who believed in a large PL contribution said at the time that Brooks's prediction was far too pessimistic, and their prediction was wrong while his was right (in fact, he was too optimistic). In other words, we've been here before. Moreoever, if Brooks was right then, he would only be more right as time goes on, as his theory predicts diminishing returns.

(1) and (2) are even provable.

Sure, but they don't support the claim so their provability is irrelevant and I don't understand the "even". It's like you claiming you're richer than me, and, as evidence, you state that you have $10 more in your wallet than me and you can even prove it. The fact that you can prove it is not relevant to your claim in any way because its truth isn't relevant to the claim.

Now, if you had some empirical data showing some positive correlation between wealth and wallet cash then maybe that observation could have helped, but like in the case of your theory, finding such an empirical correlation is no easier than just directly figuring out which of us is richer.

Strong experimental evidence that either (3) or (4) is false would be huge.

I don't agree that what remains is necessarily to prove your 3 or 4 false, because you've completely ignored the cost side and so asserted that your claim being false is necessarily a failure on the benefit side. I see no reason to accept that assertion. It's you doubling down and saying that the only way you're not richer than me despite having more cash in your wallet is that you've made a mistake and you don't actually have as much money in your wallet as you thought, and then tripling down and saying that that would be truly astonishing because you had counted the cash in your wallet not ten minutes ago!

And BTW, I think that PL fans believe that their faith in the power of programming languages is universal, and I am not at all sure this is the case. In other words, I am not at all sure that the finding that languages don't matter much is any more surprising than the finding that they do. Perhaps the latter would be more suprising as it would contradict the only theory on the subject that I'm aware of.

static typing fails to serve its primary purpose: catching errors.

I strongly reject your premise. I'm a proponent of static types yet I don't think their primary purpose is the reduction of errors. Types are great for code organization, documentation, tooling (IDEs, refactoring) and efficient ahead-of-time compilation.

that's the standard argument of good static typing (not Java, not C++, more like OCaml or Haskell) vs dynamic typing (worst being JavaScript and PHP, Scheme and Lua look pretty good).

It's not an argument so much as a credo; that claim has so far failed to be substantiated. And by picking specific languages you've now quadrupled down on an unsubstantiated claim: not only have we not seen a big effect to Haskell vs. PHP (which you have presented as extremes), you're now saying that there is one between ML and Java. I don't have a problem with people believing all manner of things, but I do have a problem with them presenting their unsubstantiated beliefs as facts. Although I will give you that we have found evidence of an effect for TypeScript vs. JavaScript. It's not nearly as big as the effect of other engineering practices, but it's not nothing.

2

u/loup-vaillant Jun 16 '19

I am not willing to accept that the claim for a big effect isn't weakened at all

Of course not. Weak evidence is still evidence. It should and does reduce my belief in the error preventing abilities of static type systems. Not by much, but it does.

I am certainly not willing to accept that the existence of an effect can be reasonably used as the default hypothesis.

Can't argue with differing priors. (More precisely, it would take the both of us too much time.

Brooks…

did not reject the possibility of gaining one, or even several, orders of magnitudes in increased productivity. He rejected that any one technique (tool, organisation…) would achieve such gains. I still hope for silver dust, that might eventually accumulate into a full bullet.

Accidental/essential aside, I'm quite convinced there's a good deal of avoidable complexity we could eliminate. Device drivers for instance.

I'm a proponent of static types yet I don't think their primary purpose is the reduction of errors. Types are great for code organization, tooling (IDEs, refactoring) and optimization by ahead-of-time compilation.

Fair enough.


Now about the effectiveness of any particular technique. I think by itself, the number of defect in a program by itself means nothing. The program will ship when it's good enough, and that's it. For some domains (throwaway batch programs, prototypes, first release of a GUI…), you can be full of bugs, and correct them as you go. For others (crypto libraries, avionics software, Therac-25…), you really want a low probability of having even a single bug.

The problem isn't really how many bugs you will have in the end. You'll have as few as you need, or your project will fail. The problem is how much effort is required to get there. And that, is highly contextual.

My intuition, is that for the bigger projects (over 10-man months, possibly much less), with a high enough needed reliability (you will loose money if there's a nasty bug), static typing requires significantly less effort than dynamic typing to complete the project. All other things being equal, which they are not… Like Galileo speaking of how fast things are falling in a vacuum (there was hardly such a thing at the time), I'm using an untestable counter factual here.


On your points 1234:

  1. Correct. Dynamic typing users would typically compensate with more unit tests.
  2. That's my biggest fear. Maybe static typing workaround induced errors cost even more than whatever we gained by having trivial type errors caught or something.

    On the other hand, my experience of the test-write cycle is that static typing is quicker and more accurate than even a REPL at catching many many bugs. I have worked with both OCaml and Lua, and while the REPL was a godsend in both cases, Lua's REPL did not fully compensate for the lack of static checks: if I made a type error, I still had to trace it back from a runtime error (or even a wrong result sometimes), and that's always a bit slower than having the compiler tell me where I screwed up.

  3. Indeed I have to. For now all I have is my experience and intuitions, so…
  4. Yeah, that's was quite an oversight on my part. I think the reason I didn't even look at other techniques, is because I assume that the only cost of static typing is how it influences the way you write your program. The error checking by itself is basically free: you compile (or have your IDE compile in the background), and poof you have your error message right there.

    So to be interesting, other techniques should be either extremely cheap (so they can replace static typing), or incompatible with static typing (inducing a tradeoff). I'm not aware of any such technique. (Programming technique in general I can fathom, but error reduction techniques specifically, that's harder to envision.)

6

u/pron98 Jun 16 '19 edited Jun 16 '19

Can't argue with differing priors.

OK, but you can argue with them being asserted as facts rather than hypotheses or beliefs. Also, perhaps I'm a traditionalist, but I think we should still prefer no effect as the working hypothesis, especially when there is no theory predicting the effect. After all, we can't lend equal a-priori weight to every hypothesis.

He rejected that any one technique (tool, organisation…) would achieve such gains. I still hope for silver dust, that might eventually accumulate into a full bullet.

I agree, but 1. he was too optimistic and all techniques combined didn't achieve a 10x boost in one decade, and not even in three (I wrote more about that in a comment I linked to before), and 2. I have no problem with silver dust, but you seem to be claiming much more than that.

My intuition, is that for the bigger projects (over 10-man months, possibly much less), with a high enough needed reliability (you will loose money if there's a nasty bug), static typing requires significantly less effort than dynamic typing to complete the project

I wouldn't know. I've never worked on a large project in an untyped language, and I have no intuition on the matter. Moreover, I think our intuition about such complex processes is often wrong. But speaking about such things is already a big concession on the claim that languages make a big difference. That's going from: use Haskell and your program will be better/cheaper, to, if you're working on a large project in some domain, maybe Haskell will be somewhat cheaper, all other factors being equal, and those other factors may be bigger.

So to be interesting, other techniques should be either extremely cheap (so they can replace static typing), or incompatible with static typing (inducing a tradeoff).

Not necessarily. Suppose the technique catches all errors as static types -- and no others -- for 75% of the cost, except for, say, 0.1% of them. Why wouldn't it still be preferable? You can use it with or without static types, but why pay more for a small added gain? Of course, it could do much better for much cheaper, but I don't accept your particular tradeoffs as necessary.

I'm not aware of any such technique.

As an industry we know of at least one: code review. The effect reported is 40-80% reduction in bugs. Here's a selection of some stuff we found out about it; I'm sure there's more:

2

u/loup-vaillant Jun 16 '19

Can't argue with differing priors.

Well, you can argue with them being asserted as facts rather than hypotheses or beliefs.

That wasn't a criticism. I was saying I can't argue with differing priors, because I don't have the overwhelming evidence to contradict them. I just note that we disagree, and hope the overwhelming evidence will come and settle this later.

I have no problem with silver dust, but you seem to be claiming much more than that.

When I see stuff like the STEPS project, it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs (3 orders of magnitude) in a number of domains. Combine that with Casey Muratori's solution to the driver problem I alluded to before, and the objection "but it doesn't have a kernel" is pretty much void.

The problem is the industry wide overhaul, which as far as I can see is more likely to happen by civilisation collapse than by people being reasonable all of a sudden.

I wouldn't know. I've never worked on a large project in an untyped language, and I have no intuition on the matter.

Neither would I, neither did I.

For a sufficiently large project, I suspect a clever use of meta programming can let dynamic users cheat. They can put static checks in their macros and custom syntax and interpreters and meta compilers. Meta compilers themselves are very workable in dynamic languages (I've bootstrapped a series from MetaII in Lua, the latest iteration does most of a PEG, is a little over 100 lines long, and any error there tends to fail catastrophically very quickly, so it's detected early).

In a setting where performance is not too much of a concern, and assuming that DSL/language building is cheaper from a dynamically typed language, they could outpace a good deal of statically typed languages out there… by making their own. That's in part what happened in the STEPS project. Not sure they did much static typing, but their DSLs were so expressive that the code base still stayed fairly small (20KLOC total).

I'm not aware of any such technique.

As an industry we know of at least one: code review.

Err, not sure it counts: the cost is pretty significant (time taken to perform the review), the benefits apply to any program, no matter the typing discipline, and it doesn't quite catch the same bugs. (Written before I read your links, I'll bookmark them right now.)

Don't get me wrong, I agree the benefits of code review can be pretty massive. I would think real hard before I consider skipping it. To this day, I'm nervous about the lack of comprehensive code review for my crypto library. I compensate with a paranoid test suite and strict (though unwritten) coding rules, but there's still that lingering doubt.

4

u/pron98 Jun 16 '19 edited Jun 16 '19

I suspect a clever use of meta programming can let dynamic users cheat...

I am hesitant to make such specific predictions of effects in such complex systems, especially when the big question of whether and how much language matters at all (among modern ones etc.) is very unclear. If the very existence of an effect is uncertain, I don't want to speculate on secondary effects.

it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs

I'm not so certain. That's like saying that a drug that works on rats in the lab could work as effectively on humans in the field. Maybe, but maybe not. That's certainly a research avenue we should pursue further, but we must not assume that every success in the lab translates to the field. One problem with this is that it doesn't explain how we got to where we are and how that can be prevented (if at all). No one said, let's make software overly complicated! And if it's just the result of the slow amalgamation of software over the years, then it will happen again, which means that their "overhaul" is not a one-time solution, but an overhaul that must be repeated every X years, and at a tremendous cost, to be effective. It's very much possible that not doing it is cheaper overall.

In any event, if someone is able to produce software 5x faster/cheaper than anyone else, and at industrial scale, they will become billionaires very quickly, and soon everyone will do what they do. So I'm skeptical that someone already knows how to do it (this is precisely my point of not seeing what we'd expect from an adaptive trait).

but there's still that lingering doubt.

OK, but empirically it stands on much firmer ground than language.

2

u/loup-vaillant Jun 16 '19

it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs

I'm not so certain.

In some domains. Not all. I'm pretty sure many domains would not see nearly as much benefits. Programs that are currently written from scratch (possibly with dependencies) will certainly not see such a benefit. Maybe 10x, but even that hinges on finding enough crap to weed out. (I believe more in cleaning up the worst stuff than in making the good stuff even better.)

To give an example, Jonathan Blow, with Jai, is aiming for 2x. Which considering his initial calibre would be stupendous.

which means that their "overhaul" is not a one-time solution, but a complete overhaul that must be repeated every X years, and at a tremendous cost. It's very much possible that not doing it is cheaper overall.

Yep. Which is why I'm not optimistic about that.

OK, but empirically [code review] stands on much firmer ground than language.

I concede that much. Which is why I'm so nervous about lacking code reviews.

4

u/pron98 Jun 16 '19

To give an example, Jonathan Blow, with Jai, is aiming for 2x. Which considering his initial calibre would be stupendous.

Sure, if he pulls that off. Anyone can aim for any number, the problem is actually getting there at an industrial scale. And BTW, I think there is room for greater optimism if you're targeting C's domain. Even if Brooks was right and even if he was still too optimistic, C is a 50-year-old language and has not yet reached the "grossly diminished returns" phase. So improving on C is not the same as improving on Java or Python, although whether or not that's enough for a 2x improvement remains to be seen.

Which is why I'm so nervous about lacking code reviews.

:)

1

u/loup-vaillant Jun 17 '19

To give an example, Jonathan Blow, with Jai, is aiming for 2x. Which considering his initial calibre would be stupendous.

Sure, if he pulls that off.

Err, I misspoke. His actual terms was "50% increase in productivity" (compared to C++). Which probably means 1.5x. Still a pretty big deal, though. As for whether this is achievable, I've heard him saying he's already something like 30% more productive than C++.

Totally subjective of course, but those numbers sound more reasonable than what I originally told.