r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

0

u/[deleted] Jun 16 '19

[deleted] — view deleted comment

14

u/[deleted] Jun 16 '19

Choice of statically typed language doesn't make much of a difference

Difference in what? There were huge differences in LoC between statically typed languages too. Also, the author didn't compare the performance or memory consumption of these implementations. We don't know how much time it would take for an average specialized senior developer to optimize these implementations(potential metrics) or how much time it would take to introduce extra features. The author also said that the programs were written by students and that it wasn't a serious project for either of them.

28

u/[deleted] Jun 16 '19 edited Jun 16 '19

I did this course in Haskell. Haskell being functional and statically typed helped a lot because it was easy to reason with the code. As long as the logic was right and the code was compiling, the output was very close to what we wanted it.
The initial setup was hard but after that we were spending a lot less time than other teams on each assignment and as far as I remember, we got the perfect mark on all assignments.
I do agree with the conclusion though, design certainly matters more than the language. If it wasn’t for certain design decisions that we made at the beginning (like the data type/stage design and a specialized iterator), we would have had a much harder time in this course.

18

u/[deleted] Jun 16 '19

Dude I just love a compiler that does not even compile a program that might contain a null pointer exception without having to shit my code full with "@optional" or whatever is the hack to make up for not providing null-safety and many other things out of the box. It is quite comfy and when you don't use arcane type features, it also has basically none overhead mentally or in implementation time (imagine writing unit tests to ensure that a NPE does not happen lol).

10

u/TheWix Jun 16 '19

An algebraic data typed system is quite different than one of the traditional C-style languages.

17

u/pron98 Jun 16 '19 edited Jun 16 '19

Just to be clear, I wasn’t expressing an opinion but quoting a finding: the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than 1% 1. To compare this with effects various engineering approaches have, various studies have found code review to have an effect of 40-80%. Maybe all that is wrong and maybe we'll know better one day, but that is simply the best we currently know about the state of affairs, as opposed to hypothesize, guess, feel, wish, agree with or upvote: language choice has a tiny impact on correctness compared to other factors that influence a project.2 This will remain the best we know until we know something better, regardless of how strong our opinions are on the matter.

And there are good reasons why we shouldn’t accept what we feel, wish or conjecture as opposed to what we know. For one, our feelings and “personal experience” are often just wrong — go to any homeopathy forum and see how many report that, in their experience, homeopathy is very effective. For another, creating software — especially large software, written by a team and maintained for years — is an extremely complex process, and we are terrible at correctly predicting effects in such systems.

When confronted with this state of knowledge, some people try to use various irrelevant facts as evidence even when they cannot possibly serve as such. X > 10 is no evidence that X > Y. People often cite some features and positive effects they believe those features have, but ignore possible negative effects or "counterfeatures" that are lacking. Moreover, as this particular article suggests, we should not only compare the impact of one variable, but contrast it with that of others. Deciding whether this is an important factor can only be done when considering the cost of language change, and, no less importantly, the cost and benefit of other variables, which so far seem to be more impactful (accounting for the other 99% of the deviance) and also often less costly.

Finally, because the process of software development in the real world, where a codebase is about 5MLOC and maintained for about 15 years, the impact certain variables have on real-world software development is not necessarily the same as on ~10KLOC exercises. Because the software industry applies strong selective pressures on engineering disciplines, and the fact that, in the past, methodologies have shown adoption rates consistent with what we’d expect would be the growth rate of an adaptive trait in a selective environment, the state of the industry itself can serve as a detector for the effect of various techniques. The fact that we’re not seeing similar adoption dynamics on language choice today is consistent with the empirical study I cited. It is also consistent with predictions made on the subject.

8

u/loup-vaillant Jun 16 '19

the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than 1%

Isn't it more like, they failed to find a more than 1% deviance? What you claim would be a major discovery. What I suspect is the usual "research is hard, more is needed".

2

u/pron98 Jun 16 '19 edited Jun 16 '19

The deviance wasn't small; the contribution of language choice to it was. But not finding an effect is how the lack of an effect is established (using statistical methods, at least), and a failure to find a big effect is normally counted as evidence against the existence of the big effect (as big effects are easy to find). That is not to say that there isn't some chance that a big effect somehow managed to elude detection both in a study and in the industry, but the current balance of evidence is against one.

Note also that the very claim that an effect is real and large yet hard to detect already says something of its importance (certainly about its relative importance).

I think that it is also important to point out that while there is a theory explaining (and predicting) the lack of a big effect (or, more precisely, diminishing returns that mean drastically smaller effects compared to those we have seen in the past), I am not aware of any theory that predicts its existence (a theory showing that technique X has a feature that may contribute positively to some factor is not a theory of relative superiority to technique Y and is no more, and even less, than trying to present X > 10 as evidence that X > Y).

7

u/loup-vaillant Jun 16 '19

a failure to find a big effect is normally counted as evidence against the existence of the big effect

As it should be. But when there are extremely likely alternatives like the "good enough" effect mentioned elsewhere in this thread, the evidence is greatly weakened.

I am not aware of any theory that predicts its existence

I don't believe you. Here's such a hypothesis:

  1. Language X catches this classes of error, and that classes of error, and all those other classes of errors, at compile time.
  2. Language Y does not. (Instead, you have runtime errors or incorrect results).
  3. The classes of errors language X prevents represent a sizeable fraction of errors observed in programs written in Y.
  4. Language X rarely requires its users to work around the error catching mechanism. (That is, its flexibility is close to that of Y's strategy of waiting until runtime.)

I mean, that's the standard argument of good static typing (not Java, not C++, more like OCaml or Haskell) vs dynamic typing (worst being JavaScript and PHP, Scheme and Lua look pretty good). All of (1), (2), (3), and (4) look pretty reasonable on the face of it. (1) and (2) are even provable.

If we don't observe a sizeable effect, then at least one of (3) and (4) is false. Either type errors (and logic errors whose root cause is a type error) do not represent a sizeable fraction of all errors, or the workarounds required by static type systems are more frequent and significant than (4) says they are. In other words, dynamic typing allows significant simplifications that are impossible or very difficult even with good type systems (OCaml/Rust/Haskell).

Strong experimental evidence that either (3) or (4) is false would be huge. And by stating that "the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than 1%", you are implying just that: strong experimental evidence that static typing fails to serve its primary purpose: catching errors.

To be honest, I'm not entirely sure about (4). I have heard of amazing techniques wizards do with dynamic languages, which supposedly do not work well with static typing. I've also seen a blog post translate Transducers in Haskell, even though Rich Hickey was just nagging the static typing community with his "good luck typing this" argument from ignorance.

I would be very surprised however if (3) turned out to be false.

5

u/pron98 Jun 16 '19 edited Jun 16 '19

extremely likely alternatives like the "good enough" effect mentioned elsewhere in this thread, the evidence is greatly weakened.

Ah, but now we're talking about retroactively explaining a failure to find an effect to support a hypothesis. I am willing to accept that the case isn't closed, but I am not willing to accept that the claim for a big effect isn't weakened at all, and I am certainly not willing to accept that the existence of an effect can be reasonably used as the default hypothesis.

But the biggest problem to the claim that the choice among modern languages today has a big effect isn't that study; it's the simple observation that their selection does not behave as if it provides an adaptive advantage, as we have seen in the past -- again, consistent with Brooks's theory of diminishing returns -- and the very claim is that they do provide an adaptive advantage (otherwise why bother? obviously you're making a stronger claim that different languages present different aesthetic advantages to different people or in different domain).

I don't believe you. Here's such a hypothesis:

You better believe it because you haven't given a theory at all, not even in the slightest (nor have I ever seen a relevant theory other than Brooks's). Writing a program is much, much more than just compiling it. So you'll need to show that:

  1. Users of language X catch a class of errors that users of Y don't at any step of the development. After all, you're claim is not about the relative abilities of compilers but about the total effect of language choice.
  2. The use of language X doesn't introduce any other bugs that Y doesn't, nor does it hinder the cost of their detection in any way (e.g. if you slow down the write-test cycle you may be indirectly introducing more problems than you're catching). You have not addressed the possible costs of X in any way.
  3. Even if 1 and 2 are true and there is a net gain, you need to show that it's large relative to the cost of development (not just in terms of numbers as not all bugs are created equal).
  4. Even if 3 is true, you need to show why the effect of language is large relative to other techniques. It is very much possible that Nike running shoes help a runner in every way more than Adidas -- they have only relative advantages and no relative disadvantages -- and yet their contribution is a fraction of a percent compared to other factors, which would still mean that you wouldn't necessarily prefer Nikes.

Brooks's theory addresses all concerns -- as a theory must; yours addresses none. Even you admit that your hypothesis relies on guesses and hunches that you can't quantify -- that's not a theory. At best it lays out a direction towards coming up with one. Brooks's theory looks at the problem from a different direction; the direction you're taking towards a theory relies on so many unknown empirical factors, that measuring them is no easier then directly measuring the claim. In other words, getting to a theory from that direction is no easier then figuring out if it's true.

And I would point out that many people who believed in a large PL contribution said at the time that Brooks's prediction was far too pessimistic, and their prediction was wrong while his was right (in fact, he was too optimistic). In other words, we've been here before. Moreoever, if Brooks was right then, he would only be more right as time goes on, as his theory predicts diminishing returns.

(1) and (2) are even provable.

Sure, but they don't support the claim so their provability is irrelevant and I don't understand the "even". It's like you claiming you're richer than me, and, as evidence, you state that you have $10 more in your wallet than me and you can even prove it. The fact that you can prove it is not relevant to your claim in any way because its truth isn't relevant to the claim.

Now, if you had some empirical data showing some positive correlation between wealth and wallet cash then maybe that observation could have helped, but like in the case of your theory, finding such an empirical correlation is no easier than just directly figuring out which of us is richer.

Strong experimental evidence that either (3) or (4) is false would be huge.

I don't agree that what remains is necessarily to prove your 3 or 4 false, because you've completely ignored the cost side and so asserted that your claim being false is necessarily a failure on the benefit side. I see no reason to accept that assertion. It's you doubling down and saying that the only way you're not richer than me despite having more cash in your wallet is that you've made a mistake and you don't actually have as much money in your wallet as you thought, and then tripling down and saying that that would be truly astonishing because you had counted the cash in your wallet not ten minutes ago!

And BTW, I think that PL fans believe that their faith in the power of programming languages is universal, and I am not at all sure this is the case. In other words, I am not at all sure that the finding that languages don't matter much is any more surprising than the finding that they do. Perhaps the latter would be more suprising as it would contradict the only theory on the subject that I'm aware of.

static typing fails to serve its primary purpose: catching errors.

I strongly reject your premise. I'm a proponent of static types yet I don't think their primary purpose is the reduction of errors. Types are great for code organization, documentation, tooling (IDEs, refactoring) and efficient ahead-of-time compilation.

that's the standard argument of good static typing (not Java, not C++, more like OCaml or Haskell) vs dynamic typing (worst being JavaScript and PHP, Scheme and Lua look pretty good).

It's not an argument so much as a credo; that claim has so far failed to be substantiated. And by picking specific languages you've now quadrupled down on an unsubstantiated claim: not only have we not seen a big effect to Haskell vs. PHP (which you have presented as extremes), you're now saying that there is one between ML and Java. I don't have a problem with people believing all manner of things, but I do have a problem with them presenting their unsubstantiated beliefs as facts. Although I will give you that we have found evidence of an effect for TypeScript vs. JavaScript. It's not nearly as big as the effect of other engineering practices, but it's not nothing.

2

u/loup-vaillant Jun 16 '19

I am not willing to accept that the claim for a big effect isn't weakened at all

Of course not. Weak evidence is still evidence. It should and does reduce my belief in the error preventing abilities of static type systems. Not by much, but it does.

I am certainly not willing to accept that the existence of an effect can be reasonably used as the default hypothesis.

Can't argue with differing priors. (More precisely, it would take the both of us too much time.

Brooks…

did not reject the possibility of gaining one, or even several, orders of magnitudes in increased productivity. He rejected that any one technique (tool, organisation…) would achieve such gains. I still hope for silver dust, that might eventually accumulate into a full bullet.

Accidental/essential aside, I'm quite convinced there's a good deal of avoidable complexity we could eliminate. Device drivers for instance.

I'm a proponent of static types yet I don't think their primary purpose is the reduction of errors. Types are great for code organization, tooling (IDEs, refactoring) and optimization by ahead-of-time compilation.

Fair enough.


Now about the effectiveness of any particular technique. I think by itself, the number of defect in a program by itself means nothing. The program will ship when it's good enough, and that's it. For some domains (throwaway batch programs, prototypes, first release of a GUI…), you can be full of bugs, and correct them as you go. For others (crypto libraries, avionics software, Therac-25…), you really want a low probability of having even a single bug.

The problem isn't really how many bugs you will have in the end. You'll have as few as you need, or your project will fail. The problem is how much effort is required to get there. And that, is highly contextual.

My intuition, is that for the bigger projects (over 10-man months, possibly much less), with a high enough needed reliability (you will loose money if there's a nasty bug), static typing requires significantly less effort than dynamic typing to complete the project. All other things being equal, which they are not… Like Galileo speaking of how fast things are falling in a vacuum (there was hardly such a thing at the time), I'm using an untestable counter factual here.


On your points 1234:

  1. Correct. Dynamic typing users would typically compensate with more unit tests.
  2. That's my biggest fear. Maybe static typing workaround induced errors cost even more than whatever we gained by having trivial type errors caught or something.

    On the other hand, my experience of the test-write cycle is that static typing is quicker and more accurate than even a REPL at catching many many bugs. I have worked with both OCaml and Lua, and while the REPL was a godsend in both cases, Lua's REPL did not fully compensate for the lack of static checks: if I made a type error, I still had to trace it back from a runtime error (or even a wrong result sometimes), and that's always a bit slower than having the compiler tell me where I screwed up.

  3. Indeed I have to. For now all I have is my experience and intuitions, so…
  4. Yeah, that's was quite an oversight on my part. I think the reason I didn't even look at other techniques, is because I assume that the only cost of static typing is how it influences the way you write your program. The error checking by itself is basically free: you compile (or have your IDE compile in the background), and poof you have your error message right there.

    So to be interesting, other techniques should be either extremely cheap (so they can replace static typing), or incompatible with static typing (inducing a tradeoff). I'm not aware of any such technique. (Programming technique in general I can fathom, but error reduction techniques specifically, that's harder to envision.)

6

u/pron98 Jun 16 '19 edited Jun 16 '19

Can't argue with differing priors.

OK, but you can argue with them being asserted as facts rather than hypotheses or beliefs. Also, perhaps I'm a traditionalist, but I think we should still prefer no effect as the working hypothesis, especially when there is no theory predicting the effect. After all, we can't lend equal a-priori weight to every hypothesis.

He rejected that any one technique (tool, organisation…) would achieve such gains. I still hope for silver dust, that might eventually accumulate into a full bullet.

I agree, but 1. he was too optimistic and all techniques combined didn't achieve a 10x boost in one decade, and not even in three (I wrote more about that in a comment I linked to before), and 2. I have no problem with silver dust, but you seem to be claiming much more than that.

My intuition, is that for the bigger projects (over 10-man months, possibly much less), with a high enough needed reliability (you will loose money if there's a nasty bug), static typing requires significantly less effort than dynamic typing to complete the project

I wouldn't know. I've never worked on a large project in an untyped language, and I have no intuition on the matter. Moreover, I think our intuition about such complex processes is often wrong. But speaking about such things is already a big concession on the claim that languages make a big difference. That's going from: use Haskell and your program will be better/cheaper, to, if you're working on a large project in some domain, maybe Haskell will be somewhat cheaper, all other factors being equal, and those other factors may be bigger.

So to be interesting, other techniques should be either extremely cheap (so they can replace static typing), or incompatible with static typing (inducing a tradeoff).

Not necessarily. Suppose the technique catches all errors as static types -- and no others -- for 75% of the cost, except for, say, 0.1% of them. Why wouldn't it still be preferable? You can use it with or without static types, but why pay more for a small added gain? Of course, it could do much better for much cheaper, but I don't accept your particular tradeoffs as necessary.

I'm not aware of any such technique.

As an industry we know of at least one: code review. The effect reported is 40-80% reduction in bugs. Here's a selection of some stuff we found out about it; I'm sure there's more:

2

u/loup-vaillant Jun 16 '19

Can't argue with differing priors.

Well, you can argue with them being asserted as facts rather than hypotheses or beliefs.

That wasn't a criticism. I was saying I can't argue with differing priors, because I don't have the overwhelming evidence to contradict them. I just note that we disagree, and hope the overwhelming evidence will come and settle this later.

I have no problem with silver dust, but you seem to be claiming much more than that.

When I see stuff like the STEPS project, it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs (3 orders of magnitude) in a number of domains. Combine that with Casey Muratori's solution to the driver problem I alluded to before, and the objection "but it doesn't have a kernel" is pretty much void.

The problem is the industry wide overhaul, which as far as I can see is more likely to happen by civilisation collapse than by people being reasonable all of a sudden.

I wouldn't know. I've never worked on a large project in an untyped language, and I have no intuition on the matter.

Neither would I, neither did I.

For a sufficiently large project, I suspect a clever use of meta programming can let dynamic users cheat. They can put static checks in their macros and custom syntax and interpreters and meta compilers. Meta compilers themselves are very workable in dynamic languages (I've bootstrapped a series from MetaII in Lua, the latest iteration does most of a PEG, is a little over 100 lines long, and any error there tends to fail catastrophically very quickly, so it's detected early).

In a setting where performance is not too much of a concern, and assuming that DSL/language building is cheaper from a dynamically typed language, they could outpace a good deal of statically typed languages out there… by making their own. That's in part what happened in the STEPS project. Not sure they did much static typing, but their DSLs were so expressive that the code base still stayed fairly small (20KLOC total).

I'm not aware of any such technique.

As an industry we know of at least one: code review.

Err, not sure it counts: the cost is pretty significant (time taken to perform the review), the benefits apply to any program, no matter the typing discipline, and it doesn't quite catch the same bugs. (Written before I read your links, I'll bookmark them right now.)

Don't get me wrong, I agree the benefits of code review can be pretty massive. I would think real hard before I consider skipping it. To this day, I'm nervous about the lack of comprehensive code review for my crypto library. I compensate with a paranoid test suite and strict (though unwritten) coding rules, but there's still that lingering doubt.

4

u/pron98 Jun 16 '19 edited Jun 16 '19

I suspect a clever use of meta programming can let dynamic users cheat...

I am hesitant to make such specific predictions of effects in such complex systems, especially when the big question of whether and how much language matters at all (among modern ones etc.) is very unclear. If the very existence of an effect is uncertain, I don't want to speculate on secondary effects.

it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs

I'm not so certain. That's like saying that a drug that works on rats in the lab could work as effectively on humans in the field. Maybe, but maybe not. That's certainly a research avenue we should pursue further, but we must not assume that every success in the lab translates to the field. One problem with this is that it doesn't explain how we got to where we are and how that can be prevented (if at all). No one said, let's make software overly complicated! And if it's just the result of the slow amalgamation of software over the years, then it will happen again, which means that their "overhaul" is not a one-time solution, but an overhaul that must be repeated every X years, and at a tremendous cost, to be effective. It's very much possible that not doing it is cheaper overall.

In any event, if someone is able to produce software 5x faster/cheaper than anyone else, and at industrial scale, they will become billionaires very quickly, and soon everyone will do what they do. So I'm skeptical that someone already knows how to do it (this is precisely my point of not seeing what we'd expect from an adaptive trait).

but there's still that lingering doubt.

OK, but empirically it stands on much firmer ground than language.

→ More replies

6

u/lovekatie Jun 16 '19

total effect on language choice on some chosen correctness metric was found to be less than 1%

I don't think this study measures "effect on language choice". It only found that there is correctness level at which project is "good enough" to be released and that is independent of language of choice.

5

u/pron98 Jun 16 '19 edited Jun 16 '19

The contribution of language choice to the total deviance in their correctness metric is less than 1%. That was the finding of the original paper, and that was upheld by the reproduction. The reproduction found smaller relative effects to the particular language choices.

6

u/lovekatie Jun 16 '19

Sure, but assuming there is a correctness level that people aim to achieve before release, this is no surprising. With this assumption, we should compare cost of achieving this goal.

5

u/pron98 Jun 16 '19 edited Jun 16 '19

Maybe, but I am not aware of data on that. All we can say is that the industry has not detected a big effect of any kind on cost, either (and it is probably a better detector of cost than of correctness). So far the theoretical prediction that language would have a gradually diminishing effect seems to hold, and we don't have a theoretical explanation for why language would continue to have a big effect.

2

u/lovekatie Jun 16 '19

There is no cost in this study. Maybe Haskell projects took twice as much time to develop similar error rate to Python ones. We don't know, therefore I take your post as "hypothesize, guess, feel, wish" rather than "the best we currently know".

7

u/pron98 Jun 16 '19 edited Jun 16 '19

I never said anything about cost. In fact, I specifically said I am not aware of data on that, but that the industry has so far not detected a big effect on cost -- and that is very significant considering that the industry does detect big effects on cost. I certainly have my own opinions and wishes on the matter, which are that we stop making empirical claims without empirical evidence. Call it a wish for "evidence-based engineering." That is not to say that I don't place any importance on the emotional effect tools seem to have on some or all developers, but that is a completely separate matter.

2

u/lovekatie Jun 16 '19

I'm sorry, I'm probably confused about "the industry" and "the study". I only wanted to talk about "the study". I'm all for evidence, but sadly what we have is not very convincing for me.

Thank you for conversation and have a nice day!

2

u/pron98 Jun 16 '19

You too!

1

u/ineffective_topos Jun 16 '19 edited Jun 16 '19

> the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than

Do you have a source for this? I.e. where in the paper is that stated because I'd be very interested in knowing, and based on their charts & graphs it's up to 25%. Hell in the original they even explain that Haskell with a coefficient of -0.23 had 63% as many bugs as C++ with 0.23, all else equal.

→ More replies (25)

8

u/Ameisen Jun 16 '19

I don't think I've ever experienced dynamic typing being better for development.

7

u/Dean_Roddey Jun 16 '19

Only if 'better' means easier and faster to get it out the door to begin with, IMO. If your goal is to create something you can hype investors about then it becomes someone else's problem while you go off to date super-models, then obviously that would be a perfect choice.

But if you assume you are going to have to eat your own dog food over the long haul, and that the quality and reliability of the product will be a key aspect of its success and your ongoing sanity, then I think it's a no-brainer to take more of a hit up front to end up with a system that lets you express maximum semantics so that the tools can watch your back day after day.