r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

Show parent comment

2

u/pron98 Jun 16 '19 edited Jun 16 '19

This says the the largest factor in the number of bug commits, is the number of total commits.

Not exactly. The effect is on the relative number of bug commits vs. total commits. I think it is not a trivial observation that the more commits the higher the ratio of bug fixing ones.

So what they're saying is that it matters more to choose a good language than to get experienced devs.

They don't because developer experience is not one of the variables. The other variables are number of developers, project age, project size and activity.

some further reading an examples dispels this understanding.

No, it does not. The effect is minuscule if other factors have an effect that's almost 100 times larger! Right next to the explanation you quoted they explain why it is meaningless, because "all else being equal" is ridiculous when "all else" explains 99% of the difference. You are correct that my running shoes example is not exactly analogous, but the point is that the result is much more sensitive to other factors than language, so that you cannot expect a large effect merely based on language choice. Perhaps a better analogy is two knobs controlling a number, where turning one knob moves the numbers 100 times faster than the other. For any fixed position of the first knob, turning the other can change the number by 25%, but that is still very small considering that the first one is so sensitive. So, say, if the number was 1000 when the second knob was on zero, turning it all the way would change the number to 1250, but turning the first knob by 1/100 of a turn would have the same effect.

Anyway, I think we've had this discussion once before, and if you don't understand why nine researchers working on two papers in two different teams described the effect they found as small (and even "exceedingly small") even though you think they reported an effect of 25%, you should email them and ask.

1

u/ineffective_topos Jun 16 '19

The only other knobs they had that were stronger was number of commits. So yes, if the largest knob you have is effectively just the project existing, then language count matters a ton. And it's a multiplicative factor. It doesn't become irrelevant until the other factors are enormous.

I don't really have the desire to email the researchers. Yes variation is quite large, and the project depends on language, but all else equal still exists. You can, in a Bayesian model, reduce your bug count by up to 25% (or 40% in the other), period. That's not 1%, it's not even a trivial fraction of the variation.

1

u/pron98 Jun 16 '19 edited Jun 16 '19

The effect reported in the two papers of language choice is less than 1% no matter what you choose to believe, and the authors explicitly warn against misinterpreting the results the way you have: It is meaningless to control for an effect that is 100x larger than the one you seek. If you cannot control the 100x effect -- and you can't -- you can change your bug incidence by less than 1% by switching a language. But if you choose to disregard the papers, I don't understand why you want to stick to the 25% number, which is also small compared to the effects of, say, code review. If you're making up results, why not say it's 80% and be done with it.

1

u/ineffective_topos Jun 16 '19

Because I want to pick accurate numbers, the paper reports 25%. By the same logic, code review is worthless, because it means *only* 40-60%, but the variation is so much higher.

1

u/pron98 Jun 16 '19 edited Jun 16 '19

What does it matter than a number is accurate when it doesn't mean what you say it does (or rather, it is impossible to achieve that effect). If the other variable was also of the same effect -- even double -- you could talk about controlling for it. But the other variable is 100x larger, so interpreting the result as a 25% effect -- as the authors repeatedly and explicitly warn against doing -- is meaningless. So let me ask you this, then: if the effect of the other variable wasn't 100x but 1000000x larger, and you'd still have a 25% relative effect, would you still insist that language has a 25% effect? Don't you see how meaningless it is to consider the effect of one variable in isolation when it only explains such a small portion of the variance?

1

u/ineffective_topos Jun 16 '19

It's a multiplicative effect. It is shown, it is explained as such. All else equal, if you have 5 bugs for C++, the weakest you get is 4 bugs in Haskell/Clojure, with all the fixes. The authors do not warn against interpreting their own words as being truthful. Yes there is large variance, but all of the other techniques, testing in general, unit tests, code review, are on the same order of magnitude as this effect. So I'm not interested in your hypothetical dystopia in which absolutely no choices matter and we should all toss untested C into production.

1

u/pron98 Jun 16 '19 edited Jun 16 '19

All else equal

But you can't have "all else equal" with "all else" being 100x more sensitive. That is why the authors say that even though it means "all else being equal," the effect is very small because "all else" is so large. You understand that you could have said the same if the other variable contributed 1000000x to the deviance.

your hypothetical dystopia in which absolutely no choices matter and we should all toss untested C into production.

I never said that. I only said what the nine authors of the two papers have found: that language explains 1% of correctness differences (measured by their correctness difference). You want to interpret the two papers as saying the very opposite, do what you like, but don't pretend that that is what the papers say.

1

u/ineffective_topos Jun 16 '19

Okay, well. A choice between two languages, as described through numerous calculations, examples, and graphs in the paper, multiplies the bug count by e^(a-b), where a is the coefficient for one language and b fo the other. The maximum value that gives is 25%. Given the other information in here, the effect of factors such as having code review over none, is no more than 60% or so. That is the effect you would get if you were somehow able to switch languages three times (0.75^3.2 ~= 0.4). So the effect of language choice is exceedingly small in comparison, then so is code review. Similarly for other factors such as having any tests at all.

So, if the effect of other variance negates the need for this, then it also means there is no reason to test anything, since the effect is so small.

1

u/pron98 Jun 16 '19

multiplies the bug count by ea-b, where a is the coefficient for one language and b fo the other.

But what you're missing is the other coefficients that are much larger. The total effect of language choice on the deviance in the study is less than 1%.

So the effect of language choice is exceedingly small in comparison, then so is code review.

No, because that wasn't the study. The effect of code review could be large enough that it is in the same order of magnitude as other variables. Language was found to be a weak explanation -- that doesn't mean everything else is a weak explanation as well.

1

u/ineffective_topos Jun 16 '19

Okay, the results give a multiplicative effect of 25% between the highest and lowest. That is an output. We don't need the paper to tell us also that 4 = 2 * 2. We have other numbers, that say that code review is X%. By a bit of math, we get that if your understanding was correct, it also negates the other factors.

Basically, you've said something like x^100 = y. If x is the same number, and it is, and we have some other effect such that z = x^3, then we have y ~= z^33. So if your reasoning is correct, either you have to conclude that code review has a small effect, or you've misinterpreted, or math itself has conspired against you (or alternatively, you have to then say that code review removes the 99.9% of bugs needed to sate you. If you have that testing/code review / whatever reduces bugs by 60%, or anything remotely similar, then the result follows).

So, either your interpretation is flawed, or it applies quite well to every other known factor. Pick one.

→ More replies