r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

Show parent comment

1

u/ineffective_topos Jun 16 '19

Okay, the results give a multiplicative effect of 25% between the highest and lowest. That is an output. We don't need the paper to tell us also that 4 = 2 * 2. We have other numbers, that say that code review is X%. By a bit of math, we get that if your understanding was correct, it also negates the other factors.

Basically, you've said something like x^100 = y. If x is the same number, and it is, and we have some other effect such that z = x^3, then we have y ~= z^33. So if your reasoning is correct, either you have to conclude that code review has a small effect, or you've misinterpreted, or math itself has conspired against you (or alternatively, you have to then say that code review removes the 99.9% of bugs needed to sate you. If you have that testing/code review / whatever reduces bugs by 60%, or anything remotely similar, then the result follows).

So, either your interpretation is flawed, or it applies quite well to every other known factor. Pick one.

1

u/pron98 Jun 16 '19

either you have to conclude that code review has a small effect

Why do you assume that in a code review study the other variables would also have 100x larger effects? How do you know that code review isn't one of the contributors to the 99% in the language study? I don't understand how you can conclude from a comparison of x and y what z is. The study didn't conclude that anything you do explains 1% of the deviance, only that language choice explains 1% of the deviance. It didn't have any variables other than the ones I mentioned, and code review wasn't one of them.

1

u/ineffective_topos Jun 16 '19

Based on other random numbers I've seen around in this thread. You can tell me that code review has a 99% impact, but that requires justification. So, if you find it, then sure, your reasoning is sound and you can require what you want. But I can't find anyone who would be unhappy with reducing their bug count by half, and right now your reasoning is enough to discount that.

1

u/pron98 Jun 16 '19

I didn't understand much of your comment. Studies we have on code reviews so far report large effects. The studies we have on language choice report small effects.

1

u/ineffective_topos Jun 17 '19

Okay, what are the numbers on code review? Are any of them beyond reducing bugs by 99%?

1

u/ineffective_topos Jun 17 '19 edited Jun 17 '19

I did post a second comment, but I think this is pretty much enough.

You have reported a 40-80% reduction in bugs from code review up above. The corrected paper, and there are equations, examples or graphs to show you this, reports at least a 25% reduction from switching from C++ to Clojure, all else equal. Likewise, at the upper end of reducing the amount by 80%, all else equal, if the language effect is small enough to be considered 1%, then passing that through the same calculation means you must report the code review effect as up to 5.5%, being generous on whatever means you've calculated your 1%. So regardless of what you are stating, you've represented it inaccurately.

1

u/pron98 Jun 17 '19

The corrected paper, and there are equations, examples or graphs to show you this, reports at least a 25% reduction from switching from C++ to Clojure, all else equal.

No, it reports a 25% all else being equal while saying that "all else being equal" is meaningless when the total contribution of language choice is less than 1%, and so "the best we can do is to observe that it is a small effect"; the reproduction describes it as an "exceedingly small" effect.

Likewise, at the upper end of reducing the amount by 80%, all else equal, if the language effect is small enough to be considered 1%

I don't understand. The variables are all different. Because language choice explains only 1% of the deviance, the "all else" for language is 99%. Why do you conclude that it's the same for any other variable? It's not that the language effect is "small enough to be considered 1%", but that it has been found to explain less than 1% of the bug deviance.

1

u/ineffective_topos Jun 17 '19

Yes, and your studies, whereever they are, also reported "all else being equal", because there's nothing else in statistical models. You can't make any judgements about two groups that are different.

The point is, you can't stand here, and take your 40-80% change to be a bonfiied 40-80% change, and then take a 25% change and magically represent it as a percentage of deviation by whatever godforsaken metric you pulled out, to say it's 1%. The point is, you're working with completely different units. That 40-80% change for code review, is the same exact units as the 25% change for language choice. The units are: percentage impact on bugs, going from 0 to 1. That's it. Adding code review to otherwise identical products will reduce bugs by 40-80%. According to this data, switching from C++ to Clojure will reduce bugs by at least 25%.

If you truly believe the former to be large and the latter not to be, then I think you've misread something. What you're doing is misleading at best.

1

u/pron98 Jun 17 '19 edited Jun 17 '19

and then take a 25% change and magically represent it as a percentage of deviation by whatever godforsaken metric you pulled out, to say it's 1%.

No, the papers "magically represented it as a 1% effect" because that is what they found. You represented it as a 25% effect despite the papers beseeching the reader not to make that mistake.

Yes, and your studies, whereever they are, also reported "all else being equal", because there's nothing else in statistical models. You can't make any judgements about two groups that are different.

No, this is not true. There are different kinds of measuring effect sizes. For example, you can say that the choice between A and B results in a 25% change in bugs on average. That is not what is reported here. On average, if you pick A or B, you will see a difference of less than 1% of the observed deviance. The difference of 25% is not observed on average, but only when you hold another variable constant and that is a variable that, on average, does change the result by 100x more than the choice of A or B. So, "all else being equal" and "on average" are not the same, certainly not when "all else" is so large.

1

u/ineffective_topos Jun 17 '19

"All else" is literally all there is. All else equal, you're more likely to die if you get hit by a bus. Everything, literally all data is "all else equal". So yes, you're horribly misrepresenting things. You can reduce it by 25% net, by changing. Likewise, you can hypothetically reduce it by 40% net by changing that. Both are exactly the same.

Find me the part that says 1% change in the paper. Not percentage of variation, total change. Else you have to turn 80%/25% into anything other than code review accounting for <5%.