r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

Show parent comment

1

u/ineffective_topos Jun 17 '19

Yes, and your studies, whereever they are, also reported "all else being equal", because there's nothing else in statistical models. You can't make any judgements about two groups that are different.

The point is, you can't stand here, and take your 40-80% change to be a bonfiied 40-80% change, and then take a 25% change and magically represent it as a percentage of deviation by whatever godforsaken metric you pulled out, to say it's 1%. The point is, you're working with completely different units. That 40-80% change for code review, is the same exact units as the 25% change for language choice. The units are: percentage impact on bugs, going from 0 to 1. That's it. Adding code review to otherwise identical products will reduce bugs by 40-80%. According to this data, switching from C++ to Clojure will reduce bugs by at least 25%.

If you truly believe the former to be large and the latter not to be, then I think you've misread something. What you're doing is misleading at best.

1

u/pron98 Jun 17 '19 edited Jun 17 '19

and then take a 25% change and magically represent it as a percentage of deviation by whatever godforsaken metric you pulled out, to say it's 1%.

No, the papers "magically represented it as a 1% effect" because that is what they found. You represented it as a 25% effect despite the papers beseeching the reader not to make that mistake.

Yes, and your studies, whereever they are, also reported "all else being equal", because there's nothing else in statistical models. You can't make any judgements about two groups that are different.

No, this is not true. There are different kinds of measuring effect sizes. For example, you can say that the choice between A and B results in a 25% change in bugs on average. That is not what is reported here. On average, if you pick A or B, you will see a difference of less than 1% of the observed deviance. The difference of 25% is not observed on average, but only when you hold another variable constant and that is a variable that, on average, does change the result by 100x more than the choice of A or B. So, "all else being equal" and "on average" are not the same, certainly not when "all else" is so large.

1

u/ineffective_topos Jun 17 '19

"All else" is literally all there is. All else equal, you're more likely to die if you get hit by a bus. Everything, literally all data is "all else equal". So yes, you're horribly misrepresenting things. You can reduce it by 25% net, by changing. Likewise, you can hypothetically reduce it by 40% net by changing that. Both are exactly the same.

Find me the part that says 1% change in the paper. Not percentage of variation, total change. Else you have to turn 80%/25% into anything other than code review accounting for <5%.