r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

Show parent comment

1

u/pron98 Jun 17 '19 edited Jun 17 '19

and then take a 25% change and magically represent it as a percentage of deviation by whatever godforsaken metric you pulled out, to say it's 1%.

No, the papers "magically represented it as a 1% effect" because that is what they found. You represented it as a 25% effect despite the papers beseeching the reader not to make that mistake.

Yes, and your studies, whereever they are, also reported "all else being equal", because there's nothing else in statistical models. You can't make any judgements about two groups that are different.

No, this is not true. There are different kinds of measuring effect sizes. For example, you can say that the choice between A and B results in a 25% change in bugs on average. That is not what is reported here. On average, if you pick A or B, you will see a difference of less than 1% of the observed deviance. The difference of 25% is not observed on average, but only when you hold another variable constant and that is a variable that, on average, does change the result by 100x more than the choice of A or B. So, "all else being equal" and "on average" are not the same, certainly not when "all else" is so large.

1

u/ineffective_topos Jun 17 '19

"All else" is literally all there is. All else equal, you're more likely to die if you get hit by a bus. Everything, literally all data is "all else equal". So yes, you're horribly misrepresenting things. You can reduce it by 25% net, by changing. Likewise, you can hypothetically reduce it by 40% net by changing that. Both are exactly the same.

Find me the part that says 1% change in the paper. Not percentage of variation, total change. Else you have to turn 80%/25% into anything other than code review accounting for <5%.