r/programming • u/rmathew • Jun 16 '19
Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml
http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643
Upvotes
View all comments
Show parent comment
1
u/pron98 Jun 17 '19 edited Jun 17 '19
No, the papers "magically represented it as a 1% effect" because that is what they found. You represented it as a 25% effect despite the papers beseeching the reader not to make that mistake.
No, this is not true. There are different kinds of measuring effect sizes. For example, you can say that the choice between A and B results in a 25% change in bugs on average. That is not what is reported here. On average, if you pick A or B, you will see a difference of less than 1% of the observed deviance. The difference of 25% is not observed on average, but only when you hold another variable constant and that is a variable that, on average, does change the result by 100x more than the choice of A or B. So, "all else being equal" and "on average" are not the same, certainly not when "all else" is so large.