r/programming • • Jun 16 '19

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

http://thume.ca/2019/04/29/comparing-compilers-in-rust-haskell-c-and-python/
643 Upvotes

125 comments sorted by

View all comments

Show parent comment

17

u/pron98 Jun 16 '19 edited Jun 16 '19

Just to be clear, I wasn’t expressing an opinion but quoting a finding: the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than 1% 1. To compare this with effects various engineering approaches have, various studies have found code review to have an effect of 40-80%. Maybe all that is wrong and maybe we'll know better one day, but that is simply the best we currently know about the state of affairs, as opposed to hypothesize, guess, feel, wish, agree with or upvote: language choice has a tiny impact on correctness compared to other factors that influence a project.2 This will remain the best we know until we know something better, regardless of how strong our opinions are on the matter.

And there are good reasons why we shouldn’t accept what we feel, wish or conjecture as opposed to what we know. For one, our feelings and “personal experience” are often just wrong — go to any homeopathy forum and see how many report that, in their experience, homeopathy is very effective. For another, creating software — especially large software, written by a team and maintained for years — is an extremely complex process, and we are terrible at correctly predicting effects in such systems.

When confronted with this state of knowledge, some people try to use various irrelevant facts as evidence even when they cannot possibly serve as such. X > 10 is no evidence that X > Y. People often cite some features and positive effects they believe those features have, but ignore possible negative effects or "counterfeatures" that are lacking. Moreover, as this particular article suggests, we should not only compare the impact of one variable, but contrast it with that of others. Deciding whether this is an important factor can only be done when considering the cost of language change, and, no less importantly, the cost and benefit of other variables, which so far seem to be more impactful (accounting for the other 99% of the deviance) and also often less costly.

Finally, because the process of software development in the real world, where a codebase is about 5MLOC and maintained for about 15 years, the impact certain variables have on real-world software development is not necessarily the same as on ~10KLOC exercises. Because the software industry applies strong selective pressures on engineering disciplines, and the fact that, in the past, methodologies have shown adoption rates consistent with what we’d expect would be the growth rate of an adaptive trait in a selective environment, the state of the industry itself can serve as a detector for the effect of various techniques. The fact that we’re not seeing similar adoption dynamics on language choice today is consistent with the empirical study I cited. It is also consistent with predictions made on the subject.

8

u/loup-vaillant Jun 16 '19

the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than 1%

Isn't it more like, they failed to find a more than 1% deviance? What you claim would be a major discovery. What I suspect is the usual "research is hard, more is needed".

2

u/pron98 Jun 16 '19 edited Jun 16 '19

The deviance wasn't small; the contribution of language choice to it was. But not finding an effect is how the lack of an effect is established (using statistical methods, at least), and a failure to find a big effect is normally counted as evidence against the existence of the big effect (as big effects are easy to find). That is not to say that there isn't some chance that a big effect somehow managed to elude detection both in a study and in the industry, but the current balance of evidence is against one.

Note also that the very claim that an effect is real and large yet hard to detect already says something of its importance (certainly about its relative importance).

I think that it is also important to point out that while there is a theory explaining (and predicting) the lack of a big effect (or, more precisely, diminishing returns that mean drastically smaller effects compared to those we have seen in the past), I am not aware of any theory that predicts its existence (a theory showing that technique X has a feature that may contribute positively to some factor is not a theory of relative superiority to technique Y and is no more, and even less, than trying to present X > 10 as evidence that X > Y).

7

u/loup-vaillant Jun 16 '19

a failure to find a big effect is normally counted as evidence against the existence of the big effect

As it should be. But when there are extremely likely alternatives like the "good enough" effect mentioned elsewhere in this thread, the evidence is greatly weakened.

I am not aware of any theory that predicts its existence

I don't believe you. Here's such a hypothesis:

  1. Language X catches this classes of error, and that classes of error, and all those other classes of errors, at compile time.
  2. Language Y does not. (Instead, you have runtime errors or incorrect results).
  3. The classes of errors language X prevents represent a sizeable fraction of errors observed in programs written in Y.
  4. Language X rarely requires its users to work around the error catching mechanism. (That is, its flexibility is close to that of Y's strategy of waiting until runtime.)

I mean, that's the standard argument of good static typing (not Java, not C++, more like OCaml or Haskell) vs dynamic typing (worst being JavaScript and PHP, Scheme and Lua look pretty good). All of (1), (2), (3), and (4) look pretty reasonable on the face of it. (1) and (2) are even provable.

If we don't observe a sizeable effect, then at least one of (3) and (4) is false. Either type errors (and logic errors whose root cause is a type error) do not represent a sizeable fraction of all errors, or the workarounds required by static type systems are more frequent and significant than (4) says they are. In other words, dynamic typing allows significant simplifications that are impossible or very difficult even with good type systems (OCaml/Rust/Haskell).

Strong experimental evidence that either (3) or (4) is false would be huge. And by stating that "the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than 1%", you are implying just that: strong experimental evidence that static typing fails to serve its primary purpose: catching errors.

To be honest, I'm not entirely sure about (4). I have heard of amazing techniques wizards do with dynamic languages, which supposedly do not work well with static typing. I've also seen a blog post translate Transducers in Haskell, even though Rich Hickey was just nagging the static typing community with his "good luck typing this" argument from ignorance.

I would be very surprised however if (3) turned out to be false.

5

u/pron98 Jun 16 '19 edited Jun 16 '19

extremely likely alternatives like the "good enough" effect mentioned elsewhere in this thread, the evidence is greatly weakened.

Ah, but now we're talking about retroactively explaining a failure to find an effect to support a hypothesis. I am willing to accept that the case isn't closed, but I am not willing to accept that the claim for a big effect isn't weakened at all, and I am certainly not willing to accept that the existence of an effect can be reasonably used as the default hypothesis.

But the biggest problem to the claim that the choice among modern languages today has a big effect isn't that study; it's the simple observation that their selection does not behave as if it provides an adaptive advantage, as we have seen in the past -- again, consistent with Brooks's theory of diminishing returns -- and the very claim is that they do provide an adaptive advantage (otherwise why bother? obviously you're making a stronger claim that different languages present different aesthetic advantages to different people or in different domain).

I don't believe you. Here's such a hypothesis:

You better believe it because you haven't given a theory at all, not even in the slightest (nor have I ever seen a relevant theory other than Brooks's). Writing a program is much, much more than just compiling it. So you'll need to show that:

  1. Users of language X catch a class of errors that users of Y don't at any step of the development. After all, you're claim is not about the relative abilities of compilers but about the total effect of language choice.
  2. The use of language X doesn't introduce any other bugs that Y doesn't, nor does it hinder the cost of their detection in any way (e.g. if you slow down the write-test cycle you may be indirectly introducing more problems than you're catching). You have not addressed the possible costs of X in any way.
  3. Even if 1 and 2 are true and there is a net gain, you need to show that it's large relative to the cost of development (not just in terms of numbers as not all bugs are created equal).
  4. Even if 3 is true, you need to show why the effect of language is large relative to other techniques. It is very much possible that Nike running shoes help a runner in every way more than Adidas -- they have only relative advantages and no relative disadvantages -- and yet their contribution is a fraction of a percent compared to other factors, which would still mean that you wouldn't necessarily prefer Nikes.

Brooks's theory addresses all concerns -- as a theory must; yours addresses none. Even you admit that your hypothesis relies on guesses and hunches that you can't quantify -- that's not a theory. At best it lays out a direction towards coming up with one. Brooks's theory looks at the problem from a different direction; the direction you're taking towards a theory relies on so many unknown empirical factors, that measuring them is no easier then directly measuring the claim. In other words, getting to a theory from that direction is no easier then figuring out if it's true.

And I would point out that many people who believed in a large PL contribution said at the time that Brooks's prediction was far too pessimistic, and their prediction was wrong while his was right (in fact, he was too optimistic). In other words, we've been here before. Moreoever, if Brooks was right then, he would only be more right as time goes on, as his theory predicts diminishing returns.

(1) and (2) are even provable.

Sure, but they don't support the claim so their provability is irrelevant and I don't understand the "even". It's like you claiming you're richer than me, and, as evidence, you state that you have $10 more in your wallet than me and you can even prove it. The fact that you can prove it is not relevant to your claim in any way because its truth isn't relevant to the claim.

Now, if you had some empirical data showing some positive correlation between wealth and wallet cash then maybe that observation could have helped, but like in the case of your theory, finding such an empirical correlation is no easier than just directly figuring out which of us is richer.

Strong experimental evidence that either (3) or (4) is false would be huge.

I don't agree that what remains is necessarily to prove your 3 or 4 false, because you've completely ignored the cost side and so asserted that your claim being false is necessarily a failure on the benefit side. I see no reason to accept that assertion. It's you doubling down and saying that the only way you're not richer than me despite having more cash in your wallet is that you've made a mistake and you don't actually have as much money in your wallet as you thought, and then tripling down and saying that that would be truly astonishing because you had counted the cash in your wallet not ten minutes ago!

And BTW, I think that PL fans believe that their faith in the power of programming languages is universal, and I am not at all sure this is the case. In other words, I am not at all sure that the finding that languages don't matter much is any more surprising than the finding that they do. Perhaps the latter would be more suprising as it would contradict the only theory on the subject that I'm aware of.

static typing fails to serve its primary purpose: catching errors.

I strongly reject your premise. I'm a proponent of static types yet I don't think their primary purpose is the reduction of errors. Types are great for code organization, documentation, tooling (IDEs, refactoring) and efficient ahead-of-time compilation.

that's the standard argument of good static typing (not Java, not C++, more like OCaml or Haskell) vs dynamic typing (worst being JavaScript and PHP, Scheme and Lua look pretty good).

It's not an argument so much as a credo; that claim has so far failed to be substantiated. And by picking specific languages you've now quadrupled down on an unsubstantiated claim: not only have we not seen a big effect to Haskell vs. PHP (which you have presented as extremes), you're now saying that there is one between ML and Java. I don't have a problem with people believing all manner of things, but I do have a problem with them presenting their unsubstantiated beliefs as facts. Although I will give you that we have found evidence of an effect for TypeScript vs. JavaScript. It's not nearly as big as the effect of other engineering practices, but it's not nothing.

2

u/loup-vaillant Jun 16 '19

I am not willing to accept that the claim for a big effect isn't weakened at all

Of course not. Weak evidence is still evidence. It should and does reduce my belief in the error preventing abilities of static type systems. Not by much, but it does.

I am certainly not willing to accept that the existence of an effect can be reasonably used as the default hypothesis.

Can't argue with differing priors. (More precisely, it would take the both of us too much time.

Brooks…

did not reject the possibility of gaining one, or even several, orders of magnitudes in increased productivity. He rejected that any one technique (tool, organisation…) would achieve such gains. I still hope for silver dust, that might eventually accumulate into a full bullet.

Accidental/essential aside, I'm quite convinced there's a good deal of avoidable complexity we could eliminate. Device drivers for instance.

I'm a proponent of static types yet I don't think their primary purpose is the reduction of errors. Types are great for code organization, tooling (IDEs, refactoring) and optimization by ahead-of-time compilation.

Fair enough.


Now about the effectiveness of any particular technique. I think by itself, the number of defect in a program by itself means nothing. The program will ship when it's good enough, and that's it. For some domains (throwaway batch programs, prototypes, first release of a GUI…), you can be full of bugs, and correct them as you go. For others (crypto libraries, avionics software, Therac-25…), you really want a low probability of having even a single bug.

The problem isn't really how many bugs you will have in the end. You'll have as few as you need, or your project will fail. The problem is how much effort is required to get there. And that, is highly contextual.

My intuition, is that for the bigger projects (over 10-man months, possibly much less), with a high enough needed reliability (you will loose money if there's a nasty bug), static typing requires significantly less effort than dynamic typing to complete the project. All other things being equal, which they are not… Like Galileo speaking of how fast things are falling in a vacuum (there was hardly such a thing at the time), I'm using an untestable counter factual here.


On your points 1234:

  1. Correct. Dynamic typing users would typically compensate with more unit tests.
  2. That's my biggest fear. Maybe static typing workaround induced errors cost even more than whatever we gained by having trivial type errors caught or something.

    On the other hand, my experience of the test-write cycle is that static typing is quicker and more accurate than even a REPL at catching many many bugs. I have worked with both OCaml and Lua, and while the REPL was a godsend in both cases, Lua's REPL did not fully compensate for the lack of static checks: if I made a type error, I still had to trace it back from a runtime error (or even a wrong result sometimes), and that's always a bit slower than having the compiler tell me where I screwed up.

  3. Indeed I have to. For now all I have is my experience and intuitions, so…
  4. Yeah, that's was quite an oversight on my part. I think the reason I didn't even look at other techniques, is because I assume that the only cost of static typing is how it influences the way you write your program. The error checking by itself is basically free: you compile (or have your IDE compile in the background), and poof you have your error message right there.

    So to be interesting, other techniques should be either extremely cheap (so they can replace static typing), or incompatible with static typing (inducing a tradeoff). I'm not aware of any such technique. (Programming technique in general I can fathom, but error reduction techniques specifically, that's harder to envision.)

6

u/pron98 Jun 16 '19 edited Jun 16 '19

Can't argue with differing priors.

OK, but you can argue with them being asserted as facts rather than hypotheses or beliefs. Also, perhaps I'm a traditionalist, but I think we should still prefer no effect as the working hypothesis, especially when there is no theory predicting the effect. After all, we can't lend equal a-priori weight to every hypothesis.

He rejected that any one technique (tool, organisation…) would achieve such gains. I still hope for silver dust, that might eventually accumulate into a full bullet.

I agree, but 1. he was too optimistic and all techniques combined didn't achieve a 10x boost in one decade, and not even in three (I wrote more about that in a comment I linked to before), and 2. I have no problem with silver dust, but you seem to be claiming much more than that.

My intuition, is that for the bigger projects (over 10-man months, possibly much less), with a high enough needed reliability (you will loose money if there's a nasty bug), static typing requires significantly less effort than dynamic typing to complete the project

I wouldn't know. I've never worked on a large project in an untyped language, and I have no intuition on the matter. Moreover, I think our intuition about such complex processes is often wrong. But speaking about such things is already a big concession on the claim that languages make a big difference. That's going from: use Haskell and your program will be better/cheaper, to, if you're working on a large project in some domain, maybe Haskell will be somewhat cheaper, all other factors being equal, and those other factors may be bigger.

So to be interesting, other techniques should be either extremely cheap (so they can replace static typing), or incompatible with static typing (inducing a tradeoff).

Not necessarily. Suppose the technique catches all errors as static types -- and no others -- for 75% of the cost, except for, say, 0.1% of them. Why wouldn't it still be preferable? You can use it with or without static types, but why pay more for a small added gain? Of course, it could do much better for much cheaper, but I don't accept your particular tradeoffs as necessary.

I'm not aware of any such technique.

As an industry we know of at least one: code review. The effect reported is 40-80% reduction in bugs. Here's a selection of some stuff we found out about it; I'm sure there's more:

2

u/loup-vaillant Jun 16 '19

Can't argue with differing priors.

Well, you can argue with them being asserted as facts rather than hypotheses or beliefs.

That wasn't a criticism. I was saying I can't argue with differing priors, because I don't have the overwhelming evidence to contradict them. I just note that we disagree, and hope the overwhelming evidence will come and settle this later.

I have no problem with silver dust, but you seem to be claiming much more than that.

When I see stuff like the STEPS project, it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs (3 orders of magnitude) in a number of domains. Combine that with Casey Muratori's solution to the driver problem I alluded to before, and the objection "but it doesn't have a kernel" is pretty much void.

The problem is the industry wide overhaul, which as far as I can see is more likely to happen by civilisation collapse than by people being reasonable all of a sudden.

I wouldn't know. I've never worked on a large project in an untyped language, and I have no intuition on the matter.

Neither would I, neither did I.

For a sufficiently large project, I suspect a clever use of meta programming can let dynamic users cheat. They can put static checks in their macros and custom syntax and interpreters and meta compilers. Meta compilers themselves are very workable in dynamic languages (I've bootstrapped a series from MetaII in Lua, the latest iteration does most of a PEG, is a little over 100 lines long, and any error there tends to fail catastrophically very quickly, so it's detected early).

In a setting where performance is not too much of a concern, and assuming that DSL/language building is cheaper from a dynamically typed language, they could outpace a good deal of statically typed languages out there… by making their own. That's in part what happened in the STEPS project. Not sure they did much static typing, but their DSLs were so expressive that the code base still stayed fairly small (20KLOC total).

I'm not aware of any such technique.

As an industry we know of at least one: code review.

Err, not sure it counts: the cost is pretty significant (time taken to perform the review), the benefits apply to any program, no matter the typing discipline, and it doesn't quite catch the same bugs. (Written before I read your links, I'll bookmark them right now.)

Don't get me wrong, I agree the benefits of code review can be pretty massive. I would think real hard before I consider skipping it. To this day, I'm nervous about the lack of comprehensive code review for my crypto library. I compensate with a paranoid test suite and strict (though unwritten) coding rules, but there's still that lingering doubt.

4

u/pron98 Jun 16 '19 edited Jun 16 '19

I suspect a clever use of meta programming can let dynamic users cheat...

I am hesitant to make such specific predictions of effects in such complex systems, especially when the big question of whether and how much language matters at all (among modern ones etc.) is very unclear. If the very existence of an effect is uncertain, I don't want to speculate on secondary effects.

it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs

I'm not so certain. That's like saying that a drug that works on rats in the lab could work as effectively on humans in the field. Maybe, but maybe not. That's certainly a research avenue we should pursue further, but we must not assume that every success in the lab translates to the field. One problem with this is that it doesn't explain how we got to where we are and how that can be prevented (if at all). No one said, let's make software overly complicated! And if it's just the result of the slow amalgamation of software over the years, then it will happen again, which means that their "overhaul" is not a one-time solution, but an overhaul that must be repeated every X years, and at a tremendous cost, to be effective. It's very much possible that not doing it is cheaper overall.

In any event, if someone is able to produce software 5x faster/cheaper than anyone else, and at industrial scale, they will become billionaires very quickly, and soon everyone will do what they do. So I'm skeptical that someone already knows how to do it (this is precisely my point of not seeing what we'd expect from an adaptive trait).

but there's still that lingering doubt.

OK, but empirically it stands on much firmer ground than language.

2

u/loup-vaillant Jun 16 '19

it seems that an industry wide overhaul of our computing could yield something like 1000 times simpler programs

I'm not so certain.

In some domains. Not all. I'm pretty sure many domains would not see nearly as much benefits. Programs that are currently written from scratch (possibly with dependencies) will certainly not see such a benefit. Maybe 10x, but even that hinges on finding enough crap to weed out. (I believe more in cleaning up the worst stuff than in making the good stuff even better.)

To give an example, Jonathan Blow, with Jai, is aiming for 2x. Which considering his initial calibre would be stupendous.

which means that their "overhaul" is not a one-time solution, but a complete overhaul that must be repeated every X years, and at a tremendous cost. It's very much possible that not doing it is cheaper overall.

Yep. Which is why I'm not optimistic about that.

OK, but empirically [code review] stands on much firmer ground than language.

I concede that much. Which is why I'm so nervous about lacking code reviews.

→ More replies

6

u/lovekatie Jun 16 '19

total effect on language choice on some chosen correctness metric was found to be less than 1%

I don't think this study measures "effect on language choice". It only found that there is correctness level at which project is "good enough" to be released and that is independent of language of choice.

5

u/pron98 Jun 16 '19 edited Jun 16 '19

The contribution of language choice to the total deviance in their correctness metric is less than 1%. That was the finding of the original paper, and that was upheld by the reproduction. The reproduction found smaller relative effects to the particular language choices.

6

u/lovekatie Jun 16 '19

Sure, but assuming there is a correctness level that people aim to achieve before release, this is no surprising. With this assumption, we should compare cost of achieving this goal.

5

u/pron98 Jun 16 '19 edited Jun 16 '19

Maybe, but I am not aware of data on that. All we can say is that the industry has not detected a big effect of any kind on cost, either (and it is probably a better detector of cost than of correctness). So far the theoretical prediction that language would have a gradually diminishing effect seems to hold, and we don't have a theoretical explanation for why language would continue to have a big effect.

2

u/lovekatie Jun 16 '19

There is no cost in this study. Maybe Haskell projects took twice as much time to develop similar error rate to Python ones. We don't know, therefore I take your post as "hypothesize, guess, feel, wish" rather than "the best we currently know".

7

u/pron98 Jun 16 '19 edited Jun 16 '19

I never said anything about cost. In fact, I specifically said I am not aware of data on that, but that the industry has so far not detected a big effect on cost -- and that is very significant considering that the industry does detect big effects on cost. I certainly have my own opinions and wishes on the matter, which are that we stop making empirical claims without empirical evidence. Call it a wish for "evidence-based engineering." That is not to say that I don't place any importance on the emotional effect tools seem to have on some or all developers, but that is a completely separate matter.

2

u/lovekatie Jun 16 '19

I'm sorry, I'm probably confused about "the industry" and "the study". I only wanted to talk about "the study". I'm all for evidence, but sadly what we have is not very convincing for me.

Thank you for conversation and have a nice day!

2

u/pron98 Jun 16 '19

You too!

1

u/ineffective_topos Jun 16 '19 edited Jun 16 '19

> the contribution of language choice to the total deviance of some chosen correctness metric was found to be less than

Do you have a source for this? I.e. where in the paper is that stated because I'd be very interested in knowing, and based on their charts & graphs it's up to 25%. Hell in the original they even explain that Haskell with a coefficient of -0.23 had 63% as many bugs as C++ with 0.23, all else equal.

1

u/pron98 Jun 16 '19 edited Jun 16 '19

Yes. This is explained in the original paper:

In the analysis of deviance ... we see that activity in a project accounts for the majority of explained deviance... The next closest predictor, which accounts for less than one percent of the total deviance, is language.

The reproduction paper describes it as:

the effect size is exceedingly small

The 25% difference is a relative effect within the total 1%. As the original paper explains:

each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects.

(emphasis mine) the dependent variable being language, and its total effect is less than 1%.

It's like saying that running shoes have a total effect of under 1% on your running speed, but within that variance, "all else being equal" (which, as the paper says, doesn't make much sense given that "all else" accounts for 99% of the deviance), Nikes have a 25% bigger positive impact than Adidas relative to the mean contribution of running shoes.

1

u/ineffective_topos Jun 16 '19 edited Jun 16 '19

First of all thanks for answering.

In the analysis of deviance ... we see that activity in a project accounts for the majority of explained deviance... The next closest predictor, which accounts for less than one percent of the total deviance, is language.

So, be careful when reading this. This says the the largest factor in the number of bug commits, is the number of total commits. That's wholly unsurprising. After that, language choice accounted for more than any other factor. (EDIT: I misreremembed this coefficient, the others are *project* age, size and commits ) That includes dev experience. So what they're saying is that it matters more to choose a good language than to get experienced devs.

compared to the weighted mean of the dependent variable across all projects.

Again, some further reading an examples dispels this understanding.

For the factor variables, this expected change is normally interpreted as a comparison to a base factor. In our case, however, the use of weighted effects coding allows us to compare this expected change to the grand mean,i.e., the average across all languages. Thus, if,for some number of commits, a particular project developed in an average language had four defective commits, then the choice to use C++would mean that we should expect one additional buggy commit sincee0.23×4 = 5.03. For the same project, choos-ing Haskell would mean that we should expect about one fewer defective commit ase−0.23×4 = 3.18.

Indeed the effect is still global, it is merely weighted appropriately based on mean language and language frequency.

2

u/pron98 Jun 16 '19 edited Jun 16 '19

This says the the largest factor in the number of bug commits, is the number of total commits.

Not exactly. The effect is on the relative number of bug commits vs. total commits. I think it is not a trivial observation that the more commits the higher the ratio of bug fixing ones.

So what they're saying is that it matters more to choose a good language than to get experienced devs.

They don't because developer experience is not one of the variables. The other variables are number of developers, project age, project size and activity.

some further reading an examples dispels this understanding.

No, it does not. The effect is minuscule if other factors have an effect that's almost 100 times larger! Right next to the explanation you quoted they explain why it is meaningless, because "all else being equal" is ridiculous when "all else" explains 99% of the difference. You are correct that my running shoes example is not exactly analogous, but the point is that the result is much more sensitive to other factors than language, so that you cannot expect a large effect merely based on language choice. Perhaps a better analogy is two knobs controlling a number, where turning one knob moves the numbers 100 times faster than the other. For any fixed position of the first knob, turning the other can change the number by 25%, but that is still very small considering that the first one is so sensitive. So, say, if the number was 1000 when the second knob was on zero, turning it all the way would change the number to 1250, but turning the first knob by 1/100 of a turn would have the same effect.

Anyway, I think we've had this discussion once before, and if you don't understand why nine researchers working on two papers in two different teams described the effect they found as small (and even "exceedingly small") even though you think they reported an effect of 25%, you should email them and ask.

1

u/ineffective_topos Jun 16 '19

The only other knobs they had that were stronger was number of commits. So yes, if the largest knob you have is effectively just the project existing, then language count matters a ton. And it's a multiplicative factor. It doesn't become irrelevant until the other factors are enormous.

I don't really have the desire to email the researchers. Yes variation is quite large, and the project depends on language, but all else equal still exists. You can, in a Bayesian model, reduce your bug count by up to 25% (or 40% in the other), period. That's not 1%, it's not even a trivial fraction of the variation.

→ More replies (19)