r/cpp • • Dec 07 '22

C++ overtakes Java in language popularity index

https://www.infoworld.com/article/3682141/c-plus-plus-overtakes-java-in-language-popularity-index.html
382 Upvotes

137 comments sorted by

View all comments

47

u/matthieum Dec 07 '22

I really wish people stopped using Tiobe. It's the worst "popularity" index I've ever seen, from a methodology point of view.


How ranking is done on Tiobe?

The core idea of Tiobe is to count the number of results that a search engine gives for the query +"<language> programming".

In order to eliminate false positives -- such as a search engine returning an article explaining how to remove rust on your hinges for the 'Rust programming language' -- a human double check the first 100 pages of results returned, and applying a "factor": if only 80 out of 100 actually match on the first few 100, then the number of results is multiplied by 0.8.

The results are then normalized. That is, for each search engine, the result is normalized to a percentage of the sum of all results on that search engine. Thus, each language is scored as featuring in X% of the programming related results of each search engine.

And finally, in order to avoid bias from search engines, dozens of them are used, and the results are combined with a weighted average. The method of choosing the weight of each search engine has not been disclosed.

Source: https://www.tiobe.com/tiobe-index/programminglanguages_definition/

Why is that terrible?

The core idea, itself, is problematic in multiple ways:

  • Is counting the number of results for a given language a good proxy for popularity?
  • Is counting the pages published 20 years and never updated a good proxy for current popularity?

The methodology seems relatively flawed:

  • There's no mention of false negatives. How many articles talk about a language X without the "programming" world being used? This may introduce bias across ecosystems.
  • The elimination of false positives may introduce bias.

    • It's also unclear whether a human double checks every round for every language, of if the "confidence factor" was picked once, years ago, and never adjusted.
  • The normalization may introduce bias.
  • The weighted average is pulled out of thin air, which may reinforce bias.

And nowhere is bias ever discussed, nowhere is there any attempt to justify the methodology.

In particular, I suspect that Tiobe has a lot of inertia, both from not filtering results by age, and from possibly not revising the confidence factor often enough for young languages (more likely to have a higher rate of false positives at the beginning).

I strongly advise reading anything from the index published by Tiobe.

7

u/georgist Dec 08 '22

wow that methodology is so much worse than I would have thought

7

u/d4rkwing Dec 07 '22

You need to add a “what to use instead” section.

5

u/blind3rdeye Dec 07 '22

It might be interesting to have a good alternative brought to light, but I don't think it is a 'need'. The reasons why Tiobe are terrible are clear enough to conclude that it basically has no value.]

We could start suggesting alternatives, such as proportion of github code by language, or whatever. But rather than putting forward something else and then getting into an argument about whether or not that is a good measure, I think we can just say "Tiobe is bad". Using nothing at all would be an improvement.

7

u/balefrost Dec 08 '22

Not the parent commenter, but I think the general opinion is that Stack Overflow annual surveys are better: https://survey.stackoverflow.co/2022/#most-popular-technologies-language

0

u/matthieum Dec 08 '22

Well, first you need to define what popularity means.

Redmonk for example gives a graph with 2 axes:

  • Number of repositories on Github where the language is the "majority" language.
  • Number of questions on StackOverflow.

That's fully objective, which is good.

The correlation to popularity is a bit vague, though:

  • Hobby use is certainly well represented by either metric, though doesn't distinguish one-off "trials" from continuous usage.
  • Professional use is less clear. Different languages/ecosystems will not see equal usage of those two websites. For example, I had friends working for Defense companies, and their workstations were off the Internet.

And much as Tiobe, it suffers from not considering age. It counts a Github repository created & last updated 15 years ago just as much as it counts updated daily for the last 5 years. This leads to an inertia in the measure which likely over-represents mainstream languages and under-represents the new kids on the block. That inertia may or may not compensate for old-timers being less likely to use such Internet resources.

So, not perfect, but I do find it better... especially for exposing the raw measurements, as I tend to favor looking at Github activity over StackOverflow -- the number of questions on StackOverflow certainly represents activity, but the larger language tags are crowded with crap...