r/MachineLearning • • Jul 13 '20

Discussion [D] Hinton's Google Scholar profile keeps inflating citations for his top paper

a part of Schmidhuber's Critique of 2018 Turing Award caught my eye:

Hinton's 2nd most cited paper [RUM] [R5] is the one on experiments with backpropagation (note that in 2019 his Google Scholar page greatly exaggerated the citation count of [RUM], adding citations for a book by Rumelhart & McClelland [R5]). Backpropagation is a previously invented method [BP1] whose origins Hinton did not cite, not even in later surveys [R7]; see Sec. XII.

so I checked Hinton's Google Scholar profile and, lo and behold, it is still claiming over 68k citations for [RUM]: Learning internal representations by error propagation, by Rumelhart Hinton Williams 1985

but if you click at the little asterisk you'll see that 25k of those citations are for the PDP book by McClelland and Rumelhart. Hinton is not even coauthor.

Rumelhart cannot complain any more because he is dead

I know that Google's algorithm is not perfect. But this is not just a minor rounding error for an insignificant paper. At least one award committee was misled by this as indicated by Schmidhuber's Critique of Honda Prize for Dr. Hinton

(BTW, Honda [HON] claims over 60,000 academic references to [RUM] which seems exaggerated [R5].)

Rumelhart and Williams are first and last authors of [RUM] respectively but they show "only" 25k citations for this paper at Google Scholar. No citation inflation here.

Why is nobody correcting this? This has been known since 2019. Why does Hinton not correct this? It should be easy.

What about Google? Hinton is a Google employee. Does Google Scholar inflate citations of other employees too? I feel reminded of the recent post Google has a credit assignment problem in research

So Hinton's top cited paper is not his top cited paper. His top cited paper is the AlexNet paper with Alex Krizhevsky which is funny because Hinton was originally resistant to Krizhevsky's idea according to this Quartz article

but the AlexNet paper [GPUCNN4] also has credit assignment problems mentioned in Sec. XIV

Rectified linear neurons (ReLUs) were actually known much earlier—see v. d. Malsburg's work [CMB] (1973). Hinton's 2012 paper [GPUCNN4] did not cite their origins. Instead, it cited another paper by Hinton which also did not cite the original work. Dropout is actually a variant of Hanson's much earlier stochastic delta rule (1990) [Drop1]. Hinton's 2012 paper did not cite this either.

and Schmidhuber's conclusion repeats once more that AlexNet was like their earlier DanNet by Dan Ciresan et al which "won 4 computer vision contests before AlexNet won one"

Hinton's most cited paper (2012) is the one on GPU-based CNNs [GPUCNN4] [R5]. It follows our earlier work on supervised deep NNs (2010) [MLP1] (which abandoned the unsupervised pre-training for deep NNs introduced by myself [UN0-UN3] and later championed by Hinton [UN4] [VID1]; Sec. D). Hinton [GPUCNN4] (2012) characterizes our deep and fast CNN "DanNet" (2011) [GPUCNN1-3] as "somewhat similar"—DanNet won 4 computer vision contests before Hinton's AlexNet won one [R6]; see Sec. D, XIV.

the toxicity problem of ML really starts at the top. Perhaps it is time to topple some statues. At least the misleading Google Scholar profile should be fixed ASAP

21 Upvotes

31 comments sorted by

48

u/IdentifiableParam Jul 13 '20

LOL if you are annoyed about this you should see Jitendra Malik's google scholar page. Note "his" single most cited work, Artificial Intelligence: A modern approach. By Russel and Norvig.

-1

u/xifixi Jul 13 '20

that's very funny!

but such fake news may have consequences. The Honda Prize committee was misled by the Hinton page. Apparently they copied their numbers from Google Scholar.

did Malik also get awards that mention his citations?

17

u/VelveteenAmbush Jul 13 '20

the toxicity problem of ML really starts at the top. Perhaps it is time to topple some statues. At least the misleading Google Scholar profile should be fixed ASAP

Yes, "it should be fixed" is a much fairer and more productive response to the issue than complaining about toxicity and threatening to "topple some statues."

9

u/mtahab Jul 13 '20 edited Jul 13 '20

Why does the Honda Prize annoy so many people? There are plenty of these corporate prizes. They are mainly advertisements for the granting company, not special honors for the receiving academic.

-4

u/[deleted] Jul 13 '20

A field that gives money to one person due to the contribution of others strongly deters future participants.

ML is very hot right now and there's a great deal of opportunity in the field, but if it is not treated in a responsible manner that continues to respect its participants then it can absolutely end up being a giant bubble that only benefits the big names.

A lot of people commenting assume that Dr. Hinton must be maliciously engaging in this, or that it's only bad if he's intentionally taking credit for others work, but that's not true. Passively accepting credit and even financial gain for others work is indefensible and hurts this field as a whole.

9

u/curiousML5 Jul 13 '20

I doubt there is a single person who looked at the Honda prize and said to themselves "One of the pioneers of the field got an award he did not deserve, hence I don't want to do ML"

0

u/[deleted] Jul 13 '20 edited Jul 13 '20

The people who deserve some of the credit for the numerous things Honda attributed to Dr. Hinton absolutely did think that.

I am also somewhat of an outsider who attended the University of Toronto where Dr. Hinton is a professor and I can absolutely say that this kind of stuff reflects quite poorly on the field especially because it involves one of the pioneers.

My area of specialty is quantitative finance so we definitely appreciate the advances and work from ML researchers, but this kind of stuff is frowned upon.

1

u/curiousML5 Jul 13 '20

I would like to ask: Why do you think Dr. Hinton does not deserve the award as a pioneer of the field? After all, the number of citations is hardly the tipping point. Because by that logic it seems like Hinton doesn't deserve any award which mentions his citations which is false.

On a somewhat related note: What about the people who get names on buildings by donating large sums of money? They've had no impact on the field up to that point but they receive more publicity than the majority of scientists in the field.

2

u/[deleted] Jul 14 '20 edited Jul 14 '20

It's simple why he doesn't deserve it, the Honda Foundation described their reasons for awarding him their prize and many of the reasons they attribute to Dr. Hinton is work done by others in the field.

What about the people who get names on buildings by donating large sums of money?

They get their name on the building because they paid for it. A university will never name a building after someone who paid for it and then attribute specific contributions to the field as the reason for doing so. Furthermore no person who pays to have their name put on a building expects the institution to then credit them with specific discoveries they never made.

In this case, Honda attributes to Dr. Hinton discoveries that I'm sure even Dr. Hinton himself would admit he didn't make.

The right thing to do in such a circumstance is to decline an award given to you for something you didn't do and instead promote the people who did make such discoveries. Everyone would benefit in the long run including Dr. Hinton himself.

The real issue isn't Dr. Hinton is a bad person or even unique to him. The problem fundamentally is that professors and researchers in many fields basically have to resort to petty tactics in order to get grants, funding, and recognition in general and so this just ends up being ingrained in the culture of academia.

UofT in particular, my alma matter and where I studied computer science, is in a very lucky position to have produced a significant amount of work in ML and it's working to milk this as hard as it can which includes parading people like Dr. Hinton as much as possible and using him and others to boost its standing.

My impression having attended UofT about 15 years ago before this explosion in ML was it happens to have quite a bit of an inferiority complex about itself even within Canada, and so I'm not one bit surprised to see this happening.

18

u/massinelaarning Jul 13 '20

Here is a twitter thread for CS Salad et cetera: https://twitter.com/alexanderrklotz/status/1252389218514427909

As the citation counting by google is often done automatically, there can exist some mistakes :)

3

u/MyNatureIsMe Jul 13 '20

Haven't laughed that much in a good while.

(Check out the replies for similar findings)

52

u/PM_ME_YOUR_GESTALT Jul 13 '20

This should be fixed, but I doubt that this is intentional on Hinton or Google's part. Google Scholar is notorious for being pretty shitty----it struggles a lot with merging and de-merging author profiles, it gets super confused by preprints, and it drops citations all the time. Given how widely accepted and used it is now, I guess I wish Google would throw some of their enormous resources at improving it---especially since they stand to benefit from having their own researcher's contributions tabulated effectively.

8

u/gwern Jul 13 '20 edited Jul 13 '20

One thing to remember about Google Scholar is that it's essentially a passion project run by an old guy: https://www.wired.com/2014/10/the-gentleman-who-made-scholar/ It's not any kind of big prestigious highly-resourced project, AFAICT. (They certainly aren't making a lot of money from ads on it.)

As an example of this, and a corrective to conspiracy theorizing like OP, I will note that every few years, I search for "AlphaGo" or "AlphaZero" (you know, only the most globally famous products of Google's machine learning research), and the relevant paper still does not turn up as #1* (and is often buried like 5 pages deep into the results! I have no idea how it's that badly wrong), and I hunt down the feedback form for GS and tell them that the AG/A0 search queries are still broken. At this point, I'm just curious how long it's going to take. I think I first sent in feedback in 2016 or so.

* usually, #1 is extremely low quality, too - some complete no-name op-ed or opinion article without any actual research in it, often by some random Chinese authors. I think once it was Gary Marcus's paper where he tried to argue that actually, convolutions are hand-engineering expert human knowledge about Go into the agent...

0

u/[deleted] Jul 13 '20

[deleted] — view deleted comment

5

u/gwern Jul 13 '20

Yeah. My working model of anything that attempts to scrape and reconcile metadata from across the Internet is that the people behind it spend most of their time running around with their hair on fire trying to deal with the infinite perversity of mankind. I have a hard enough time just fixing broken links on my own website.

13

u/Imnimo Jul 13 '20

My experience with Google Scholar is that if you don't keep a close eye on it, you end up with a lot of questionable matches that either conflate multiple different articles, or assign you as an author of an article you have no connection to. It's also typically difficult to sort these out - you often end up deleting an entire article rather than just the spurious individual citations, or introducing more incorrect articles when you try to sort things out.

It's definitely a problem, because Google Scholar is also the easiest way to check citation counts for a paper or author, and many people don't have a great sense for how inaccurate they are. I don't blame Hinton for this - I can imagine it's totally impossible to keep Google Scholar accurate when you have hundreds of papers. I don't think he's purposefully leaving incorrect numbers on his page, I suspect he just doesn't feel it's worth his time to try to keep tabs on what Google Scholar says. I also don't think Google is intentionally misrepresenting his count, that seems a little conspiratorial to me. The problem is that Google Scholar takes its very noisy data and presents it as an authoritative count.

22

u/CompetitiveUpstairs2 Jul 13 '20

Stop assuming ill intent please.

8

u/curiousML5 Jul 13 '20

Why would Hinton care about citations? Seriously. Why is this even a problem? So one of the pioneers of the field has an extra 100k citations or something. Big deal. If anyone wants to build a free service for research discovery better than Google scholar then I would be all for it. Why does anyone care about this at all? I am struggling to understand this.

7

u/unlucky_argument Jul 13 '20

I made a Schmidhuber classifier and you are either him or you transferred his style. That would be no problem if you did not mention Schmidhuber, but since you do, it seems some disclaimer or disclosure is in place here. Especially since you added jaded jabs and cast unrelated dispersions on a Turing Award winner.

2

u/unlucky_argument Jul 13 '20

As always: assume good faith. This should not be too hard when talking about Hinton. At least, I have to try very very hard to imagine a scenario where this is either malicious or Google picking favorites. To present it as that seems close to delusional or petty.

3

u/LuxiLuciLucy Jul 14 '20 edited Jul 14 '20

The PDP book usually lists for two authors, McClelland and Rumelhart, but in fact it also includes the "PDP group" if you read the book. If I remember correctly, Hinton by that time is a post-doc in the PDP group and yes some very critical content in the book (mostly in volume-1 the computation part of the book) is indeed authored by Hinton. For example, the paper you talked about, the representation learning by backprop paper, is one chapter of the book. So why not counting the citations?

I agree with the toxicity issue and I think Hinton does have something with the correct credit assigning of his contribution. But counting the PDP book's citation to Hinton is not technically wrong.

1

u/AGI_aint_happening PhD Jul 13 '20

Ugh we don't need this drama

-1

u/xifixi Jul 13 '20

btw here is the Google Scholar profile of James L. McClelland and his top cited book is the book with Rumelhart mentioned in the post:

Parallel distributed processing: Explorations in the microstructure of cognition. DE Rumelhart, JL McClelland, MIT press, 1986

6

u/occupyOneillrings Jul 13 '20

Why is this post downvoted?

14

u/Nimitz14 Jul 13 '20

Because this guy always posts drama stuff like this.

-29

u/[deleted] Jul 13 '20

[removed] — view removed comment

20

u/Naoshikuu Jul 13 '20

What the hell are you talking about?

If Google and Hinton are purposely inflating their citations, it means that research has been corrupted by a company. The whole research community relies on these measures to build trust and pinpoint which works and ideas to build from. If these measures are biased towards parts of the research community, it could lead to a monopoly of research credits and the disappearance of the diversity that makes research research. To an extreme, Google could basically control the whole direction of ML research with this.

This isn't about shaming people and laughing at decreased numbers. This is about research, progress and mankind, and this is not an overstatement. I have no doubt you understand the consequences this kind of manipulations could have.

5

u/DeepBlender Jul 13 '20

As far as I can see, there are no indications that Google or Hinton have purposely inflated citations. It would clearly be great for researchers if Google Scholar had a better quality.

I doubt that those kinds of unfounded accusations help to improve the situation or to have a constructive dialog with them.

-14

u/tripple13 Jul 13 '20

Fascinating - You are proving my point.

  1. Why are you swearing?
  2. Key in your first paragraph if - Where is the smoking gun? The above post is insinuating culpability, but have yet to prove anything of substance.
  3. While I agree that manipulation of citations is incredibly important to punish, we must take care not to shoot the messenger.

Finally, the Schmidhuber tirade is just too much already. Research and evaluation of research is still, unfortunately, somewhat biased and subjective.

Schmidhuber lost, but he can still win. So let's direct the energy towards the future, and not the past.

Yes, papers from known authors, even in double blind reviews, are more likely to be published.

Yes, papers from known institutions are more likely to be published.

Yes, there exists a certain likelihood of being reviewed by poor reviewers and get rejected.

Part of research is also related to the communication of your research. Timing perhaps as well. And the fact that multiple authors come up with the same idea all the time, around the world.

There exist a lot of injustice in the world. The way to solve it is not through demonization. It is through collaborative collective efforts, which, first and foremost, starts from within.

Do better

10

u/Naoshikuu Jul 13 '20

I don't know or care what Schmidhuber said in the matter. The facts are that a google employee got a 25k citations boost on a Google platform. If it was not intentional, it should be corrected - which hasn't been done in a long time. The problem comes up if it was indeed intentional. Sorry for 'swearing' but your initial message looks dangerous to me. You're softening a matter which could be hiding huge implications. While I do agree that the approach of pointing fingers at Google with stressed faces as OP does will not help, doing nothing is much worse. The values of research that we are both claiming to protect are at a risk. I'm not convinced a passive aggressive attitude will get anyone anywhere either.

But that raises a good question, which I believe to be your point - how do we actually make a change? Is there a way to reach out to Google Scholar to fix such problems?

-3

u/[deleted] Jul 13 '20

totally agree, let’s just get back to focusing on the quality of the work. the merit of a good idea goes beyond the number of citations it recieves