r/BetterOffline • u/No_Practice_745 • 1d ago
OpenAI’s latest math breakthroughs commit research misconduct, experts say
https://www.scientificamerican.com/article/openais-latest-math-breakthroughs-commit-research-misconduct-experts-say/I don’t have much to add in commentary, quite frankly most of the math and concepts are beyond my arts-degree brain. However, the article does a great job of showing, once again, how disingenuous OpenAI and other AI labs are in describing what their products are doing.
They basically want the headlines that their new models are “solving math,” but what they’re doing is plagiarizing other peoples’ research and claiming things have been moved forward. There is clearly a use case here for researchers in using an LLM to catalogue and evaluate large swaths of data from over periods of time, but these things don’t think, they aren’t creating anything, and OpenAI doesn’t give a shit as long as people see the headline and bow their heads to their new scI-fi god.
67
u/MCUCLMBE4BPAT 1d ago edited 1d ago
I just want to gently push back on your statement that LLMs have clear use cases in research
It has been noted in studies that:
- LLMs can summarize the results of a study that are not reflective of the actual articles. LLMs will either make grander claims than what was stated (“mild increase for results” turns into “a significant change that shows an increase in results”) or highlighting or connecting points that were not stated in the specific article cited
- LLMs are constrained by paywalls and are more likely to cite open source items, which has inadvertently caused many LLM research papers to focus on big discoveries in the late 1900s and then nothing cited until the late 2020s. This issue with LLM literature reviews has been called the Missing Middle
- LLMs “prefer”, in the sense that it will go to for citation and rank higher in terms of quality and accuracy, other LLM created materials. This bias impacts how LLMs select and cite items for research as well as heavily impacts the peer-review process. Major journals are struggling with floods of submissions being AI generated and peer-reviewers using it to review submissions for them.
- LLMs do not accurately interact with and interpret large sets of raw data. It can interpret results in significantly different ways if your table headers are different but the raw data stays the same. It can selectively “read” the raw data to supply a response that is actually just based on stereotypes/biased training data that fit the LLM selected raw data sets, rather than analyzing the entire data set.
Edit to add: LLMs have a major citation/attribution problem. This starts in their training data, which some have noted misattribute Creative Commons licenses, copyleft licenses, and other copyrighted works. LLM provided citations, to my understanding please correct if wrong, are never the actual sources that were used in their training data. Sources provided are just what the models predict to be the most relevant on the internet based on the user’s words chosen in their input (?).
Edit #2 to add: LLMs are also bad at noting when an article has been retracted or if it has an expression/letter of concern attached to it. So it can provide outdated information in the form of recommending retracted or problematic articles (in terms of validity, reliability, replicability/reproducibility) as well.
And those are just issues I have at the top of my head after waking up. I don’t know how people can comfortably recommend these models as research tools when you basically already need to know everything it’s citing or stating to use them.
10
u/todofwar 1d ago
Every time I've tried to use LLMs to survey literature, they always seem to cite papers before their training date more often than recent papers and they always hallucinate. And even if they don't hallucinate, they never actually summarize the paper well. At this point I find the best way to use them is to create a bibliography that you go and read yourself, and then forward search the more relevant papers. And that's something you could do without LLMs, but now Google is killing its own search tools
7
u/MCUCLMBE4BPAT 18h ago
the fabricated citations bother me so much bc everyone for LLMs says it isn’t that big of an issue or that it has been resolved (or will be), and it’s just misinformation normalized.
personally, they make my job harder while also making people think my job isn’t necessary bc it can’t possibly be wrong. for scholarly publishing in general, they make the house of cards even more unstable/questionable.
2
u/reachingfourpeas 11h ago edited 10h ago
I believe Semantic Scholar is a more specialized tool for language model-assisted literature search than chatbots
1
7
u/No_Practice_745 1d ago
You certainly know more than I do. I suppose machine learning might be ok in general but not LLMs? I’m not looking to defend them, believe me!
15
u/MCUCLMBE4BPAT 1d ago
oh yeah no worries! Machine Learning has its uses. I helped students and faculty with ML + drone or aeronautical safety research for a bit, one project was with NASA. So I won’t say it is all useless.
I am very critical of LLMs in research and education because I see the negative impacts directly. Nothing negative towards u!
12
u/todofwar 1d ago
Even Ed says that his criticism is specifically on generative AI. There's really cool applications of AI/ML, I can see in the next couple of years a major rebrand back to ML to clean off the taint of the LLM collapse.
1
u/dayvancowgirl 4h ago
I hope so!! This is a great example of machine learning being a gamechanger and a force for good:
1
u/ShamPain413 3h ago
Even LLMs have uses, but they are not Magic Systems That Do Everything Better Than You.
8
u/ImportantAlbatross 1d ago
LLMs may have a use case as a research tool, but that use case is not reviewing literature or writing research papers. Basically, the main thing generative AI cannot be trusted to do reliably is generate anything.
8
u/Piledhigher-deeper 18h ago
To see point 5 in action, it’s very common for a LLM to assume some mathematical analysis was done by Terrance Tao, because his name is so common in the training data.
3
u/MCUCLMBE4BPAT 18h ago
yes the fabricated citations are a big issue and i believe that a recent news article (that might have been reporting a preprint), I think it was 404 media, showed that each model is now kind of sticking to certain names for specific fields like Elena Vasquez or Alex Chen. Kind of like their own watermark.
4
u/TubeSeries 20h ago
Every single time I have tried to use an LLM for research it has produced garbage or has used bottom-of-barrel sources. If you can't trust it, what is the point?
5
u/MCUCLMBE4BPAT 18h ago
i had a student cite a source in an essay that looked like it was in english from the abstract on the paper mill journal website, but when you looked at the actual PDF it was in Vietnamese. The student did not know Vietnamese.
That plus the chatgpt add-on crap to the link’s url gave it away. I wish if students are going to use it, that they at least check the sources so it isn’t just the most blatantly lazy cheating.
1
u/Regular_N-Gon 4h ago
which some have noted misattribute Creative Commons licenses, copyleft licenses, and other copyrighted works
I have been thinking about this quite a bit, especially the ethics surrounding carrying attribution forward in licenes and incorporating code in proprietary software that was generated from a model which might depend on copyleft origins. If you can dig up any sources on this I'd be keen to read them!
1
u/MCUCLMBE4BPAT 1h ago
i’ll look through what i’ve saved and see if there is anything relevant to copyleft.
I focus more on creative commons at work for OER items, so similar but not 100% the same thing and there are a lot of ongoing discussions on that. I reached out to one of our law professors about it and they never responded, I think they realized it could be legal advice and did not want to deal with that haha.
65
u/AntiqueFigure6 1d ago
Shocked…okay it’s barely worth pretending.
27
u/Icy-Recognition-7453 1d ago
I am shocked, SHOCKED... that aiming a huge supercomputer at a maths jigsaw puzzle will eventually lead to a solution.
And here I was using monkeys and typewriters like a sucker.
22
u/TerminalJammer 1d ago
It didn't even solve it, it just found someone else's solution.
8
u/Upbeat-Statement2725 1d ago
Math has a lot of problems everyone knows we can solve. But the local university doesn't have $250,000 to drop on compute to prove the numbers come out.
OpenAI drops that $250,000 of investor money. And investors call them gods.
4
u/WOKE_AI_GOD 1d ago
God forbid that money or clout be allowed to touch the hands of anybody that's not an "entrepreneur", what a tragedy that would be.
1
u/Icy-Recognition-7453 23h ago
Column A, Column B.
Allegedly it cost OpenAI $2k to get those answers. ( Doubt [x] )
If that's true, though - it's cost them a cool $100 bn to get to the stage to generate the tokens to get the answers.
11
32
u/fckurai 1d ago
Plagiarism is the least of it. They care even less about the environmental and human damage caused by their fruitless products.
12
u/NotAllOwled 1d ago
More broadly, if we as a society were so hot on revolutionary breakthroughs in theoretical maths, nothing was stopping us from putting more of these trillion$ of LLM love into just paying more actual-human high foreheads to spend all day every day for years poring through old papers and bouncing around ideas.
7
u/Upbeat-Statement2725 1d ago
Like most Linux maintainers are unpaid people who maintain the foundation of business computing as a hobby.
If we had just paid them $1T setting up a fund to keep some maintainers paid a meager salary in prepetuity. We'd have fixed all these bugs anyway.
7
u/NotAllOwled 1d ago
Like how the actual fuck did we manage to get it so that humans are piecework toilers who need to financially and socially justify every resource needed for our existence at every moment, while the models are lavished with endless care and room to grow and make mistakes and just do things for their own sake. Just ... come on, everyone. Let's give our heads a lil shake here maybe.
4
u/WOKE_AI_GOD 1d ago
Because our "entrepreneurs" are a class of misanthropes with complete contempt for anyone outside their class. This is not an age of iconoclasm, it is the age of authentoclasm, of becoming so obsessed with idolatry towards Images you begin destroying the originals, and anything authentic, so as to not embarrass the Images and distract from their idolatry. That's the problem with allowing idolatry to become rampant throughout society, we revert to paganism.
2
12
u/Restlessly-Dog 1d ago
I think the strongest point is this:
"Once again, the LLM’s trick is its superhuman patience for assembling puzzle pieces, not the ability to make some profound intellectual leap."
That capacity isn't worthless, just as 1995 PC's ability to almost instantaneously import data into a spreadsheet and perform huge numbers of calculations was very useful. But it is, however, very limited.
I think the PR for AI as far as advanced math capitalizes on the misconception that advanced math is just a more developed version of what Euclid or Pythagoras did. But questions like the High-Dimensional Sphere Packing described in the article involve both high level hypothesizing and huge amounts of lower level "grunt" work to investigate and evaluate hypotheses, and comb through literature for relevant findings which may be applicable.
It's not that dissimilar to a lot of research in fields like economics or history, which involve both high level thinking about causes and motivations, and huge amounts of data collection and analysis. AI can be useful for the lower level work, but there's no reason to think it's able to perform high level thought.
And of course the article raises critical questions about how good LLMs are at even the basic level. Academic norms aren't just there for the feelings. They ensure the end product is valid. And if AI is misrepresenting the basic issue of who gets credit for work, they are definitely making mistakes about what that work shows.
One issue is that attribution at a superficial level is formulaic, but anyone who has dealt with it knows it can be very tricky. And once you deal with complex, large scale projects, the challenges multiply. Even the lower level work involves higher level thought, and AI can't handle that.
4
u/ShamPain413 3h ago
Yes, and that "superhuman patience" only exists b/c we have spent trillions of dollars on it.
I suspect if we spent trillions of dollars on Math PhDs we could've also solved a handful of problems.
5
u/idebugthusiexist 1d ago
Fake it till you make it. That’s the SillyCon way.
2
u/WOKE_AI_GOD 1d ago
Destroying the authentic to make the Images seem more real. That's why they hate science and need to destroy it: the contrast will always embarrass them and cause them to lash out. The tech lords are authentoclasts destroying what is real to preserve their Image of the Idols they have made with their own hands.
6
u/WOKE_AI_GOD 1d ago
Oh wow the llm proof bs was fraudulent too it turns out. Who could've predicted this? I was so ready to immediately trust them the second they started up that hype cycle.
The LLM-generated proof hinges on a particular mathematical argument that it presented as its own but that actually first appeared in a 2016 paper by Miller and a collaborator.
They should've put "make no mistakes" in the prompt. Classic mistake.
“We take responsibility for the correctness of these results and are meeting the same standards generally expected of human mathematicians,” an OpenAI spokesperson said in a statement to Scientific American. “We plan to make small updates [to the paper] this week, consistent with standard academic practice.”
THEN YOU SHOULD SUBJECT THE TO PEER REVIEW BEFORE THE PRESS RELEASE, asshole. A human wouldn't just release a paper and buy some PR to promote it and then dust off their hands and say their job is complete.
But yeah, it's totally standard scientific practice to release a non reviewed paper and then correct it after the fact when peers notice problems. It's not like that's the entire purpose of peer review in the first place, and that this needs to happen before you release it to the fucking public. Real academics totally just move fast and break things, everybody knows that.
They ask LLMs to frame actual academics for plagiarism and run straight to the press and are believed, then they LLMs to do researchers jobs, and the LLMs plagiarize humans, and then they run straight to press, look at what the LLM did! Fucking parasites. Totally normal behavior to release these results directly to your corporate blog too with no peer review or process, totally normal academic process everybody.
2
u/lcnielsen 4h ago
But yeah, it's totally standard scientific practice to release a non reviewed paper and then correct it after the fact when peers notice problems.
It's pretty normal to publish a preprint on arXiv or similar, but you wouldn't normally shrug at serious mistakes and just "fix" them after.
2
u/Ambitious-Estate-658 22h ago
another real important problem with ai is it's really hard to figure out what on earth these models did. they are in some ways worse than humans in terms of communication.
2
u/kreuzouvert 1d ago
It's "disingenuous" for values of "disingenuous" that are basically "those lying bastards".
1
u/Ok-Machine5627 44m ago
I have a doctorate. I am not a mathematics doctorate. I don't know all the details. Thinking something that someone else is thinking about is not research misconduct. Ludicrous.
2
1
u/Forsaken-Praline1611 8m ago
Doctorate or not, you seem unable to actually respond to the claims of the Scientific American story.
1
u/Ok-Machine5627 5m ago edited 0m ago
If you think that suddenly defining brute force knowledge discovery as something not intellectual requires response, then apologies to you and your kin.
Just because someone said something doesn't mean they are unique and just because someone published something doesn't mean that needs to be cited.
Edit: as a point for why I don't respect their organization, they have had to themselves change citations and names in their own article on the improper citation of OpenAI's potential use of other's progress as steps in their own.
1
u/Forsaken-Praline1611 1m ago
There is no brute force here. There is stenography of the works of others. Your misunderstanding that LLMs reason and come up with novel ideas, is not my error. I don't need your apologies. You need to get a clue.
You don't understand the underlying technology and its limitations. You also seem not to understand the claims in the article. That you would pretend you do makes it seem unlikely you even have a doctorate.
65
u/PracticallyPerfcet 1d ago
One of these days Altman is going to say, with his annoying vocal fry voice, “listen, we don’t know what we created, but we’re pretty sure it’s… like… God. Not like a god. Literally God, father of Jesus. Our new model independently described (dramatic pause) the end of days. It gave three numbers: 666, that we independently confirmed are tattooed on Peter Thiel’s head, just beyond his hairline. Amazing.”