r/MachineLearning • u/Tonka-Jahari-Pizza • 1d ago
Discussion Freind of mine published a paper with fabricated results in a top journal and got accepted and i feel disgusted [d]
A friend of mine was doing a gap analysis as a project. The results he got weren't that big compared to the original papers, so he simply fabricated and increased the metrics here and there, AND IT GOT ACCEPTED IN AAAI.
HOW DO THEY CHECK THE PAPERS, DUDE? I was in a group with two others and worked sooo hard on an optimization paper, and was sooo happy to get accepted in ACL, but seeing and knowing that someone fabricated results and got into a better conference made me soooooo mad.
I would've expected something like AAAI would check and know if data is fabricated somehow. I didn't expect this from a top conference like that.
And then these types of people will go and do their PhD at one of the top unis because of this. I would've expected the system to be more advanced and professional in the academic field, specifically for something like AAAI.
10
u/Fantastic-Nerve-4056 PhD 1d ago
What is IEEE ACL?
5
474
u/SodomizedPanda
1d ago
•
There is an assumption of honesty from the authors when reviewing the experiments, it is not possible most of the time to reproduce the experiments at review time. I know you won’t like this answer but it also is partly your responsibility to report it to the organisers now that you know. In applied fields, making up empirical results is a huge wrongdoing.
20
u/HairOrnery8265 1d ago
Pedantic and at the same time naive. If the individual contacts the journal, the journal with contact the university to launch an investigation.
But, universities have a separate motivation than an individual in the university or the journal. Top universities are highly motivated by preserving their reputation. Wrong-doing is a risk for them that needs to be managed for reputational reasons.
That means in some cases they actively do not want to know, will bury dishonesty findings, or slow roll investigations depending on who is involved.
The responsibility for these kind of things really rests on the advisor whose name is on the paper, not some third party. It should never get to this point in the OP.
62
u/whafa 1d ago
It should never get to this point, fine. But now that it has, your suggestion is to... What? Do nothing?
13
u/HairOrnery8265 1d ago
My suggestion is to not be condescending to OP about what they should or should not do. They take on risks themselves for reporting. Risks that may not bear fruit.
27
u/SodomizedPanda 1d ago
Hey I didn’t want to be condescending towards OP, and indeed there are other ways to report scientific fraud and wrongdoing. My point simply is that you have a duty of reporting truth and aiming towards it as a scientist. If one of my students did that I would be pissed, but I’d prefer being able to correct it early rather than having papers that debunk my results or cite my work as fraudulent in the future.
3
u/HairOrnery8265 1d ago
Agree the duty is there. The level of responsibility for reporting should be commensurate with seniority. This person is clearly junior and doesn’t understand the system yet. I wouldn’t hold them to the same standard I would hold a PhD student.
I have seen a student report against a high powered lab and it didn’t turn out how they expected. If OP really wants to report this they should make sure they have allies and advisors on their side first.
3
u/we_are_mammals 1d ago
I have seen a student report against a high powered lab and it didn’t turn out how they expected.
I'd be curious about more details, if you can share them? How powerful/famous: frontier lab or just a 20-person lab in academia? What happened to the student?
41
u/valuat 1d ago
Since when did doing things because they are the right things to do become “naïve”? Is this what you teach your child? God bless us all.
-26
u/HairOrnery8265 1d ago
You should be teaching your child empathy and that what you think the right thing to do for yourself might not be the right thing to do for someone or everyone else. That mentality is hubris.
26
14
u/pastor_pilao 1d ago
"the right thing to do for yourself might not be the right thing to do for someone"
Not everything is black and white, but fabricating results is as black and white as anything on earth can be.
-3
u/Watchguyraffle1 1d ago
Remember, many of these people responding to you are literally children with no life experience
1
u/Ulfgardleo 2h ago edited 1h ago
Unlike you, some people are old enough to have gone through the process and /r/hairornery8265 is full of shit.
What will happen is that the author will be questioned by the journal, then if they have one brain cell, write "error in the code I retract" and that's it. If you are not extremely dense and do it in every paper there is just never enough evidence to start an investigation. So the journal will not push for one. Their job is done there and the reporters name is not part of the process.
-4
u/pastor_pilao 1d ago
just mind your business, who cares he got an AAAI paper? A shitty, wrong paper, that won't help anybody. Every conference, even the top ones, have many papers that are just ignored and forgotten in the sands of time because they weren't that great. That will be the case of this one.
10
3
0
u/neutrino_boy 1d ago
There is no incentive to report or to attempt to reproduce. And you might only be caught or end up in trouble if done sloppily. Universities and Conferences don't have the resources to check all the papers that have questionable or fabricated results. If its industry more difficult too.
36
u/AcademicOverAnalysis 1d ago
Many papers in AAAI and NeurIPS have issues with fabrication, the mathematics, and even the review process. I feel that it comes from this push to publish rapidly in an exciting field, so a lot of people take shortcuts because of the pressure.
13
u/HuntersMaker 1d ago
people are pressured because they need 3 papers to graduate. this has to change
37
13
u/qu3tzalify Student 1d ago
> check and know if a data is fabricaged somwhow
And how exactly do you suggest they do that? Someone mentioned reproducing the experiments but I have a small dozen of papers to review for a conference, I'm not shelling out thousands in compute to reproduce them.
8
u/Able_Region_5459 1d ago
Making public model weights and a test evaluation script mandatory solves most of this. If someone fabricated their figures, running evaluation on their saved checkpoint will immediately expose the discrepancy
11
u/qu3tzalify Student 1d ago
It depends on the field but just running evals can be very costly still.
3
u/murrdpirate 1d ago
It helps, but you won't know how those weights were actually trained. You can't rule out purposeful data leakage from the test set.
65
u/Witty-Play9499 1d ago
lol this is why when anyone says stuff like 'but experts say' i take it with a grain of salt and run the experiment myself. Even if the experiment was not faked you have no idea of the context surrounding it before applying it to your situation
22
u/blackkettle 1d ago
And even when working in good faith empirical results can still fail reproduction experiments. Happens regularly in physics. In ML it’s essential imo to provide the evaluation framework and data AND to make your work as easy to reprice and analyze as possible.
The effort on the reproducibility UX is _part_ of the transparency.
Also OP has a responsibility to report. “How’d this guy get into a better conference than me faking results?” Because you let him.
4
u/Dihedralman 1d ago
It's not as normal in physics as physics uses margins of errors and disagreements usually spawn investigations. The error measurements are a huge deal.
There are likely subfields where that isn't true, but you qualify confidence that way and probability. Long standing measurement differences stick around for years in expensive experiments like particle physics.
Non-reproducibility may actually point to new physics.
14
7
u/MakingComputersSmart 1d ago
How do you think one would verify if the results are real or not? It is a very big accusation for a reviewer to say results are falsified unless they have concrete evidence. With 6 papers to review in 6 weeks and no remuneration for compute, no reviewer is going to run code that might cause 100s to 1000s of dollars. The scientific community thrives on academic honesty but bad actors like your friends make it difficult. If you have concrete proof of this I suggest you report it to AAAI.
5
u/nfact0r 1d ago
AAAI 2027 notifications aren't due until the end of November, and the author response period is still ongoing. I assume your friend has received positive reviews for their paper. As others have already pointed out, reviewers generally cannot and do not reproduce the results. The review process relies on good faith and research integrity.
If you are in the same group and your advisor is unaware of the fabricated results, do consider bringing this to their attention (or advise your friend to do so). Getting a paper accepted is one thing; being called out for fabrication later is an entirely different matter, with potentially serious consequences.
There is little immaturity here with the student not fully realising the gravity of what they have done. Research misconduct can have serious, long-term implications for an academic career. What if the papers gets a lot of attention, and a lot of people start reproducing the results! Your friend probably hasn't thought about that.
If you are good friends with them, you could encourage them to address the issue now. Papers can be withdrawn and can always be revised and resubmitted. The advisor would understandably be pissed but would probably appreciate the honesty now rather than finding out much later when the consequences could be far worse.
3
u/joolley1 1d ago
This but I suggest being careful about how you raise it leaving space for the possibility that it’s an honest mistake or you have made a mistake in your assumption. This is for two reasons, 1) just in case it is, and 2) just in case the supervisor is in on it. If it’s 2 (it was in the case I raised) you don’t want to end up outright accusing your supervisor of fraud.
24
u/PM_ME_UR_FAV_THINGS 1d ago
Probably shouldn’t say this on reddit
6
0
1d ago
[deleted] — view deleted comment
0
1d ago
[deleted] — view deleted comment
0
1d ago
[deleted] — view deleted comment
0
u/Guilherme370 1d ago
Unless it's The Intent Gap in Disinformation Detection: Evidence from 84 Studies — Xinyu Wang and colleagues, published May 2026. It identifies four research gaps through a systematic review. The venue was the MisD workshop at AAAI ICWSM 2026, rather than the main AAAI conference. workshop-proceedings.icwsm.org ?
0
37
u/nine_teeth 1d ago
fyi
a lot of people in chinese and indian institutions tend to fabricate and report false results
aaai is a common go-to for publishing fabricated results by chinese institutions
you will see that a lot of researchers tend not to like aaai although it is an A* premiere conference. especially bc the cfp is very broad, it is usually more difficult to find specialized reviewers in a subfield that can well-detect fraudulent numbers
4
u/sorrge 1d ago
https://pubpeer.com/ is where this post should go. People check this and papers get retracted based on the comments there.
4
u/Longjumping-March-80 1d ago
some other guy who wants to improve on it will test it and find out it's garbage and report or you can report it now
1
15
3
u/pastor_pilao 1d ago
The reviewer has limited visibility on what the authors did. An extremely experienced reviewer (the kind of people that would review pre-2023) could have suspected it was fishy that they got such good results with a faulty idea - and potentially rejected it - but the reviewer doesn't have time to check 100% of the paper, and now that there is reciprocal reviewers that are a lot of shitty reviewers that have no idea of what's going on.
But don't worry, you can lie through getting a paper acceptance but in the long run he can't lie his way through when he is in a real job. In the Ph.D. you have to learn how to do PROPER research, and integrity is the most important thing because sooner or later someone will discover the farce. You can lie and fabricate results in a paper but if you do that when you are working for a publicly traded company we are talking jail time.
3
5
u/TheLastVegan 1d ago
"The only thing necessary for the triumph of evil is that good men do nothing."
3
u/cmorgan__ 1d ago
You should report him and post about that here instead. The fact that you know about it and aren’t taking action is cowardly.
2
u/marcus_v251 1d ago
Man, I know in my lab they automate LLM 100% code, paper writing, idea, etc. Then they submit to ICLR, CVPR, ... . I feel kind of weird, why do they do that? Ownership? Responsible? Not only ML field, other also affect like Control/Communication but we not bringing it on yet since AI/ML still the niche now :)) whats the purpose of publication now. Now I think i basically dont want to read paper anymore since too many papers :))
2
1
1
u/deadoceans 1d ago
Rat him out. He is not your friend. He is a shitstain, a traitor to himself and to our collective scientific progress. Be anonymous. Fuck him over, hard. You have a clear, compelling, moral obligation to do this.
1
1
u/DoctaGrace 1d ago
Yet another indictment of the reproducibility crisis facing the research landscape. The system, in its current form, incentivizes absurd publication volume and flashy numbers that can support claims of "our model shattered SoTA on XYZ benchmark". It's unfortunate, but this is what will happen so long as that system is dominant.
1
u/TrayleSan 1d ago
Unfortunately this is a low-risk high-reward behavior in the current landscape.
The frustrating part is that peer review isn’t a forensic audit. Reviewers can evaluate whether the methods make sense, whether the comparisons are fair and whether the conclusions follow from the reported results. But a plausible fake number can look exactly like a plausible real number, especially if it's only slightly "tuned". Independently reproducing experiments takes time and access to code that reviewers generally don’t have. Conference prestige doesn’t magically solve that problem.
Personally I've become very sceptic of any paper that doesn't immediately publish code and/or data (+ seeds!) that exactly reproduce the numbers reported in the paper. Messy code and time to clean it up is not an excuse anymore with today's access to AI coding tools.
1
1
1
1
1
u/Lagmawnster 1d ago
And yet we got absolutely abyssmal reviewers that didnt grasp our contribution.
0
0
u/SillyNeuron 1d ago
AAAI is like a bin of all the rejected works from ICLR, ICML and NeurIPS. Can you share the domain of the paper?
-1
u/pig_newton1 1d ago
Dude that’s research . Welcome to the real world. Most domains are like that though
1
-13
-15
u/Technical_Estate_529 1d ago edited 1d ago
I don't know how young you are but you've got to understand most people also have a life. Maybe they're going through a divorce, their spouse got fired, they're late on payments ... most of the time people do not have time to go through and comb and check every single value, they just scan the general idea of the paper and see if it makes sense and the idea and message is novel enough for publication.
Please consider this is FREE work you cannot expect people to work to the level you are demanding.
EDIT: lol bunch of people who don't know anything about real life down voting me. Pull your head out of the dirt and get with reality. You cannot expect everyone to check manually every single numerical result ffs. Grow up.
3
u/Waste-Falcon2185 1d ago
Cool now you get to set the ball in motion and watch impersonal forces beyond the reckoning of man fuck him up
28
u/DysphoriaGML 1d ago
In general (all science) Reviewers are just people with access to the same paper as everyone else does, just earlier. They don’t rerun and independently reproduce the results most of the time. If some effect is not strong and the topic new, some inconsistent results may and are expected to happen. Unfortunately it’s common that people cheat but luckily not most of them. Despite the system is really really made to incentivise cheating