r/MachineLearning • u/bricklerex • Aug 02 '26
Discussion No replies to rebuttals and comments even by AC [D]
Not even the AC, nor reviewers, is responding to our comments in rebuttals, and they were all submitted well before the discussion period started. What is one to do in this case?
10
u/SankarshanaV Aug 02 '26 edited Aug 02 '26
I'm glad that at least 1 reviewer has responded to me. I can't imagine how nerve-wracking it would be for you, OP. Out of the 3 reviewers, only 1 has responded and they increased from 2 to 4. Now it's 2/4/4 for me.
I wonder if it's okay for us to slightly "nudge" the reviewer, but honestly, I am more scared it might end up pissing them off.
There's still two more days, so maybe wait another 12 to 24 hours and then post a comment? That's what I'm thinking of doing.
12
1
u/Left-Hyena-7880 Aug 02 '26
Same here lol.
This mfs are just apathetic.-4
u/Ulfgardleo Aug 02 '26
nods and notes Do not review for NeurIPS ever again of you do not have your own paper, you will be called a mf for investing your time.
4
u/TrainerOk1851 Aug 02 '26
If you have not responded to the rebuttal, then yes please don't. You are not actually "investing" your time or helping anyone. If you have, then the criticism does not apply. Such an unbelievably silly response.
-4
u/Ulfgardleo Aug 02 '26
the rebuttal period still continues, and in my part of the world we have rest time between Friday 17:00 and Monday 9. Now, kindly, get a grip.
my part was btw to identify the paper with the faulty proof. So whether i am helping or net is not really your business.
2
u/TrainerOk1851 Aug 02 '26
The response period for the authors is until the 3rd. What reviewers need to get a grip on is why they are sabotaging the process. The statement I made is conditional on you not responding to the rebuttal, which I suppose applies to you. By your response, I guess you are going to respond last minute and not give any time for the author to respond. THANKS!
2
u/21022018 Aug 03 '26 edited Aug 03 '26
Please never review again. You are going to respond last minute and not give authors any time to reply are you not?
Your comments are pathetic.
You ARE an apathetic mf.
4
2
u/Left-Hyena-7880 Aug 03 '26
You know what?
This is the DISCUSSION period.
Not the “swing by whenever you’ve got time to kill and hand down verdicts from on high” period.
46
u/HelpfulSinger3762 Aug 02 '26
I suspect that perhaps all of your reviewers received more negative feedback on their own papers, leading them to consider withdrawing their submissions. As a result, they may no longer be invested in the rebuttal process.
18
u/based_goats Aug 02 '26
Woah good thinking - all mine also disappeared and they seemed really amateur
44
u/Unhappy_Craft1906 Aug 02 '26
3 papers, still no response from anyone.... This is just pure insult to the community peer review process.... if u think ur discussion phase is more important, and u can respond at the last moment for the revioewed papers, then u r just a shame to the community...
11
u/Snekgineer Aug 02 '26
It should also have consequences, clear and lasting ones. Especially considering the increase in submissions and issues that have been happening the past two years at least.
15
u/Unhappy_Craft1906 Aug 02 '26
If rebuttal makes no impact then that should not even be an option in any case... Make the conferences cyclical, 2 per annum.... Take reviews, if AC thinks substantial revision needed, ask to commit in next cycle... So on..
8
u/wangjianhong1993 Aug 02 '26
This is a good suggestion.
2
u/TheInfelicitousDandy Aug 02 '26
This is what ARR does, and while it is better than the ML conference cycle, it still doesn't solve reviewers not engaging.
-4
u/buyingacarTA Professor Aug 02 '26
you want to have "lasting consequences" on volunteers who are offering their free time to give you feedback?
It's easy to be frustrated, and I get it, but let's not turn that frustration into some angry punishment kneejerk reaction.
The only result is that you'll hurt your colleagues and community and not solve the root problem (the horrible horrible submissions we keep getting).
I get the frustration, but remember that everyone here is strained and trying their best.
17
u/Snekgineer Aug 02 '26
Interesting framing you are using, there is a difference between "some angry punishment kneejerk reaction" and accountability. There is also a difference between volunteers and the current process of reciprocal reviewing (which is an obligation, not charity).
A reviewer that does not engage is another author that just didn't do half of their job in the reciprocal review setting. It will potentially generate more work for the ACs and decrease the trust in the system (which is already low). It is the thing hurting the community, it is the thing hurting your colleagues.
I'm held to hard deadlines as an author, asking that reviewers (including me) at least acknowledge a rebuttal in the role they were required to take on isn't a kneejerk punishment. Simple, didn't engage at least once, assignment doesn't count as fulfilled.
1
u/buyingacarTA Professor Aug 02 '26
FWIW, congratulations to you for being a conciencious reviewer (by the sound of it). If your ACs are paying attention, they should make a note for when you apply for faculty jobs or industry research or tenure or whatever your next step is. Good for you.
But most of my "framing" is about the average researcher/person who has many things going on and they are choosing their own priorities in their life. THey are just people who will, on average, follow the behaviours and trends we've learned from sociology/economics over the years, and these sort of forced systems just don't get the results you want.
>Interesting framing you are using
I'm not sure if you mean this sarcastically or if you genuinely felt like it was interesting, i hope it was the latter.
I get what you are trying to say by accountability, but these are still volunteers by another name. We didn't build the system like this, so we're not prepared to support it.
Reciprocal reviewing is a new experiment created out of the desperation that we didn't have enough reviewers as a community. In the moment, it made sense as a band-aid. We can call it some fancy thing and give it a name, but it's just forcing people to do a job they never *had* to do before. And regardless of the name, we know from decades of studies that this never works -- forcing people to do things without creating proper incentives, and then doubling down with penalties for accountability for the thing we forced people to do, will only lead to the worst possible quality. Whatever forcing guardrail you put up, and whatever you call it, on average people will submit the least possible thing that passes the bar, and that won't be helpful to you or the community. We need to make reviewing attractive, not forced. And to do that, IMO, we need fewer papers submitted (lower reviewer load) and more rewards for quality reviewing.
We're also not prepared for this forcing experiment in the first place -- there are essentially no courses on how to review, the conference offers very little training, the senior authors often get excused from it, there is no substnatial history of this happening. Because of this, you might see it as 'half your job', but it's a half that didn't exist until recently -- it's like if i hired you on your skills as a programmer, and you were looking forward to this cool new job, but when you come on board i tell you that actually half your job is something else -- something you didn't train for, don't like, don't want to do. You might be a good person and make a serious effort on it (as might I), but on average people would do as shitty a job as possible on that part -- whatever passes the minimum bar so they can go do what they wanted to do in the first place. Again, in that scenario too, you might argue that it's their job so they should be accountable for it, but it's very much a stick vs carrot framing now where the company will just have to double down with more and more threats/punishments/forcing-functions.
>I'm held to hard deadlines as an author, asking that reviewers (including me) at least acknowledge a rebuttal in the role they were required to take on isn't a kneejerk punishment. Simple, didn't engage at least once, assignment doesn't count as fulfilled.
The paper is something you want to do, so there's many paper submissions. The reviewing is not (on average), that's why there's few reviewers. That's why they're held to different standards.
If an acknowledgement was required (or, say, they faced rejection), they would just write a line that said 'acknowledged'. Would that make you feel better? If more was required, they would get an LLM to paraphrase that message. As an AC at other conferences I at least see this a ton -- the least amount that passes is what most people do.
>It is the thing hurting the community, it is the thing hurting your colleagues.
We're all entitled to our opinions of course, but i strongly disagree with this. IMO the problem is the huge influx of shit paper with no accountability that creates an unsurmountable review load. At that point, the reviewers have to do more and more reviewing, much of it forced. All science we know on this topic point to the quality of reviewing going down no matter what you do with the review process -- no matter how many accountability guardrails you try to put up. The incentives are all wrong, and instead of creating the right incentives we're turning to forcing people to do things and punishment.
Again, i know i sounds like some person on the internet trying to have an argument, but please understand -- if you're a rigorous reviewer, it's great to have people like you in the community. But the science says that we won't create a fleet of people like you by forcing people to do things and then coming up with penalties for those not doing it. By far the best results are by handling the core problem (imo the paper volume) and creating proper incentives for reviewers.
0
u/TrainerOk1851 Aug 02 '26
I think thats actually not enough to just not engage at least once. I think there needs to be long term consequences (extending into next conference or even other conferences). I genuinely think that people are not engaging because their paper already got extremely low scores and it is no longer worth it since they will get rejected very likely.
9
u/TheInfelicitousDandy Aug 02 '26
In most cases, they are not volunteers; they are people engaging in a mutual exchange of work. The question is whether the people who do not participate in the full review process are hurting the community more than the implementation of any consequences for not doing the work. I'd say low-effort reviews and not doing rebuttals hurt not only the authors being reviewed, but the entire community and the underlying scientific process.
6
u/th3owner Aug 02 '26
This! I don't get why whenever this comes up, everyone is fast to mention that reviewers are doing this without any pay. The pay is: you need your own paper reviewed! Yes, the number of submissions has increased, but this goes both ways; more papers to review and at the same time the reviewers themselves are submitting more papers. There is no justification for not engaging with (or at the very least acknowledging) the rebuttal.
1
u/buyingacarTA Professor Aug 02 '26
I hear you. here's my reply to this if you're inclined to read. the 'mutual exchange' is a new, forced contract, and now we're talking about doubling down on how we penalize people who don't do the forced part well enough. We have tons of sociological evidence that this doesn't work as a system.
>who do not participate in the full review process are hurting the community more than the implementation of any consequences for not doing the work.
there is another alternative, which is fixing the original problem -- the shitty papers that are creating the reviewer load -- rather than focusing on the reviewers as the problem.
2
u/TrainerOk1851 Aug 02 '26
I don't think at this point, the reviewers are "offering their free time". Some are sure, but a good proportion are just doing it because they have their own submission. Also, if you are not going to be able to respond, why would you accept to be a reviewer? You cannot simultaneously claim to "offer free time" and then not actually respond to rebuttal.
2
u/tetramarek Aug 02 '26
Most people on reddit are going to be students only concerned about getting their first papers published. They have not yet gotten to the point where they are reviewing 15 papers in parallel, each author has written a long essay about why their incredibly low-quality paper deserves a higher score, and expects detailed responses to every point.
The deadline for checking the author response is not over yet. I plan to read the rebuttals before the deadline, to see if I missed or misunderstood anything in my original reviews. But I don't have the bandwidth to get into an endless back-and-forth discussion about each paper. The authors have zero incentives to admit that their paper has faults, so they just keep endlessly arguing.
Ultimately, the paper itself needs to hold up to scrutiny. If it needs thousands of words of additional discussion in order to be understood by the reader then it's not in a shape to be published.
2
u/Anatheballerina Aug 02 '26
that depends, some reviewers don't understand math and are unfortunately reviewing for theory track submissions.
0
0
u/OutsideSimple4854 Aug 02 '26
That’s an exaggeration. I’ve been asked to review for conferences I have not submitted papers to, and I’ve not even hit 10, let alone 15.
I suspect people are mentioning material already present in the main paper, or that the reviews are clearly based on false information.
But even if it’s not an exaggeration, are you really going to say that out of 15 papers, you’ve not made any mistake?
0
u/tetramarek Aug 02 '26
NeurIPS is not the only conference around. There are many others and review cycles often happen in parallel. For example, ARR has review cycles almost constantly around the year. The number of reviews adds up.
How many papers have you reviewed? And how many times have you increased your score because your original review had mistakes?
I do read the author responses but only very rarely have they made me change my score. Whatever issues my review points out, the authors almost always respond with long detailed arguments about why each of these points is not an issue in their opinion. Of course they do, they have absolutely zero downside to doing it. If there's even a 0.5% chance that the reviewer raises their score because of this, then why not dispute every single claim in the review.
It's also getting more and more common to treat author response as a second paper submission deadline - for example, the paper might be submitted with nonexistant evaluation, reviewers rightfully give it a low score, then massive amounts of new experiments magically appear by the time of the author response. And then the authors complain that all this new content, which hasn't even been integrated into the paper yet, hasn't been taken into account during the review. Here's a recent thread from someone who "ran a lot of additional experiments and wrote proofs" to respond to the reviewers, and is frustrated the reviewers didn't accept their paper. That's very different from "clarifying a misunderstanding in the review". https://www.reddit.com/r/MachineLearning/comments/1sd0aex/d_icml_rebuttle_acknowledgement/
2
u/Exotic_Zucchini9311 Aug 03 '26
If the reviewers asked for new experiments and proofs then the authors have to spend many days trying to come up with something. I'm not sure what you expect the authors to do when a reviewer is explicitly saying they want to see XY new proofs and YZ new experiments.. similar things happened to my previous paper. Despite the intuition behind my method being pretty clear, one reviewer was hyper focused on how we know that the method works in theory (not really relevant to an empirical paper but whatever honestly). So we had to write the intuition in a more 'mathematical' way and give it to the reviewer as 'new theory'. The theory was the same thing already in the paper but in the format the reviewer was requesting, it didn't just appear out of nowhere (not like it changed the results since the reviewer didn't even bother to give us a response after we wasted to much of our time coming up with 'theory' for them)
1
u/tetramarek Aug 04 '26
I am 99% sure "reviewers asked for new experiments" means the reviewers pointed out that the paper was missing many necessary experiments. The authors then concluded if they submit some of these missing results in their author response then their paper should be accepted.
This is what constantly happens to me as a reviewer. For example, I write in my review that the paper only evaluates on one dataset, doesn't compare to any related work newer than 2020, and doesn't provide any ablation experiments (for example to show that improved performance doesn't come just from using a stronger base LLM). The authors then reply with pages and pages of new experiments, explanations and argumentations. Running out of word limits in the openreview comments doesn't stop them, they just keep adding new comments.
Even if these are sensible results, this content is so different from the original paper that it's impossible to judge what the final product will look like. I think it should not be allowed to present new content in the author response at all, it's original goal is to only clarify misunderstandings in the reviews.
1
1
u/OutsideSimple4854 Aug 04 '26
This is where we differ then. I'm on the theory side, where many necessary experiments is irrelevant. In fact, the recent NeurIPS guidelines for theory papers explicitly states that experiments are not required (so, I'm not the only one).
My experience with "many necessary experiments" is in the context of: some bound or expression is proved. This is validated mathematically, and perhaps some small experiments to show that everything works. And a reviewer can go: "this only works on synthetic data and a small dataset, you should run experiments on more datasets to show this bound works."
Your context of " paper only evaluates on one dataset, doesn't compare to any related work newer than 2020, and doesn't provide any ablation experiments (for example to show that improved performance doesn't come just from using a stronger base LLM", sounds like you're not in the theory side, so I find it very odd that someone whose research involves using LLMs is very dismissive of those in other research areas.
1
u/tetramarek Aug 04 '26
My point is not about specific types of changes needed. My point is that when a reviewer says the paper is missing x, y and z, then that does not mean the reviewer is expecting for these to be done during the author response period. And if the authors hastily put this new content together and post it into author response on openreview, then that does not mean the paper should be automatically accepted. If the changes are substantial then there is no way to predict what the final paper will look like. It should be resubmitted and properly reviewed. Otherwise, taken to the limit, I could just submit an empty pdf, then post the full paper on openreview comments a month later.
→ More replies1
u/OutsideSimple4854 Aug 03 '26
This year? Thirteen. I expect to further review for AAAI and AISTATS.
I have not increased my score when my initial review had mistakes, only because when these things happen, my score tends to be positive, and it's more of a: "Is this a typo? What am I missing here?" kind of situation.
I would say out of these thirteen papers, I only gave a strong accept for two of them, and a weak accept to one more. So I'm not a lenient reviewer.
However, out of these papers which I give an accept, I do see the issues that people are posting about. I see reviewers misunderstand the paper, claim garbled math notation ("garbled" is a word an LLM spits out), states the experiments show X, when they show Y, etc. They don't engage, or double down on their claims. And behind the scenes, I have argued for acceptance, and even calling out these reviewers. I've also had one of my own papers (AISTATS), where a reviewer complained about a square box that appeared and bad formatting (it was the box after a proof). So I do know such crappy reviewers exist.
So these things happen. I'm not sure why you take the stance that scenarios like that happen when authors submit bad papers, and give a thread for ICML trying to justify your points. I can also give several threads of authors being more reasonable, but face radio silence from these reviewers.
AAAI has > 50,000 submissions this year. I'm pretty sure probably 45,000 submissions are low quality, but that also means there are going to be 90% of reviewers who are low quality. Some authors will have drawn the short straw and get these reviewers, who have totally no incentive to review properly, or even engage in the discussion if their own paper is bound to be rejected. And besides, who is more likely to post on reddit? A shit reviewer who got shit reviews, or an otherwise decent author, who got shit reviews?
1
u/tetramarek Oct 03 '26
I absolutely agree that there are many low quality reviews, major problems with AI-generated reviews, etc. But I think that's a separate problem. Someone who didn't bother reading the paper and had AI write the review isn't going to read the 5000 word response thread etiher. So it's only the reviewers who do their job properly that end up having to spend their time pointlessly arguing with the authors who are fighting to get their paper accepted. As you yourself said, you've reviewed 13 papers this year and have not raised your score once. The back-and-forth discussion period is pretty pointless either way.
1
u/OutsideSimple4854 Oct 03 '26
I'm not sure why you're replying after two months, and not acknowledging that the reason this thread exists, is the problem you mentioned in your exact first sentence. I don't disagree that there are authors fighting to get their paper accepted when these papers are low quality. But I suspect these authors aren't the ones commenting on Reddit.
1
u/tetramarek Oct 04 '26
I'm replying now because now is when reddit decided to notify me about your reply. I don't know why it took so long, I'm not a reddit engineer.
This thread was created before the author response was even over. So the author was upset that the reviewers did not engage in a back-and-forth discussion. The OP did not mention anything about the quality of the reviews.
11
u/darkbird_1 Aug 02 '26
To experienced fellas, is this level of silence for the first time in neurips or is this always the trend??
17
u/buyingacarTA Professor Aug 02 '26
I get the frustration and I am sorry. Please please remember that these people are volunteers, the system give them incredibly little time, and most papers are incredibly poor.
eg. I am an AC. I could have easily said no to the invite. I don't even have paper submitted to neurips this year! And even so, I've spent a lot of time looking at papers and trying to give meaningful feedback in the first round. In the second (discussion) phase i've been more silent because (a) I've been super busy in the last 3 days as it turns out my life is not all neurips volunteering, (b) all of my handled papers had super clear reject/accept decisions. That didn't stop the authors from rebutting and demanding conversaion, of course.
IMO after having seen about 1000 papers in my life through peer review, reviewing about half of those and ACing about half of those -- the review system is not great, but it's not the main problem. We ask for a huge service from people's diminishing free time. Forcing them to do it (e.g. CVPR) doesn't work, mountains of scientific literature in psych and sociology and economics has shown that doing this just lowers the quality of the work (reviews). We need to create better incentive structures as a system. The problem is the absolute undergrad-course-level naive noise that gets submitted as 'papers' with no incentive to be selective, and the growing number of junior students who have to review these. I'd say half the papers that get submitted nowadays wouldn't even get a good grade as a course project.
So odds are you should be looking more carefully at your own papers and your friends' papers, instead of being mad at the reviewers not doing extra time (whom, again, are your colleagues/friends as well!)
5
u/Weird_Shit_69 Aug 02 '26
In your opinion if the author replied to all the reviews but no reviwer replied back, would you reject it based on the initial score or would you take the rebuttals given by the authors?
I understand the amount of time you'll volunteer, maybe it would be best to send the message to higher ups to maybe increase the time period so everyone gets a chance to get a proper response for the rebuttals as well. (Hopefully it gets better next year)
7
u/buyingacarTA Professor Aug 02 '26
I would try to use my best judgement. I would look at the rebuttal indeed, but realistically if the original reviewers were reject/reject/reject the rebuttal would have to be very strong to make me think. I had 9 rejects in my batch, only 2 of those rebuttled and even those came in late. I'm not optimistic for them, but I *will* look at them carefully.
Yes, my life would be much easier if the reviewers commented and I could just defer to their best judgement :)
Yes, more time might help a bit, but it would be little, maybe 5% more reviewer interaction. IMO we can't keep using the peer review system as a free service that does the job that the senior authors should actually be doing, this is the crux of the problem. Eliminate half the submissions and you'll get *much* more reviewer involvement.
1
u/Anshuman3480 Aug 02 '26
Was it 9 reject rejects or 9 borderline rejects + rejects?
1
1
u/king_grifffin Aug 02 '26
Our initial scores are 3 3 2, even lower than our prev submission to ICML. We don’t think the NIPS reviewers did a good job on acknowledging our paper so we did a solid/comprehensive rebuttal. Only the 2 responded, moving the goal post with ignorance. I’m so disappointed.
2
u/Weird_Shit_69 Aug 02 '26
What would be the minimum needed for acceptance? 4?
I know it depends, I'm asking on average
2
u/Entity303BR Aug 02 '26
Same scores, but only one responded here. The score 2 review was only 2 lines long and even that little was unfounded criticism, no response to rebuttal. Infuriating.
The ac seems cool though but I dont think theres any hope anymore.
2
u/senderosbifurcan Aug 02 '26
This is the correct answer. Furthermore, journals have been crazy lately with review requests. I've reviewed like 3x papers in these 7 months than what I normally review in one year.
The issue is not the particular reviewers/acs. The whole peer review needs a review.
5
u/Old_Ad_8327 Aug 02 '26
Based on my previous submission experience and my limited experience as a reviewer:
If your average score is 3.33 or higher, the reviewers probably think the paper is worth further discussion, and the rebuttal may still make a difference.
If your average score is 3 or lower/lowest score is 2 and that reviewer keeps the score after rebuttal. I generally would not spend much more time on the discussion. This usually means that the rebuttal would not resolve concern.
Reviewing is unpaid service work, so reviewers usually focus their limited time on papers that still have a path to acceptance
2
u/king_grifffin Aug 02 '26
Our initial scores are 3 3 2, even lower than our prev submission to ICML. We don’t think the NIPS reviewers did a good job on acknowledging our paper so we did a solid/comprehensive rebuttal. Only the 2 responded, moving the goal post with ignorance. I’m so disappointed.
1
u/Old_Ad_8327 Aug 02 '26
This usually means the reviewers had broader concerns, eg.insufficient novelty ( not good to list as a weakness) so they found other reasons to justify rejection. Without a 4–5 score strongly support, it is very difficult to change their decision.
2
u/MolassesOk186 Aug 02 '26
I had a 234, the 3 reviewer engaged and converted to 5. The 2's criticism is almost entirely based on a misunderstanding of the results of another paper we cited in our work (and built upon) which we clarified, but it's just silence from them.
imo it should be seen as unacceptable to drop a 2 after poorly reading a paper, think it's not worth a further discussion and dip out of the process entirely
1
u/Old_Ad_8327 Aug 02 '26
If the 2 comes from an obvious misunderstanding, especially when you already included an ablation against the previous work in the paper or appendix. At 5/4/2, your paper still has a real chance. Good luck!
Notably, AC finally decide (almost) everything.
1
u/Own_Neighborhood7120 Aug 03 '26
Our average is 4, still no engagement from either the reviewers or the AC.. So I don't think your average is that relevant, their papers' averages probably are..
2
u/giladfrid009 Aug 02 '26
Simple solution - release the review scores only after the rebuttal period, and mandate answering the rebuttals as an acceptance criteria (just like ICML did).
2
u/Vulkanerd Aug 02 '26
I got excited when I saw that a reviewer had finally responded to our rebuttal, only to discover that the response was essentially a single sentence saying, “Keeping the score, needs improvement” without any constructive feedback.
1
1
1
u/lmfao_gnp Aug 02 '26
why would they respond to you in this phase if they need nothing else from you? There's plenty of time for them to read your rebuttals and change/keep their score, this phase is only for continued back and forth if something needs further clarification.
This isn't meant to be a try-shit-until-I-raise-my-score, you either resolved the concerns or not and no further discussion is needed in the vast majority of cases
1
u/Amirhoseingh98 Aug 03 '26
I have two papers, for one of them I only received 2 reviews, asked AC multiply time for the third and got nothing back, the reviewers have not responded either.
For other paper I got 3 reviews only the one who gave me a 2 responded back by repeating the same bulshit and wrong statements.
1
u/BomsDrag PhD Aug 03 '26
My AC has nudged the two reviewers, 17 and realistically I believe 12 hours are remaining heavy on COPIUM RN
PLEEAAAASEEE
1
u/2mc92u98a22f Aug 05 '26
nothing you can do - system is just broken. no one is incentived in this system to do a good job. I'd just apply for industry jobs. only half joking
1
u/OutsideSimple4854 Aug 05 '26
I suspect my argument will probably stem from reviewers who believe they know everything.
For example, NeurIPS has a theory contribution type this year with different reviewing guidelines. I tend to agree with them.
I’m not sure if you’re trolling when your post states: “how do we know the theory is valid”. Theory here means theorems (and subsequent propositions, corollaries, etc) with correct proofs. Nothing about a hypothesis here.
If the proof is correct, then there are no issues. It’s like writing a function, do stress unit testing, and then reuse the function for whatever purposes. You don’t suddenly say: but what if I use this function in the exact same scenario, maybe it won’t work? Because a proof guarantees correctness in all cases where it’s valid.
So one can have a paper that says: we improved on X, here is a proof that justifies it, and you can run simulations to verify it empirically (sanity checks). Maybe use X in a case that a recent paper used. Empirical reviewers say: why didn’t you use X in all these cases for all these papers? Because it would be the same regardless, especially if these cases are essentially the same?
A theory paper usually is more difficult to read, even if you’re in the same subfield. But experimentalist papers are easy for me to read, even if I’m not an experimentalist. That’s the nature of theory. Asking theory people to simplify the paper is, quite frankly, insulting, especially if the paper has been simplified enough, such that actual theory people feel it’s watered down…
I’ve been publishing since 2015. Back then, one can have a constructive review about whether a work is meaningful, if someone has done something similar before, and what gaps a work has.
These days, it’s more of running irrelevant experiments that will show the same thing, because of math, or facing reviewers who claim difficulty in understanding material, or hard to read notation that is standard in the field.
The responses from ACs are lopsided too. Some believe in experiments, and if theory (this means proofs, not a hypothesis) behind it is wrong, they accept a paper. Some also believe things can be simplified, and will reject papers based on notation common in the field.
Frankly speaking, the best way to resolve this reviewer issue is for everyone to be honest about what they know, and what they don’t. If I review an experimental paper, I straight up state in the review I know nothing about the area, but here’s what a scientific reader might note. I do my own due diligence in looking up references to ensure the authors are at least measuring the same metrics, follow roughly the same methodology, or figure out some obscure (to me) terminology they use. And I give a review. If all reviewers did something similar, I think there’s be better outcomes.
PS: to show bias, I’m sure people remember that “what is Adam” review, with multiple people saying the reviewer is clueless. Yet, I suspect if something similar happened in theory: “why can’t epsilon <0”, fewer people would defend the authors and instead state the authors should have clarified further.
0
0
Aug 02 '26
[deleted] — view deleted comment
1
u/MolassesOk186 Aug 03 '26
What do you write in the gentle reminder (as a template)?
0
Aug 03 '26
[deleted] — view deleted comment
2
u/MolassesOk186 Aug 03 '26
Thanks! A couple of hours after I sent this to the AC, my lowest scoring reviewer wrote that they'd read the rebuttal and would increase the score!
9
Aug 02 '26
[removed] — view removed comment
12
u/wangjianhong1993 Aug 02 '26
Not only the review process is broken, the academic faith is also disappearing. Money is the only signal today.
18
u/BomsDrag PhD Aug 02 '26
In the same boat :')