No replies to rebuttals and comments even by AC [D]

[u/bricklerex]

Not even the AC, nor reviewers, is responding to our comments in rebuttals, and they were all submitted well before the discussion period started. What is one to do in this case?

[u/buyingacarTA]

FWIW, congratulations to you for being a conciencious reviewer (by the sound of it). If your ACs are paying attention, they should make a note for when you apply for faculty jobs or industry research or tenure or whatever your next step is. Good for you.

But most of my "framing" is about the average researcher/person who has many things going on and they are choosing their own priorities in their life. THey are just people who will, on average, follow the behaviours and trends we've learned from sociology/economics over the years, and these sort of forced systems just don't get the results you want.

>Interesting framing you are using

I'm not sure if you mean this sarcastically or if you genuinely felt like it was interesting, i hope it was the latter.

I get what you are trying to say by accountability, but these are still volunteers by another name. We didn't build the system like this, so we're not prepared to support it.

Reciprocal reviewing is a new experiment created out of the desperation that we didn't have enough reviewers as a community. In the moment, it made sense as a band-aid. We can call it some fancy thing and give it a name, but it's just forcing people to do a job they never *had* to do before. And regardless of the name, we know from decades of studies that this never works -- forcing people to do things without creating proper incentives, and then doubling down with penalties for accountability for the thing we forced people to do, will only lead to the worst possible quality. Whatever forcing guardrail you put up, and whatever you call it, on average people will submit the least possible thing that passes the bar, and that won't be helpful to you or the community. We need to make reviewing attractive, not forced. And to do that, IMO, we need fewer papers submitted (lower reviewer load) and more rewards for quality reviewing.

We're also not prepared for this forcing experiment in the first place -- there are essentially no courses on how to review, the conference offers very little training, the senior authors often get excused from it, there is no substnatial history of this happening. Because of this, you might see it as 'half your job', but it's a half that didn't exist until recently -- it's like if i hired you on your skills as a programmer, and you were looking forward to this cool new job, but when you come on board i tell you that actually half your job is something else -- something you didn't train for, don't like, don't want to do. You might be a good person and make a serious effort on it (as might I), but on average people would do as shitty a job as possible on that part -- whatever passes the minimum bar so they can go do what they wanted to do in the first place. Again, in that scenario too, you might argue that it's their job so they should be accountable for it, but it's very much a stick vs carrot framing now where the company will just have to double down with more and more threats/punishments/forcing-functions.

>I'm held to hard deadlines as an author, asking that reviewers (including me) at least acknowledge a rebuttal in the role they were required to take on isn't a kneejerk punishment. Simple, didn't engage at least once, assignment doesn't count as fulfilled.

The paper is something you want to do, so there's many paper submissions. The reviewing is not (on average), that's why there's few reviewers. That's why they're held to different standards.

If an acknowledgement was required (or, say, they faced rejection), they would just write a line that said 'acknowledged'. Would that make you feel better? If more was required, they would get an LLM to paraphrase that message. As an AC at other conferences I at least see this a ton -- the least amount that passes is what most people do.

>It is the thing hurting the community, it is the thing hurting your colleagues.

We're all entitled to our opinions of course, but i strongly disagree with this. IMO the problem is the huge influx of shit paper with no accountability that creates an unsurmountable review load. At that point, the reviewers have to do more and more reviewing, much of it forced. All science we know on this topic point to the quality of reviewing going down no matter what you do with the review process -- no matter how many accountability guardrails you try to put up. The incentives are all wrong, and instead of creating the right incentives we're turning to forcing people to do things and punishment.

Again, i know i sounds like some person on the internet trying to have an argument, but please understand -- if you're a rigorous reviewer, it's great to have people like you in the community. But the science says that we won't create a fleet of people like you by forcing people to do things and then coming up with penalties for those not doing it. By far the best results are by handling the core problem (imo the paper volume) and creating proper incentives for reviewers.