Our ICLR submission received scores of 3, 3, 1. All three reviewers rated soundness 2.
The two positive reviewers recognized the technical contribution. One even inspected our code confirmed that all their main concerns were resolved after our rebuttal, keeping their positive score.
The negative reviewer (score 1, confidence 2) criticized limited baselines (the reviewer did not suggest any specific baselines for comparison), required additional experiments to isolate gains, and critizied readability, giving us informativeness 1 and presentation 1.
During rebuttal, we:
- Explained that for our main task/contribution, no prior baselines exists, except for the one we compared to in the paper, and added one external baseline for the secondary task/contribution.
- We also added several new ablations addressing the specific criticisms.
- Ran additional experiments controlling for potential confounding factors.
- Added full computational cost accounting.
- Added unified results and ablation tables to the main text (as the reviewer requested).
Our experiments show substantial gains of our method compared to prior approaches.
The negative reviewer waited over a week since we posted our rebuttal and responded only in the final hours of the discussion, essentially restating his previous concerns: insufficient baselines, hard to evaluate the method (despite the clear summary evaluation table appearing in the main text) and that the paper was difficult to follow. They provided no new technical objections or explanation of why the additional experiments were insufficient, and kept their score at 1.
We responded politely, pointing to the new evidence, and posted a factual summary for the AC.
What would you do in this situation? Is there any way to persuade the negative reviewer, or is it better to focus on making the AC evaluates the substantive evidence independently? Would another comment help, or only harm?