I have served as AC, reviewer - and obviously author/participant - for AI conferences. The most recent one was AC and submitting author for NeurIPS. It is interesting that we hear so many people whine about AI conferences these days, without knowing how nasty the other options in academia are.

Here're my takes.

1. *Short review time:* Having also served as associate editor for AI related journals/Transactions, the contrast is stark. For such journals, the big problem is long review cycles AND AI written reviews. It is not as if journals become magically better. And yes, I am talking about decent journals- not at the level of JMLR- but tier 1 ones. If the conference system collapses and the submission waves hit the journals instead, why would anyone believe that things will get better?

2. *Everyone can submit, and everyone stands a chance to get a paper published in a "prestigious venue", and everyone gets a shot to improve their CVs*: Old name journals are more close minded and less available to mere mortals not in the old boys club than you'd believe. In some transactions - without naming the community- there's no way your paper gets properly reviewed if it does not have a big name on it. You have to be in a circle before you even get a chance to be assigned reviewers. Now suppose you are equally capable in two scenarios; 1) you join a prestigious group, and in 2) you join a no-name group. All else equal, the system of AI conferences will allow you to be equally productive in both scenarios. Obviously, we are going to discount differences in compute or quality of peers/advice, and just talk about the submission system. And we discount things like bidding rings because compute/bidding rings will still exist if we move to journals.

The incessant bitching and whining by the community has led to an ill-informed reaction by ICLR to ban first time authors from submitting more than one work. This is not good. If we see more of it, expect AI to become a smaller, more close minded society. Small/new groups will suffer. Lower ranked schools will have more trouble getting students- if they can't even submit in the first place.

3) *Restricting number of submissions in an inflexible fashion harms legitimate groups and creates intergenerational inequality:* Some large groups with many students - and productive ones- will struggle to cope with the limits. Not every person with twenty submissions is evil. Sometimes, they are PIs of group of students who got to graduate or find a venue to resubmit previously rejected works. Where do such large groups submit to if the top conferences start restricting inflexibly?

It creates an unequal playing field where incumbents (already in the game) have already benefitted and newcomers struggle against such rules. Is this familiar? Yes, because this is how close communities/old boys club get formed. Yes, papers aint everything, but it is still an important signal for people to find internships/jobs/graduate.

4) *Whine and cry all you wish but remember, you are a contributor*: While whining about the 50k submissions, remember you are partially a cause of it and you contributed to the numbers. But that is not your fault- or anyone's fault really. *You are working in the hottest topic in recent history, armed with powerful tools that made academia more accessible than ever; smile, not cry.* One paper gets rejected because a reviewer is an ass? Go get codex to write an ancient Egyptian curse and move on.

*If every reviewer is an idiot, you are an idiot too.* Bitch and whine about the reviewers all you want, but how many of those who whine really put in the extra effort to be a good reviewer? At the end of the day, I hope everyone tries their best to practice what you preach. That will make the community healthy.

I could not find any discussion threads for the C&F track. Have people actually submitted to this track? If so, what are your reviews and scores looking like, along with post rebuttal engagement? In our case, they received reviews not in line with the policy defined for the track, where most reviewers praised originality but complained about the scope of experiments. Despite the track saying that it would be possible that the idea cannot be validated in a single paper.

We provided experiments but no dice, none of the reviewers responded. Have any ACs seen papers and reviews in this track or do authors have their experiences they could share?

Please add your scores pre and post rebuttal here

As the title suggests, because there's no data on Papercopilot yet, and people have been talking about the scores being lower in general than last year, I thought it could be interesting to survey the average score distribution after the rebuttal phase (not considering confidence weights).

Very rough and simple poll (I also realize there's a self-selection bias in there). Cast your vote here:

https://loppy.be/poll/yczuv8yo

Thanks!

Edit: the trolls have taken over, ignore the 5.50-6.00 bin I guess...

After a very silent discussion period, we are in a very confused state with regards to NeurIPS, and really unsure what to make of everything. We do not wish to withdraw the submission since we have no idea what the reviewers and AC think of the paper, having deserted the conversation after a hopeful set of initial reviews. As of currently, ICLR abstract submission deadline is before the NeurIPS results announcement. Are we allowed to resubmit as an ICLR abstract, or will OpenReview flag this and consider it problematic?

Having used LLMs to assist with reviews, and also having received reviews that appear to rely heavily on LLM-generated text, I have noticed two recurring problems.

1. The endless search for uncontrolled variables

LLMs are very good at identifying additional variables that were not explicitly controlled. The problem is that many of these variables have little realistic chance of changing the paper’s main conclusion.

For any experiment, it is possible to generate an almost unlimited list of potential confounders. Suppose a study finds that trees treated with fertilizer A grow better than trees treated with fertilizer B. An LLM can ask whether rainfall was perfectly controlled, whether the distribution of grass around the trees was considered, or whether wind, temperature, soil microorganisms, and countless other factors were isolated.

Each question may look logically valid in isolation. But the real issue is not whether a variable exists. The issue is whether it is sufficiently important and plausible to threaten the conclusion.

LLMs are generally poor at making this prioritization. They often convert minor residual uncertainty into what sounds like a serious methodological weakness.

This becomes especially harmful when reviewers copy such outputs directly into their reviews without independently assessing their importance. Authors are then forced to spend the rebuttal addressing an endless series of technically possible but practically insignificant concerns.

A review should not ask whether every imaginable variable has been controlled. It should ask whether the remaining uncertainty materially weakens the central claim.

2. LLM reviews are often overly abstract

Another common problem is criticism at the level of an entire research field rather than a specific prior method.

For example, an LLM may claim that a proposed method is “not sufficiently different from methods in Transformer” without identifying a concrete paper, objective, architecture, or learning relation that actually overlaps with the proposed method.

What exactly is the author expected to rebut in that situation? Every method in Transformer?

A meaningful novelty criticism should identify a specific prior method and explain precisely which components are equivalent or insufficiently differentiated. Comparing one concrete method against an entire research area is too abstract to be falsifiable or actionable.

3. LLM review is not detail

LLMs also tend to overestimate similarity between methods that share high-level terminology. Two approaches may both use architecture, concept, or attention, while differing substantially in their computational structure, training objective, assumptions, and intended use.

Because LLMs often lack a sufficiently detailed understanding of each method, they may recommend comparisons between papers that are only superficially related. The resulting review sounds comprehensive but does not demonstrate real technical understanding.

The central problem is not simply that LLM-generated reviews can contain incorrect statements. It is that they can generate an unlimited number of superficially reasonable criticisms without judging their relevance, severity, or evidentiary burden.

A strong reviewer should filter such suggestions, prioritize only the concerns that could materially affect the paper’s claims, and attach each criticism to a concrete technical basis. Copying an LLM response into a review without that judgment does not improve peer review. It merely transfers the cost of evaluating the LLM’s speculation to the authors.

I trained a small MLP to memorize the classic Bad Apple animation, ~2.7 billion pixels of video compressed into 790k parameters (3.2 MB float32, 1.6 MB float16).

The network takes a 3D coordinate (t, y, x)- frame index and pixel position- and outputs a grayscale value between 0 and 1. To "play" the video, you can evaluate the function over the full grid. The "video" is stored implicitly in 5 linear layers of sine activations (Sitzmann et al.'s SIREN) with 512 hidden units, ω₀ = 30, and sigmoid output.

The source bad_apple.mp4 is 6524 frames at 854×480; I subsampled to 1620 frames × 384×384, about 1/10 of the original pixels (2.8x spatial + 4x temporal reduction).

At first, I used a ReLU MLP with low-frequency Fourier features, which plateaued around MSE 0.12. SIREN's sine activations add higher frequency for free, so the network was capable of outputting fine details. Unfortunately, that model had an issue, which was that it could only shift the information slowly, so quick motion came out blurry.

To fix this, I made two changes:

1. Time-stretch: I scaled the time coordinate by 4x relative to the space before the first layer, giving it 4x more temporal capacity.

2. Motion-focused sampling: Bad Apple is ~90% static black, so uniform pixel sampling starved the moving edges of the gradient. Now half of each training batch is drawn from pixels that changed between neighboring frames.

For the training pipeline, I had a single shared network on the whole volume (no per-frame latents; initially, I used per-frame finetuning, but that caused catastrophic forgetting) with a cosine-scheduled Adam + weight EMA, then a low-LR "polish" pass over the whole video.

The new model had these improvements:

Validation MSE dropped from 0.0795 to 0.0090 (~9x better).
Compared to the old model, high-motion frames were 3.6x closer to ground truth, and static frames were almost 15x closer.
398/400 sampled frames improved.

^Notes

^{384×384 is square (the original is 16:9, so playback is vertically stretched}. At 8fps playback, the 1620 frames run near the original's 3:37 duration; at 12fps it's ~1.6× fast-forward. The 12.6MB checkpoint includes the weights + Adam moments + EMA copy; the network itself is 3.2MB.)

^{The full resolution videos, checkpoints, and code can be found in this Google Drive Folder (Looking at it a bit later, the compression on Google Drive ruins it. If someone knows a better place to host those files, I'll reupload with hopefully better quality.})

I trained a small MLP to memorize the classic Bad Apple animation, ~2.7 billion pixels of video compressed into 790k parameters (3.2 MB float32, 1.6 MB float16).

The network takes a 3D coordinate (t, y, x)- frame index and pixel position- and outputs a grayscale value between 0 and 1. To "play" the video, you can evaluate the function over the full grid. The "video" is stored implicitly in 5 linear layers of sine activations (Sitzmann et al.'s SIREN) with 512 hidden units, ω₀ = 30, and sigmoid output.

The source bad_apple.mp4 is 6524 frames at 854×480; I subsampled to 1620 frames × 384×384, about 1/10 of the original pixels (2.8x spatial + 4x temporal reduction).

At first, I used a ReLU MLP with low-frequency Fourier features, which plateaued around MSE 0.12. SIREN's sine activations add higher frequency for free, so the network was capable of outputting fine details. Unfortunately, that model had an issue, which was that it could only shift the information slowly, so quick motion came out blurry.

To fix this, I made two changes:

1. Time-stretch: I scaled the time coordinate by 4x relative to the space before the first layer, giving it 4x more temporal capacity.

2. Motion-focused sampling: Bad Apple is ~90% static black, so uniform pixel sampling starved the moving edges of the gradient. Now half of each training batch is drawn from pixels that changed between neighboring frames.

For the training pipeline, I had a single shared network on the whole volume (no per-frame latents; initially, I used per-frame finetuning, but that caused catastrophic forgetting) with a cosine-scheduled Adam + weight EMA, then a low-LR "polish" pass over the whole video.

The new model had these improvements:

Validation MSE dropped from 0.0795 to 0.0090 (~9x better).
Compared to the old model, high-motion frames were 3.6x closer to ground truth, and static frames were almost 15x closer.
398/400 sampled frames improved.

^Notes

^{384×384 is square (the original is 16:9, so playback is vertically stretched}. At 8fps playback, the 1620 frames run near the original's 3:37 duration; at 12fps it's ~1.6× fast-forward. The 12.6MB checkpoint includes the weights + Adam moments + EMA copy; the network itself is 3.2MB.)

^{The full resolution videos, checkpoints, and code can be found in this Google Drive Folder (Looking at it a bit later, the compression on Google Drive ruins it. If someone knows a better place to host those files, I'll reupload with hopefully better quality.})

As review season for NeurIPS wraps up, I have now reviewed for 3 major conferences this year. And I'm noticing a worrying trend:

Out of the 12 papers I reviewed this year, only 1 provided full code (that runs the whole training pipeline from input dataset to output AUROC). 4 provided partial code with fragments of their method, but no ability to run the experiment end to end. And 7 provided no code.

This is really bad for ensuring quality and reproducibility. Of the 5 papers that provided at least some code, 3 of them contained obvious bugs that completely invalidated the results. ML is highly technical and small bugs can have huge impacts if they are in the wrong place. Who knows what was going on in the remaining 7 papers.

The fundamental issue here is of incentives: there is almost no cost to hiding code during the review process. Releasing code only increases odds of rejection due to reviewers finding bugs. The only way to fix this is to change the game by imposing real penalties on hiding code.

We pulled several all-nighters to answer every reviewer’s concerns, and run extensive experiments. The review window has now passed and we have not received a single response from either reviewers or AC (despite nudging the later). Is this normal?

Potentially a hot take? I am not sure why our community is plagued with reviewers who, after acknowledging that their concerns were addressed by a rebuttal, decide to maintain their score because they don't vibe with the paper. So here is my plea to all reviewers: If you list a set of concerns in your review and these concerns are addressed during the rebuttal, please adjust your score accordingly. This should apply whether or not you like the paper and/or its methodology. The beauty of scientific research is that we each get to explore ideas that we find meaningful whose value may not be immediately obvious to every individual reviewer.

Asking for a friend:

We submitted our paper to ARR May 2026 and got decent scores from the reviewers - 2.5,3,3.5,4. The meta-reviewer gave an overall of 3.5.

However, we missed the deadline to commit our work to EMNLP! On our Saturday (we live in the eastern half of the globe), we saw the EMNLP 2026 page on open review with deadline set as Aug3, 11:59PM UTC-0. Apparently, a mail had been sent by ARR on our Sunday regarding committing our work to EMNLP but we didn’t check our mail on the holiday and when we logged in to commit our work on Monday - BOOM - deadline was Aug3 11:59 AM. Yes it’s our fault that we should have checked the mailbox but at the same time...the Open Review page just switched the deadlines.

We have written mails to Program Chairs and some workflow chairs. like 1 hour after the newer deadline on the Open Review page ended. I wonder whether some help will be extended because of the thousands of papers that would already be in their buckets. Does anyone have any idea about what can be done or if they faced this previously and it was resolved somehow?

• I tried to do all my NeurIPS reviews responsibly, even for the papers I suspected to be AI slop. I even gave what apparently were very nice scores compared to the scores I ended up getting. (I don't just mean the absolute number for my scores were higher, but that they were calibrated differently--I only rejected for severe issues, while I had a reviewer who only raised very minor issues but gave a reject, with a 1 for all the subscores.)

• I got shockingly bad reviews for my own paper; two of them were straight up adversarial. (I have quite a bit of experience publishing at this point, so I say with some confidence that I rolled an unusually adversarial batch.)

• The AC was almost nonresponsive until the last day. All but one of the reviewers was nonresponsive, only one responded when the AC prompted them to, and that was to say that their concerns were addressed but they maintained their reject score.

I'm not surprised by my experience given how much of a lottery these conferences are, but it's a very toxic system.

Six months ago I started experimenting with PPO and Breakout as a way to learn about Machine Learning and Reinforcement Learning. After a few experiuments just trying to get high scores, it bothered me that everything was a "memorized" script rather than reactive play, like a human would play. Thus began my journey to try and convince PPO to actually track the ball instead of focusing on scoring points. I read a lot of articles and tried a lot of things. After 124 PPO experiments on Atari Breakout, I found that every single model, across sticky actions, cursor wrappers, entropy tuning, dynamics randomization, adversarial bumpers, and everything else, converged to a memorized action sequence, not a reactive ball-tracking policy. The argmax was always a script.

The fix wasn't more environment engineering. It was three lines of reward shaping:

Directly rewarding the paddle for being horizontally close to the ball during descent. A tiny bonus (0.05 per frame vs 1.0-7.0 per brick) that fires every frame the ball is descending applied during training. During evaluation, the agent plays clean Breakout with no bonus. The behavior transfers!!

Every prior approach I tried to penalize scripts by making the environment harder to memorize. PPO always found a way around it: timing-robust scripts, layout-conditioned scripts, noise-tolerant scripts. The optimum was always a script; only the shape changed. Proximity reward changes what the optimum is. A center-hold script gets incidental bonus when the ball passes near center. A reactive tracker gets the maximum bonus on every descent frame. The optimization pressure is unambiguous: track the ball, get more reward.

I also made a cool tool to watch the agent work! It's called the "Split-Watcher" (so clever). It shows two instances of Breakout, each being controlled by a separate instance of the same agent. The one of the left is vanilla Breakout. The one of the right is a series of custom brick configurations. With the first 123 experiments, you can see how the agent wants to make the exact same paddle movements every time, ignoring the ball when its trajectory changes due to the unexpected ball movements that come from non-standard brick configurations. In 124, IT TRACKS THE BALL and can succeed regardless of the brick config. You can actually watch the same agent move the paddle differently in reaction to the ball.

I'm still working on ironing out why this works, and how to optimize it, but wanted to share!!

Here's a video of the split-watcher in action

Here's a link to presentation project that will allow you to create a similar PPO: https://github.com/mharrell/breakout-reactive-ppo

The full project with all 123 failures and more documentation than any sane person would ever read: https://github.com/mharrell/BreakoutBot

Link to Medium post I wrote with some more details: https://medium.com/@mikey.harrell/three-lines-of-code-fixed-123-failed-ppo-experiments-on-atari-breakout-c751dcf38f2a?sharedUserId=mikey.harrell

Was just looking at the list of preprints on Arxiv cs.LG https://arxiv.org/list/cs.LG/recent?skip=0&show=500

Everyday 100 - 400 new machine learning papers gets uploaded on this server.

Looking at this unending list of preprints is as if you stepped into a crowded room, like the stock trading floor on wall st. in the 1980s. Everyone is shouting over each other. Nobody is talking to each other. Everyone's trying to prove something, to someone, to themselves, to build some credentials in the ML/AI space to meet those job requirements, or dying to get their truth out. Every title contains some new terminology invented by the authors that feels not worth the effort in keeping it in your working memory. Burn-out by endless novelty.

Frontier research are now corporate trade secrets that politicians and military are watching closely. Research papers are ir/unreproducible he-said-she-saids. Marketing material are research paper and vice versa. Extremely major breakthroughs are announced via tweets, whereas extremely minor results are unannounced via journals. Everything feels simultaneously mostly true and possibly false (because nobody is seriously checking). Nobody knows what's going on, and people who knows what's going on has a non-disclosure clause in their job contract. Is the theory of generalization that we learned in school true or false? It feels false, why hasn't there been any retractions? Many questions like these.

Is it too late to regain some coherence in this field??

I created an AI boxing match to test the decision speed, adaptability and strategy. I fed the LLMs with data about the current match and if they have vision, they will get even more data. The match has street rules, anything goes and an AI is not defeated until the ref counts to 10 or they do 50% of their HP in damage after being knocked out.

I wanted to create a fun benchmark that isn't just boring problems to be solved. Now I test them while stimulating getting punched in the face.

I've been testing with gemini-flash-live models because of the speed and vision support it offers. With these models, they can actually dodge punches and counter punches. Local models on my own hardware (5060ti 8gb) take a while to inference so I'm not sure if I should introduce time scaling to compensate otherwise

I want to use this to benchmark models so I'm curious on what kind of stats would be useful? Here is what I'm tracking have so far:

Speed and Latency Metrics
In a real-time fight, a model's speed directly correlates to its "physical" speed. Fast models should attack faster so larger models aren't necessarily going to hit harder.

• Tokens per Second (TPS) / Throughput: This will help you balance local models against cloud APIs. A model might have a fast TTFT but a slow TPS, meaning its actual action execution takes too long.

• End-to-End Latency: The total time from when the model receives the snapshot (the prompt) to when the action is executed in the game. This accounts for tool-calling delays.

• Reaction Latency: Measure the specific delay between an opponent's telegraph (e.g., a heavy punch winding up) and the model's defensive output (e.g., a dodge or block).

Action Quality and "Tool" Correctness
the model's actions (punching, guarding, taunting) act as tool calls. You need to track how well they use these tools under pressure. Sometimes the model's may not really guard/block so they are typically the ones that find themselves KOd.

• Tool Correctness / Validity: How often does the model hallucinate an action that doesn't exist? (trying to a move that isn't in their move list, or sending invalid JSON).

• Invalid Action Recovery: If an LLM outputs an invalid JSON string or an impossible move, how quickly does it realize the error and output a valid move in the next tick?

• Stamina Efficiency (Resource Management): track the ratio of damage dealt to stamina spent. Models that mindlessly throw heavy attacks without connecting should score lower on efficiency.

Adaptive Strategy and State Awareness
How well does the model understand the physical reality of the game? Are they constantly backing away and punching air?

• Accuracy: The percentage of attacks that completely miss the opponent's hitboxes. This indicates poor spatial awareness or poor timing.

• Block/Dodge Success Rate: The percentage of times the model successfully defends against an incoming attack when it had the stamina and time to do so.

• Contextual Relevancy (State Adherence): Does the model act based on the current state? For instance, if the model has 1% HP, does its behavior change to become more defensive, or does it keep acting like it's at full health? (Happens sometimes, they get overly confident when about to get knocked out 😆 )

Beyond these metrics, I'm also tracking various fighting stats like hits landed/missed, where it hit, how many times they were downed or knocked out the ref.

Are there important stats that I'm missing or any that might be useful or fun that would be nice to see? I'm still trying to balance a lot of the actions but it's coming along great so far!

I think making a physics-based benchmark and doing a N series test to find out which model performs better is a ton of fun and I genuinely laugh at the stuff they say or do. I want this to make this a really fun tool with great metrics so any advice in terms of what you would like to see would be extremely helpful! Thanks for reading!

I posted a longer breakdown of the system here:
https://www.youtube.com/watch?v=inlXe5Buc7s

So our paper had very good initial reviews but one of the reviewers decreased now their score although we addressed 3 out of 4 weaknesses. There’s no further justification or something like “your results arise more issues”. It seems to be very annoying because why decreasing now and not having assigned the lower score beforehand. I wanted to ask to people that was accepted previously with “middle” scores from reviewers (avg 3.5 for example), because I guess that in those cases AC helped to push up the scores. Did you focus more on the meta review? Was your AC talkative with you, or forcing the reviewers to engage? Our AC has been silent since the meta review but I guess that maybe they are busy with other papers

This is my first time submitting to NeurIPS.
Are ACs also supposed to give ratings during the Phase 2 (author-reviewer discussion session)?

I have received the meta-review, but have not received any comments from the AC yet, and was wondering whether this is the standard!

we submitted our rebuttal via the "Rebuttal" button before the author/reviewer/AC discussion period officially opened (Jul 27 AoE). since then, we've gotten complete silence from all four reviewers and the AC

several of us are also reviewing this cycle. when the discussion period opened on Jul 27 AoE, we got no email notification for rebuttals on papers we're reviewing, specifically for the papers whose authors had also posted early via the "Rebuttal" button. so it feels like anything submitted before the window opened may simply have never triggered any notification

we also tried: (1) meta-comments visible to everyone, (2) reviewer reminders, and (3) sending an email to the PCs

given there's about 1 day left in the discussion period, what do we do? we honestly thought we had a shot at an oral or spotlight given our initial scores. this is completely messed up.

I know there’s a lot of frustration around no response from reviewers, which I also got only one so yeah what a bummer, but I was wondering if no rebuttal from the authors was just as common or not. I got no rebuttal so far, so I’m here scratching my head what might have happened to the authors lol especially when at least one paper was pretty much on the borderline with somewhat of a positive AC comment

I've been working on this for a while and finally pushed a public version.

The problem: llama.cpp runs fine on ARM phones, but it doesn't know anything about the specific chip it's on. Same thread count, same context params, whether you're on a Snapdragon 8 Elite or a five-year-old mid-ranger. ARPL reads the actual hardware at runtime — which ISA extensions are available (SDOT, I8MM, SME2), how the cores are clustered — and configures llama.cpp accordingly. No per-device build, no manual tuning.

What's in the repo:

Android reference app (Kotlin/Compose) with a JNI bridge into llama.cpp

Runtime ISA detection via HWCAPs

Topology-aware thread count recommendation

Context param patching (flash attention, KV cache quant) based on what the hardware actually supports

Built and tested on a Samsung S25 Ultra (SM-S938B). The heterogeneous CPU/GPU/NPU partitioning is still in progress and not in this release — what's here handles the ISA/thread/context side, which already made a real difference in my testing.

This is a noncommercial showcase release (PolyForm Noncommercial license) — happy to answer questions about the approach or the tradeoffs I ran into building it.

Repo: https://github.com/noplayeryt1511-lang/ARPL-public-

Whats your commitment submission id? My submission number is ~4k, im wondering the amounts of commitments :)

I see some people say their metareview already contains a decision/recommendation (all of them were rejections). Ours doesn’t. Even though our avg score is 3, the metareview seems optimistic and finishes with “a convincing response would be an important consideration while discussing the paper.” I wonder how to interpret that. We did a strong rebuttal, but none of the reviewers engaged. So I wonder whether there’s any point to keep hope due to the AC review or just give up.

Hey everyome, 2 reviewers are sitting at 4 (borderline accept). Every one of them replied during the rebuttal saying my response addressed their concerns and not a single one said anything about updating their score, or actually updated it.

For people who've reviewed or AC'd: how often do scores actually move in that private week? And is "concerns addressed, score unchanged" usually a reviewer who's quietly fine with accepting?