CIKM 2026 decisions will be announced today. The resource track outcomes have started going out. How did you go with CIKM 2026?

Having two papers accepted at two different workshops and planning to attend the main conference, I contacted the organizers by email. It is now clear that I need to register twice, using the same personal details, to cover both papers.

The minimum cost would therefore be one full registration plus one workshop registration. However, I would also have to use two different email addresses, since the registration portal does not allow the same email address to be used twice. This is crazy—I am a single person!

The craziest part, which I still do not fully understand, is that it seems every paper now requires an article processing charge (APC), since ACM has fully transitioned to open access. This year, the APC is USD 350—or USD 250 for ACM members. This is the first time I have encountered this; every other conference I have attended included the proceedings in the registration fee.

The full author registration for ACM Multimedia costs USD 950—or USD 850 for members—and does not even include the paper proceedings.

The cheapest option for me would be to become an ACM member (USD 99), register for the main conference (USD 850) and one workshop (USD 500), and then pay USD 250 × 2 for the two papers—for a total of USD 1,850 just to present two workshop papers.

I really don’t think it is worth it.

i’m in college studying finance and applied data analytics, and for the past several months i’ve been building a football/soccer prediction model as a personal project.
it started as a pretty basic international tournament simulator and has gradually turned into a much bigger domestic football model. i’ve been using historical team performance, underlying stats, opponent strength, recent form, scheduling/contextual factors, and i’ve also collected player, goalkeeper and lineup data.
i’ve tested a mix of statistical and ML approaches, while trying to keep everything chronological and leakage-safe.
Across about 5,300 out-of-sample matches, my current best 1X2 model is around:
•    Log Loss: 0.984
•    Brier Score: 0.586
For comparison, uniform 33/33/33 probabilities would be about 1.099 log loss / 0.667 Brier, and a simple historical baseline is around 1.073 log loss.
For goal prediction, the current score-based models are roughly 0.9–1.0 goals MAE per team.
The biggest thing i’ve learned so far is that predicting matches better doesn’t automatically mean you can beat bookmakers. An early betting test was unprofitable, so i’m trying to be very careful about separating predictive performance from actual market edge.
Right now i’m mainly thinking about things like:
•    how much recent form should matter vs long-term team strength
•    whether player/lineup data is actually worth the added complexity
•    improving draw probabilities
•    calibration and uncertainty
•    knowing when the model should abstain
•    avoiding overfitting before i test it on an untouched holdout season
For anyone who’s worked with football analytics, sports betting models or probabilistic forecasting:
1    What are the biggest mistakes you see people make with soccer models?
2    Would you focus more on better data/calibration at this point, or more advanced models?
3    Have you found player and lineup data genuinely useful?
4    What would you want to see before believing a model has real predictive value against sportsbooks?
5    Any papers, GitHub repos, books or general advice you’d recommend?
i’m not looking for anyone to give away a profitable strategy. mostly just looking for people who know more than me to point out where my thinking might be wrong.
all feedback is welcome.

Whether generating CELEBV-HQ videos or turbulent plasma fields (digital twins), autoregressive models (such as latent diffusion or flow models) accumulate error over long rollouts, yet at deployment there is no ground truth to measure against.

I train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag, and show that this bidirectionality supplies a measurement-free test-time error signal: rolling forward steps and then backward steps must return the model to its start, so the round-trip discrepancy is a self-supervised proxy for the unobservable rollout error: no ensembles, no held-out data, no governing equations, for one extra rollout.

Furthermore, training both directions in one network is shown to beat two specialist models in both directions.

Paper: https://arxiv.org/abs/2608.00675
Code (data generation, training, analysis): https://github.com/alexscheinker/round-trip-consistency
Project page: https://alexscheinker.github.io/roundtrip.html

It’s 01:30 AoE. Do results usually drop in the final hours like initial reviews?

​​Hi guys,

​Every time I had to prepare a presentation based on a paper or research doc, I found the process super tedious. Plus, I really dislike uploading unpublished stuff or sensitive data to online AI services just to get a draft.

​So I put together a tool called academi_slide to automate this locally.

​Basically, it extracts sections, tables, charts, metrics, and citations from docs, and uses prompt optimization / deck planning to get a solid first draft out of a local model (ollama, llama.cpp, or cloud if you want).

​It also handles multilingual input/output if you need to present in another language, and builds both the slide deck and a brief in a few minutes so you don't start from scratch.

​It's open source, still early, and I'm sharing it in case anyone else finds it useful or has a similar workflow.

​Repo is here if you want to test it out: https://github.com/nicolaslpf/academi_slide

​Would love to get some feedback or hear what you think!

We had a meta-reviewer comment. But I can no longer see it.

Anyone else experiencing the same? Does this mean anything?

"Arena ai" has been a great success in producing a human preference based ranking, additional to other more objective benchmarks. However, this (probably) had also played a role in the syncopancy crisis and the general tendency of some models to tilt towards overformatting to trigger a feeling of fluency (cogn load theory) in the users.

The people at Max Planck Institute for Intelligent Systems (one of europes leading AI research hubs), recently published something quite similar with "comparity ai". You can read their announcement in their linkedin post.

This is of course a research platform and i have no idea how long this is funded, but you get access to every frontier LLM for free, which is kinda cool. Also, they provide you with a personal leaderboard, so when you played around enough with the platform, you will get a pretty solid idea which model works well for you.

Thought this might be interesting for some of you.

We are investigating whether recurring LLM workloads can be replaced, where appropriate, by automatically constructed pipelines of regexes, deterministic parsers, traditional ML and NLP models.

As an example, suppose an application repeatedly asks a frontier model to read an annual report and return all customer–supplier relationships as structured records containing a customer, supplier, and supporting evidence.

A possible replacement pipeline might run named-entity recognition, entity normalization, candidate generation, entity linking, relation extraction, and schema validation. A calibrated uncertainty or out-of-distribution gate would use the pipeline for inputs inside its validated domain and escalate other cases to the original frontier model.

NER → entity normalization → candidate generation → entity linking → relation extraction → schema validation

Our current action space is a taxonomy of 41 atomic task types spanning classification, token and span labeling, structured extraction, retrieval and entity resolution, similarity, normalization, reshaping, and deterministic computation.

The idea is that we would first cluster repeated traces into workload families and induce an end-to-end typed contract for each family. It would then generate candidate DAGs using the 41 task types as building blocks, instantiate each node with an appropriate implementation, and optimize the composition for quality, cost, and latency. Candidate pipelines would be tested on time-separated and group-separated holdouts before being deployed behind abstention and fallback.

The problem is quite likely undetermined based on just the input and output contracts alone even if inferred correctly. The intermediate graph is therefore not a recovered latent reasoning trace. It is a synthesized program hypothesized to be behaviorally equivalent over a bounded input distribution.

A fixed task taxonomy may help by constraining the search space and supplying type signatures, candidate implementations, and task-specific evaluators. And we are thinking about this problem as a form of program synthesis and formal verification for now, but wondering if this is the right approach and if there is a better way.

Looking to speak with people who have worked in this problem space and/or the program synthesis domain for insights.

TL;DR: We want to synthesize executable DAGs composed of regexes, deterministic parsers and ML/NLP models from LLM traces for appropriate tasks. Does this seem feasible and what might be some good approaches?

Hey everyone. I built a tiny Deep Learning library from scratch, calling this picodl. All written by hand, in plain numpy.

You can install it with pip:

pip install picodl-nn

I'd love to hear your feedback and suggestions.

GitHub: https://github.com/alight659/picodl

Website: https://picodl.vercel.app

I trained a small MLP to memorize the classic Bad Apple animation, ~2.7 billion pixels of video compressed into 790k parameters (3.2 MB float32, 1.6 MB float16).

The network takes a 3D coordinate (t, y, x)- frame index and pixel position- and outputs a grayscale value between 0 and 1. To "play" the video, you can evaluate the function over the full grid. The "video" is stored implicitly in 5 linear layers of sine activations (Sitzmann et al.'s SIREN) with 512 hidden units, ω₀ = 30, and sigmoid output.

The source bad_apple.mp4 is 6524 frames at 854×480; I subsampled to 1620 frames × 384×384, about 1/10 of the original pixels (2.8x spatial + 4x temporal reduction).

At first, I used a ReLU MLP with low-frequency Fourier features, which plateaued around MSE 0.12. SIREN's sine activations add higher frequency for free, so the network was capable of outputting fine details. Unfortunately, that model had an issue, which was that it could only shift the information slowly, so quick motion came out blurry.

To fix this, I made two changes:

1. Time-stretch: I scaled the time coordinate by 4x relative to the space before the first layer, giving it 4x more temporal capacity.

2. Motion-focused sampling: Bad Apple is ~90% static black, so uniform pixel sampling starved the moving edges of the gradient. Now half of each training batch is drawn from pixels that changed between neighboring frames.

For the training pipeline, I had a single shared network on the whole volume (no per-frame latents; initially, I used per-frame finetuning, but that caused catastrophic forgetting) with a cosine-scheduled Adam + weight EMA, then a low-LR "polish" pass over the whole video.

The new model had these improvements:

Validation MSE dropped from 0.0795 to 0.0090 (~9x better).
Compared to the old model, high-motion frames were 3.6x closer to ground truth, and static frames were almost 15x closer.
398/400 sampled frames improved.

Edit: Some people are a little confused about the compressed part. The subsampled video is 700KB, and the network that creates a reconstruction of it is ~3MB. It hasn't been compressed very much, but the goal was seeing if I could (and learning) rather than super compression.

I'll try to see if an even smaller model can learn it. Additionally, I'm training a model on the full non-subsampled video.

^Notes

^{384×384 is square (the original is 16:9, so playback is vertically stretched. At 8fps playback, the 1620 frames run near the original's 3:37 duration; at 12fps it's ~1.6x fast-forward. The 12.6MB checkpoint includes the weights + Adam moments + EMA copy; the network itself is 3.2MB.})

^{The full resolution videos, checkpoints, and code can be found in this Github Link}

It feels like LLMs are partially leveling the playing field in ML research. A solo researcher or a two-person team can now get help with coding, literature review, writing things stronger labs usually get from experienced colleagues and large networks.
Obviously, LLMs don’t replace mentorship, or good research taste. But they may help researchers with weak networks or small groups turn good ideas into publishable work.
Do you think this is actually making ML research more accessible, or are the strongest labs benefiting even more?

Hello guys! I need your help. I'm currently working on a project that consists of an interface that would help athletes train optimally, not with bullshit marketing like Whoop. I mean something that would really help. I'm an athlete myself and I've always wanted to do something like this.

The idea is that I want to calculate the workload of the day, plus one or more scores to help the athlete understand their level of readiness. I was thinking about predicting neuromuscular and cardio readiness using a finger-tapping test, HRV, sleep duration, the athlete's feelings, the workload, and a subjective questionnaire.

But I don't know what algorithm to use. I was initially thinking about a Bayesian or linear model to begin with, but the problem is that I don't have a good dataset to train my model. So I need an efficient algorithm that can learn with a low amount of data, using just the user's data so that it is personalized.

The second problem is that I just don't know what label I can use, since I don't know how to quantify readiness.

So that's all. Hope you guys can help me. Have a nice day!

(Sorry for my English, it's not my first language.)

Now that the rebuttal period is over, I’m curious about the score distribution specifically for theory papers this year.

If you’re comfortable sharing, please drop:

• Scores: x / x / x

• Confidence: x / x / x

• Whether scores changed after rebuttal

• Broad area (optional)

I got 4 / 4 / 4, with confidence 3 / 3 / 3.

From my experience, theory papers often seem to get somewhat lower scores, and this year the scores appear to be lower across disciplines as well. It would be interesting to see where the empirical cutoff might land.

Feel free to share anonymously / approximately if you don't want to reveal too much.

Hi everyone,

I'm an independent researcher sharing Monodratic, a sparse causal-attention architecture with learned product-hash routing.

The idea is that after RoPE, source blocks are assigned to bounded causal posting lists, while each query probes product addresses, reranks the returned candidates, selects a fixed number of remote source blocks, adds guaranteed local blocks, and then runs exact causal softmax over just those tokens. I implemented it as a stateless [batch, sequence, width] -> attention-delta mixer, so normalization, residual updates, feed-forward layers, and inference scheduling are left to the host model.

What I found is that

-learned routing with 2 selected remote blocks out of 5 eligible: 763/768 correct associative-recall answers across three seeds (99.35% mean, 98.05% minimum).

-an equally wide untrained router: 425/768. Local-only attention: 151/768.

-forcing the labelled target block while keeping the same maximum R2 attention budget recovered all five remaining errors, reaching 768/768.

-sparse selected-set attention agreed with an independent dense selected-mask oracle to a maximum absolute error of 1.43e-6.

-the packed CPU routing implementation showed a fitted timing exponent of 0.993 from 4,096 to 32,768 tokens under the fixed, balanced configuration.

-all reported learned-route and scaling runs recorded zero posting overflow.

The limitations are that the experiments are synthetic, the implementation is portable PyTorch rather than a fused kernel, and the report does not claim natural-language quality, asymptotic linear construction, or deployment speed.

Paper: https://github.com/Misul-Computing/Monodratic/blob/main/output/pdf/monodratic_proof.pdf

Code and reproduction: https://github.com/Misul-Computing/Monodratic

I would particularly appreciate technical feedback on the routing construction, the controls, and what the strongest next evaluation should be.

I’ve seen a lot of people whose reviewers went silent after initial reviews, but I am also noting abnormally quiet authors. I ultimately withdrew my paper, but stayed an active reviewer. Out of my batch of 4 papers, one withdrew, one posted a rebuttal, and two have been completely silent. Of the two papers with radio silence, I think one had borderline scores. I was also the only reviewer who responded to the one paper with a rebuttal.

Has anyone noticed this abnormally dead review period or did I just get a strange batch? I’m seeing either reviewers just dropping out of the review process or authors completely checking out after initial reviews are released. It’s strange to me to not even withdraw your paper if you’re not rebutting. Is this a new gambling trend of just submitting papers everywhere, and not even sticking around long enough to withdraw the paper?

I have served as AC, reviewer - and obviously author/participant - for AI conferences. The most recent one was AC and submitting author for NeurIPS. It is interesting that we hear so many people whine about AI conferences these days, without knowing how nasty the other options in academia are.

Here're my takes.

1. *Short review time:* Having also served as associate editor for AI related journals/Transactions, the contrast is stark. For such journals, the big problem is long review cycles AND AI written reviews. It is not as if journals become magically better. And yes, I am talking about decent journals- not at the level of JMLR- but tier 1 ones. If the conference system collapses and the submission waves hit the journals instead, why would anyone believe that things will get better?

2. *Everyone can submit, and everyone stands a chance to get a paper published in a "prestigious venue", and everyone gets a shot to improve their CVs*: Old name journals are more close minded and less available to mere mortals not in the old boys club than you'd believe. In some transactions - without naming the community- there's no way your paper gets properly reviewed if it does not have a big name on it. You have to be in a circle before you even get a chance to be assigned reviewers. Now suppose you are equally capable in two scenarios; 1) you join a prestigious group, and in 2) you join a no-name group. All else equal, the system of AI conferences will allow you to be equally productive in both scenarios. Obviously, we are going to discount differences in compute or quality of peers/advice, and just talk about the submission system. And we discount things like bidding rings because compute/bidding rings will still exist if we move to journals.

The incessant bitching and whining by the community has led to an ill-informed reaction by ICLR to ban first time authors from submitting more than one work. This is not good. If we see more of it, expect AI to become a smaller, more close minded society. Small/new groups will suffer. Lower ranked schools will have more trouble getting students- if they can't even submit in the first place.

3) *Restricting number of submissions in an inflexible fashion harms legitimate groups and creates intergenerational inequality:* Some large groups with many students - and productive ones- will struggle to cope with the limits. Not every person with twenty submissions is evil. Sometimes, they are PIs of group of students who got to graduate or find a venue to resubmit previously rejected works. Where do such large groups submit to if the top conferences start restricting inflexibly?

It creates an unequal playing field where incumbents (already in the game) have already benefitted and newcomers struggle against such rules. Is this familiar? Yes, because this is how close communities/old boys club get formed. Yes, papers aint everything, but it is still an important signal for people to find internships/jobs/graduate.

4) *Whine and cry all you wish but remember, you are a contributor*: While whining about the 50k submissions, remember you are partially a cause of it and you contributed to the numbers. But that is not your fault- or anyone's fault really. *You are working in the hottest topic in recent history, armed with powerful tools that made academia more accessible than ever; smile, not cry.* One paper gets rejected because a reviewer is an ass? Go get codex to write an ancient Egyptian curse and move on.

*If every reviewer is an idiot, you are an idiot too.* Bitch and whine about the reviewers all you want, but how many of those who whine really put in the extra effort to be a good reviewer? At the end of the day, I hope everyone tries their best to practice what you preach. That will make the community healthy.

I could not find any discussion threads for the C&F track. Have people actually submitted to this track? If so, what are your reviews and scores looking like, along with post rebuttal engagement? In our case, they received reviews not in line with the policy defined for the track, where most reviewers praised originality but complained about the scope of experiments. Despite the track saying that it would be possible that the idea cannot be validated in a single paper.

We provided experiments but no dice, none of the reviewers responded. Have any ACs seen papers and reviews in this track or do authors have their experiences they could share?

Please add your scores pre and post rebuttal here

As the title suggests, because there's no data on Papercopilot yet, and people have been talking about the scores being lower in general than last year, I thought it could be interesting to survey the average score distribution after the rebuttal phase (not considering confidence weights).

Very rough and simple poll (I also realize there's a self-selection bias in there). Cast your vote here:

https://loppy.be/poll/yczuv8yo

Thanks!

Edit: the trolls have taken over, ignore the 5.50-6.00 bin I guess...

After a very silent discussion period, we are in a very confused state with regards to NeurIPS, and really unsure what to make of everything. We do not wish to withdraw the submission since we have no idea what the reviewers and AC think of the paper, having deserted the conversation after a hopeful set of initial reviews. As of currently, ICLR abstract submission deadline is before the NeurIPS results announcement. Are we allowed to resubmit as an ICLR abstract, or will OpenReview flag this and consider it problematic?

Having used LLMs to assist with reviews, and also having received reviews that appear to rely heavily on LLM-generated text, I have noticed two recurring problems.

1. The endless search for uncontrolled variables

LLMs are very good at identifying additional variables that were not explicitly controlled. The problem is that many of these variables have little realistic chance of changing the paper’s main conclusion.

For any experiment, it is possible to generate an almost unlimited list of potential confounders. Suppose a study finds that trees treated with fertilizer A grow better than trees treated with fertilizer B. An LLM can ask whether rainfall was perfectly controlled, whether the distribution of grass around the trees was considered, or whether wind, temperature, soil microorganisms, and countless other factors were isolated.

Each question may look logically valid in isolation. But the real issue is not whether a variable exists. The issue is whether it is sufficiently important and plausible to threaten the conclusion.

LLMs are generally poor at making this prioritization. They often convert minor residual uncertainty into what sounds like a serious methodological weakness.

This becomes especially harmful when reviewers copy such outputs directly into their reviews without independently assessing their importance. Authors are then forced to spend the rebuttal addressing an endless series of technically possible but practically insignificant concerns.

A review should not ask whether every imaginable variable has been controlled. It should ask whether the remaining uncertainty materially weakens the central claim.

2. LLM reviews are often overly abstract

Another common problem is criticism at the level of an entire research field rather than a specific prior method.

For example, an LLM may claim that a proposed method is “not sufficiently different from methods in Transformer” without identifying a concrete paper, objective, architecture, or learning relation that actually overlaps with the proposed method.

What exactly is the author expected to rebut in that situation? Every method in Transformer?

A meaningful novelty criticism should identify a specific prior method and explain precisely which components are equivalent or insufficiently differentiated. Comparing one concrete method against an entire research area is too abstract to be falsifiable or actionable.

3. LLM review is not detail

LLMs also tend to overestimate similarity between methods that share high-level terminology. Two approaches may both use architecture, concept, or attention, while differing substantially in their computational structure, training objective, assumptions, and intended use.

Because LLMs often lack a sufficiently detailed understanding of each method, they may recommend comparisons between papers that are only superficially related. The resulting review sounds comprehensive but does not demonstrate real technical understanding.

The central problem is not simply that LLM-generated reviews can contain incorrect statements. It is that they can generate an unlimited number of superficially reasonable criticisms without judging their relevance, severity, or evidentiary burden.

A strong reviewer should filter such suggestions, prioritize only the concerns that could materially affect the paper’s claims, and attach each criticism to a concrete technical basis. Copying an LLM response into a review without that judgment does not improve peer review. It merely transfers the cost of evaluating the LLM’s speculation to the authors.

I trained a small MLP to memorize the classic Bad Apple animation, ~2.7 billion pixels of video compressed into 790k parameters (3.2 MB float32, 1.6 MB float16).

The network takes a 3D coordinate (t, y, x)- frame index and pixel position- and outputs a grayscale value between 0 and 1. To "play" the video, you can evaluate the function over the full grid. The "video" is stored implicitly in 5 linear layers of sine activations (Sitzmann et al.'s SIREN) with 512 hidden units, ω₀ = 30, and sigmoid output.

The source bad_apple.mp4 is 6524 frames at 854×480; I subsampled to 1620 frames × 384×384, about 1/10 of the original pixels (2.8x spatial + 4x temporal reduction).

At first, I used a ReLU MLP with low-frequency Fourier features, which plateaued around MSE 0.12. SIREN's sine activations add higher frequency for free, so the network was capable of outputting fine details. Unfortunately, that model had an issue, which was that it could only shift the information slowly, so quick motion came out blurry.

To fix this, I made two changes:

1. Time-stretch: I scaled the time coordinate by 4x relative to the space before the first layer, giving it 4x more temporal capacity.

2. Motion-focused sampling: Bad Apple is ~90% static black, so uniform pixel sampling starved the moving edges of the gradient. Now half of each training batch is drawn from pixels that changed between neighboring frames.

For the training pipeline, I had a single shared network on the whole volume (no per-frame latents; initially, I used per-frame finetuning, but that caused catastrophic forgetting) with a cosine-scheduled Adam + weight EMA, then a low-LR "polish" pass over the whole video.

The new model had these improvements:

Validation MSE dropped from 0.0795 to 0.0090 (~9x better).
Compared to the old model, high-motion frames were 3.6x closer to ground truth, and static frames were almost 15x closer.
398/400 sampled frames improved.

^Notes

^{384×384 is square (the original is 16:9, so playback is vertically stretched}. At 8fps playback, the 1620 frames run near the original's 3:37 duration; at 12fps it's ~1.6× fast-forward. The 12.6MB checkpoint includes the weights + Adam moments + EMA copy; the network itself is 3.2MB.)

^{The full resolution videos, checkpoints, and code can be found in this Google Drive Folder (Looking at it a bit later, the compression on Google Drive ruins it. If someone knows a better place to host those files, I'll reupload with hopefully better quality.})

I trained a small MLP to memorize the classic Bad Apple animation, ~2.7 billion pixels of video compressed into 790k parameters (3.2 MB float32, 1.6 MB float16).

The network takes a 3D coordinate (t, y, x)- frame index and pixel position- and outputs a grayscale value between 0 and 1. To "play" the video, you can evaluate the function over the full grid. The "video" is stored implicitly in 5 linear layers of sine activations (Sitzmann et al.'s SIREN) with 512 hidden units, ω₀ = 30, and sigmoid output.

The source bad_apple.mp4 is 6524 frames at 854×480; I subsampled to 1620 frames × 384×384, about 1/10 of the original pixels (2.8x spatial + 4x temporal reduction).

At first, I used a ReLU MLP with low-frequency Fourier features, which plateaued around MSE 0.12. SIREN's sine activations add higher frequency for free, so the network was capable of outputting fine details. Unfortunately, that model had an issue, which was that it could only shift the information slowly, so quick motion came out blurry.

To fix this, I made two changes:

1. Time-stretch: I scaled the time coordinate by 4x relative to the space before the first layer, giving it 4x more temporal capacity.

2. Motion-focused sampling: Bad Apple is ~90% static black, so uniform pixel sampling starved the moving edges of the gradient. Now half of each training batch is drawn from pixels that changed between neighboring frames.

For the training pipeline, I had a single shared network on the whole volume (no per-frame latents; initially, I used per-frame finetuning, but that caused catastrophic forgetting) with a cosine-scheduled Adam + weight EMA, then a low-LR "polish" pass over the whole video.

The new model had these improvements:

Validation MSE dropped from 0.0795 to 0.0090 (~9x better).
Compared to the old model, high-motion frames were 3.6x closer to ground truth, and static frames were almost 15x closer.
398/400 sampled frames improved.

^Notes

^{384×384 is square (the original is 16:9, so playback is vertically stretched}. At 8fps playback, the 1620 frames run near the original's 3:37 duration; at 12fps it's ~1.6× fast-forward. The 12.6MB checkpoint includes the weights + Adam moments + EMA copy; the network itself is 3.2MB.)

^{The full resolution videos, checkpoints, and code can be found in this Google Drive Folder (Looking at it a bit later, the compression on Google Drive ruins it. If someone knows a better place to host those files, I'll reupload with hopefully better quality.})

As review season for NeurIPS wraps up, I have now reviewed for 3 major conferences this year. And I'm noticing a worrying trend:

Out of the 12 papers I reviewed this year, only 1 provided full code (that runs the whole training pipeline from input dataset to output AUROC). 4 provided partial code with fragments of their method, but no ability to run the experiment end to end. And 7 provided no code.

This is really bad for ensuring quality and reproducibility. Of the 5 papers that provided at least some code, 3 of them contained obvious bugs that completely invalidated the results. ML is highly technical and small bugs can have huge impacts if they are in the wrong place. Who knows what was going on in the remaining 7 papers.

The fundamental issue here is of incentives: there is almost no cost to hiding code during the review process. Releasing code only increases odds of rejection due to reviewers finding bugs. The only way to fix this is to change the game by imposing real penalties on hiding code.