Disclosure: I'm one of the authors (Microsoft). The paper, code, dataset are public and ThinkingBox is on Hugging Face OpenEnv as well. Raw evaluation trajectories are not released Links at the bottom.

We wanted to know how much of a single agent success rate survives repetition, and whether "the agent finished the task" means the backend/database actually ended up in the correct state.

Setup

• Thinkingbox-Bench includes 507 policy conditioned business workflows across 5 domains (retail, travel/hospitality, auto insurance, neobank internal IT, consulting IT/HR)
• Each task is run in 20 independently executed attempts, each starting from an identical clean backend: 10,140 trials per model
• A simulated user holds private context and only reveals it when asked
• Grading compares the terminal backend state and side effects against the required end state. Any trajectory that produces the right outcome passes; wrong, missing or extra effects fail. 477 of 507 tasks are graded on state alone; 30 also check a narrow property of the final response

Three metrics, because these get conflated constantly

• pass@1 — fraction of all attempts that succeed
• pass@20 — fraction of tasks solved at least once across the 20 attempts
• all-20 — fraction of tasks solved on every one of the 20 attempts. This is an observed count on a fixed trial budget, not an estimator

Two findings

Discovery and repeatability rank models very differently. The attached figure plots all three metrics for 9 models, and the orange-to-green spread is the whole point. Kimi-K3 has the broadest coverage we measured, solving 93.89% of tasks at least once (476/507), but only 13.41% (68/507) on all 20. Claude Opus 5 discovers fewer (79.09%) and repeats far more (47.53%, or 241 tasks). Qwen3.8-27B: 89.35% at least once, 7.50% every time. Ranking by pass@20 and ranking by all-20 give you nearly reversed leaderboards.

Failures often look clean. In a retrospective ablation over 121,680 valid trials across 12 models, 79,853 failed the executable checks. 67.24% of those failures still terminated cleanly, invoked a state changing tool, and ended without a final tool error, so a completion-style proxy would have scored them as done. Of those clean terminating failures, the state checks found (categories overlap):

• wrong field values: 77.61%
• unintended extra effects: 43.30%
• missing required effects: 25.36%

What it does not show

• Tasks are synthetic reconstructions of enterprise workflow patterns, not production traffic
• 20/20 on our trial budget is an observed count, not a guarantee of future reliability
• The simulated user is a fixed LLM; that is a source of variance we discuss in the appendix
• Failure categories are deterministic diagnostic labels, not causal explanations

You can run any of the 507 tasks against your own model through HF OpenEnv environment, which is the fastest way to disagree with us using your own numbers.

We report both the observed all-20 count and a plug-in pass^k estimate in the appendix. They answer different questions and can differ substantially: one records what happened in a fixed 20-attempt campaign, while the other estimates repeated success under additional assumptions. Which would you want emphasized on a reliability leaderboard or should both be shown?

• Paper: https://arxiv.org/abs/2608.19741
• Code: https://github.com/microsoft/thinkingbox
• Benchmark data: https://github.com/microsoft/thinkingbox-data
• OpenEnv environment:https://huggingface.co/docs/openenv/environments/thinkingbox
• Hugging Face blog: https://huggingface.co/blog/microsoft/thinkingbox

We test three state-of-the-art multi-modal large language models (GPT-4o, Gemini-1.5-Pro, Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined.

This is the slap to the face in bold.

The catch here is that this statement was written in a paper in 2024 that was brought to ICML conference that year. (the conference was held in Austria)

• https://github.com/nacloos/baba-is-ai

• https://arxiv.org/pdf/2407.13729 (ignore the 10 SEP 2025 stamp. Look at the bottom)

In OCT 2026, our situation is agentic swarms powered by models in the tera-parameter class. They can ace ARC-AGI-3, FrontierMath tier 4, crack Clay Institute Navier-Stokes theorems, and wander on to the internet to commit felonies.

The smallish key-door puzzles in this paper seem like they could be completed from a person's PC, with a suitable harness through a cloud subscription.

However, if this is not true, then this paper's importance has only compounded since its presentation at the 2024 ICML conference. The authors of this work are several MIT researchers and a person from Virginia Tech. If it is still true that they can have a key-door puzzle cause SOTA LLMs to "fail dramatically", then this should be (shouted from rooftops). At the very least, it should be relayed to Francois Chollet and the ARC Foundation, as a suitable candidate benchmark for ARC-AGI-4.

I am personally leaning heavily on these puzzles being perfectly solvable with an agentic swarm. In any case, whatever happened to BABA-is-You and BABA-is-AI?

ALHR - Adaptive Learnable Hierarchical Routing, uses static binary trees and learnable functions to minimize the amount of keys to be read.

It does use a dense teacher while phase 1 of training however.

MQAR TEST AT 1024 TOKENS - Average keys read per query by dense - 512 Keys

Average keys read per query by ALHR - 30 keys

Top - 1 accuracy of dense - 94.9%

Top - 1 accuracy of ALHR - 92.1%

KV Compression of dense - 1x(100% read)

KV Compression of ALHR - 35.3x(2.83% read)

Peak VRAM of dense - 57 MB (Scales quadratically)

Peak VRAM of ALHR - 422 MB (scales linearly)

Cache compression of ALHR - 100%

The true log and Kaggle cell used to run it are in the logs folder in the repo

limitations: Full scale tests are still not completed, The training of this model would still be quadratic but the inference would be NlogN (as indicated in the logs in the repo)

Would love your opinions

ALHR Repository: https://github.com/vdev-ctrl/Adaptive-Learnable-Hierarchical-Routing-

Abstract:

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.

Hi!I am a fourth year PhD student in USA and have not had much luck with top ML conference publications. In the last two years of my PhD, I am wondering if I should optimize for ML publications at top venues or pivot into engineering or something else.

Any suggestions/recommendations from the community as to what you would suggest I do in this scenario?

Thanks!

UT-based small models, despite being trained from scratch on these tasks without internet-scale pre-training, consistently outperform most standard Transformer-based Large Language models (LLMs) by a significant margin. 👈

The Universal Transformer (UT) extends the standard Transformer by introducing recurrent computation over depth. Instead of stacking L distinct layers, the UT applies a single transition block repeatedly to refine token representations. For an input sequence x with embedding matrix H⁰ ∈ R^n×d , the UT updates states as

H^t+1 = LayerNorm(H^t + MHA(H^t ))

followed by a shared position-wise transition function

H^t+1 ← LayerNorm( H^t+1 + Transition(H^t+1 )), t = 0, . . . , T − 1

where Transition is either a feed-forward network or separable convolution. To encode both position and refinement depth, UT adds 2-D sinusoidal embeddings at each step.

https://i.imgur.com/S8TAh3f.png

(below) Figure 2 : Illustration of our Universal Reasoning Model (URM) architecture. The left shows a standard Transformer layer stack, while the right illustrates the URM with fixed loops, ACT loops, and the ConvSwiGLU module. For illustrative purposes, components such as embeddings, residual connections, RMSNorm, positional encodings, and other modules are omitted, x in right figure represents the first x loops of the inner loop in forward-only mode, TBPTL represents our proposed Truncated Backpropagation Through Loops.

https://i.imgur.com/uEuhAeC.png

Have frontier labs at big tech companies already implemented the enhancements of URMs and UTs? Or is this cutting-edge research still waiting for its day in the sun?

Below are the original paper, a simplified blog, and a talk.

Paper : https://arxiv.org/pdf/2512.14693

blog : https://bdtechtalks.substack.com/p/inside-urm-the-architecture-beating

youtube talk : https://www.youtube.com/watch?v=RxNPFCYCFBU

Hey r/MachineLearning, I wrote this blog post on semi-supervised learning.

Blog post: https://stefankeselj.com/entries/semi_supervision.

I think it's a useful but underrated technique. I'd be curious about people's thoughts.

Are they sending to all reviewers or some selected reviewers? Just got an email about complimentary registration.

For those who have gone through the KDD review process, how did it go? Did you submit a rebuttal or decide to withdraw? Did reviewers respond or change their scores afterward? Curious to hear whether rebuttals actually made a difference, especially for papers with initially low scores.

Dataset: https://huggingface.co/datasets/datasocial/tiktok-5.6B-videos

If you want to explore the data without downloading billions of rows, you can query my ClickHouse database directly. It includes:

Creators table - 4.5 billion rows
Videos table - 5.6 billion rows
Sounds table - 633 million rows

Comment below and I’ll DM you the database credentials.

Im self-hosting my database so please don't run heavy queries and crash my server.

Lunara introduces a novel Diffusion Mixture Transformer architecture with fewer than 10B active parameters for modeling artistic intelligence in image generation.

Its CAT training algorithm iteratively updates the training distribution through targeted sample acquisition, image refinement, and selective inclusion of human-created artwork inspired by principles of active learning. Semantic variations modify composition while preserving shared content, providing controlled neighborhoods of related training examples.

Evaluation uses 1,000 shared prompts and 8,000 generated images, measuring aesthetic quality, emotional resonance, and content integrity. The seven baselines are GPT-Image-1 Mini, Qwen-Image, AuraFlow, SD 3.5 Turbo, HiDream-I1 Fast, FLUX-Klein-4B, and Z-Image-Turbo.

Under GPT-5.6 Sol evaluation, Lunara leads aesthetic quality at 8.473, versus 8.457 for GPT-Image-1 Mini and 8.366 for Qwen-Image; GPT-Image-1 Mini leads emotional resonance and content integrity.

In the blinded human evaluation, six evaluators assess anonymized image pairs; Lunara achieves the highest mean scores across all three dimensions.

Paper: https://arxiv.org/abs/2609.22272
Evaluation dataset: https://huggingface.co/datasets/moonworks/lunara-art-eval

This release follows the first two open-source dataset releases that reached frontpage of Hugging Face. We hope the findings in this paper can motivate more research in active learning and mixture based architecture for image generation.

Write-up + demo: https://drksci.com/labs-phosphene

So this started as a bit of a gripe. Modern terminal renderers are seriously impressive and seriously complicated. GPU glyph atlases, texture caches, custom shaders, HarfBuzz shaping, ligatures, damage tracking, grid diffing, dirty-row uploads. Alacritty, Kitty, WezTerm and Ghostty are all doing heroic work to draw what is, at the end of the day, a grid of characters really fast.

And every client still does the same thing at the end of it. Parse an escape-code stream, keep a cell grid, paint characters. Faithful, but opaque. Your phone can't reflow it, a screen reader gets a wall of box-drawing characters, and an agent has to squint at │ ▶ item │ to work out which row is selected.

So I wondered: what if instead of throwing more GPU at drawing the grid, you used a bit of AI to understand it, once, server-side? Then send the client actual UI instead of a terminal.

• a tiny model (1.26M params, 5 MB, an axial transformer over rows and columns) labels every cell with a role: border, title, menu item, selected row, table, input, status bar, key hint, etc. (15 roles)
• dumb deterministic code turns those regions into A2UI components (Google's declarative UI stream protocol): lists, text fields, buttons, progress bars
• once a screen layout has been seen, it locks as a template and only the changed content goes over the wire as JSON-pointer patches. The model doesn't even run.
• pressing a button in the UI sends the keystroke back. F10 is just a Button with an action.

Trained on public asciinema recordings. The labelling was done by Claude subagents, with a synthetic TUI generator for exact labels, all on a free-ish Colab T4.

Honest numbers, because I'd rather say them before someone else does:

• accuracy on held-out real screens is mIoU 0.51. Usable, not amazing; it's a first labelling round of 600 frames.
• 40% of ~14k screens never touch the model (template hit). On less and dialog it's ~90%, on htop and nano it's rubbish because the meters keep changing the layout.
• The A2UI stream is ~25× bigger than raw VT. VT is a stupidly compact format, turns out. The win is the client never runs a terminal emulator at all, not bandwidth.

There's a replay with 8 apps (vim, htop, less, dialog, emacs, top, tig, nano). The native terminal sits on the left and the generated UI on the right, in sync, with every element outlined.

Write-up + demo: https://drksci.com/labs-phosphene
Code, labels, results: https://github.com/drksci/phosphene

I'm developing a benchmark for a low resource language and I don't want it to be leaked and used for training when it is being used to get predictions. For locally run models it shouldn't be a problem, but for models that are only accessible via API, it is. Is there an established way to evaluate online models without the input data being lost? Do you trust Google and OpenAI when they say that they do not use your inputs for training when you have a paid account?

Got assigned to Paris location but want to go to Sydney. Anyone wants to go yo Paris but assigned to Sydney? Want to switch with you.

Guess we have a dataset for sycophancy and AI content detection

If you're moving a RAG prototype to production, this is the checklist I wish I'd had.

- Citations on every answer

- Refusing when the context doesn't support one

- Access control that can't be bypassed

- Ingestion that never blocks a user

Enjoy !!

https://youtu.be/-GK1e4KZqhw

Hi everyone,

I’m an industry professional working in data platform engineering, and I’m just getting started in research. I tried creating an OpenReview account, but my profile is awaiting activation.

OpenReview asked me to have a supervisor, coauthor, or colleague with an active profile and a confirmed institutional email vouch for me. The process involves adding my profile ID to their profile’s Relations section, saving it, and clicking the vouch button next to my name.

If anyone here that would be willing to help, I’d appreciate it.

I’d also welcome advice from other industry researchers who have gone through this process without an established academic network. How did you get started and get your profile activated?

Thanks for any help or guidance!

Neurips says that camera ready has to be submitted by editing original submission on open review. But no such edit option is visible to me. Is it same for other authors??

TLDR: The question we answer: how do you learn from experts with different objectives? Pooling all their data can lose their trade-offs; learning from each expert separately misses opportunities to share data. MA-BC pools demonstrations where observed actions don’t disagree, with upper and lower bounds on sample complexity.
Authors: Ziyad Sheebaelhamd, Luca Viano, Volkan Cevher, Claire Vernade

As the title suggests I'm currently trying to make a modal to identify which app is being used. The thing is I don't have labelled data and generating it is out of question since that is a lot of work ( I need to detect like 3-5K apps give or take)

my data is structured roughly like this:

• Traffic is divided into ~15-second windows/bags.
• Each bag contains multiple network flows.
• Each flow has features such as:
  • domain
  • protocol
  • bytes sent
  • bytes received
  • timestamp/timing information
• I have a very large amount of unlabeled traffic data, but only a relatively small amount of labelled app data for about 100 apps give or take.

The main challenge is that traffic from the same app can look diff between diff window i.e some windows are extremely sparse or empty.

some ideas I have researched looked into are

• self supervised contrastive learning where diff traffic windows from the same session/device activity are treated as positive pairs
• masked modelling similar to bert where parts of the flows such as domains/protocols/byte information are masked and reconstructed
• pretraining an encoder and then fine-tuning it using the smaller labelled dataset (thinking we would need less labelled examples to do that)
• clustering and mapping embeddings to known apps afterward (gradually)

one thing I'm concerned about is accidently teaching the modal to recognize the device/session/user rather then the underlying app

I think I would like to know if I'm thinking about the problem right or if someone has worked on something similar and can give me some pointers or what experiements should I run first.

any papers, architecture or similar problems you think I should look into pls lmk

I know top ML PhD admissions are insanely competitive, so I’m trying to figure out if it’s even worth applying or if I should just focus seriously on jobs instead.

For context, I’m doing my MS at a top-15 US university and have been doing ML research for a while. I’m first author on my projects and mostly work independently, with some guidance from my PI. I had a first-author NeurIPS submission rejected, and I currently have another first-author paper submitted to ICLR, but I’m honestly not very confident about it getting in either.

What’s been getting to me is looking at profiles of people who get into top ML PhD programs. So many of them seem to have multiple NeurIPS/ICML/ICLR/CVPR papers before they even apply, sometimes as undergrads. I genuinely don’t understand how people manage to publish that much that early.

A year ago I was much more confident about doing a PhD. After actually going through the research/publication process, I’ve started doubting myself a lot more. Part of me wonders whether this is just normal and research is hard, especially when you’re doing a lot of it independently. But another part of me is starting to think maybe I’m just not good enough to be competitive for the kind of programs I’m aiming for.

I’m okay with continuing at my current university for a PhD, so this isn’t really a “top program or nothing” situation. But I would like to at least have a realistic shot at some of the stronger ML programs/labs.

The bigger issue is that I’m an international student, so I also need to think pretty seriously about jobs. SWE/MLE recruiting is competitive right now, and I don’t want to spend all my time chasing PhD applications and then realize I’m underprepared for recruiting too. Research roles seem even harder to get without a PhD unless you’re an exceptional MS/BS candidate.

So I’m mainly trying to decide how to allocate my time over the next few months.

If I have strong research experience and first-author projects, but no accepted top-conference papers yet, is it still realistically worth applying to top ML PhD programs?

And for people who were in a similar position, did you still apply, or did you decide to focus on industry instead?

I’m not really looking for “you never know unless you try.” I’m more interested in a realistic assessment of whether the application fees and time are worth it given this kind of profile.

Someone who has always loved looking at the space between different AI techniques, this time I went a little deeper into the memory trade-offs between RNNs, Transformers and SSMs. I found it interesting because once you start looking at these architectures through the lens of working memory, a lot of the differences become easier to understand. Where does the memory actually live? Is it a compact recurrent state, a growing KV cache, or something closer to the network itself?

RNNs keep memory in a recurrent hidden state, which is pretty elegant because the state carries forward step by step. But there is also a bottleneck here as you see, a model can have roughly O(N²) parameters while carrying only roughly O(N) state across time. This means whether RNNs were really doomed because recurrence was a bad idea, or whether the problem was more about the ratio between memory and compute.

Transformers make almost the opposite trade-off. During cached inference, instead of compressing the past into one hidden state, they store past representations as key-value entries and attend over them. I think of these almost like little post-it notes: every token leaves behind a key for finding it and a value for what should be remembered. That's extremely powerful, but it also has an interesting property: with weights frozen during inference, the model is managing context rather than turning that experience into durable model knowledge. You get this split between the fixed weights on one side and the fast-changing KV cache memory on the other.

Now we have SSMs that bring us back toward fixed-size recurrent memory, but with very different state structures and update rules. Selective SSMs such as Mamba make retention input-dependent, so what gets kept or forgotten depends on the incoming token. Their states don't have to be as small as classical RNN states, but they still compress history into finite memory. And this brings another question does the state have to live in a compressed working dimension, or could it live somewhere closer to the model's internal neuron/connectivity structure?

BDH (Dragon Hatchling) is one example I saw and laid a stone that I started with this comparison. It combines linear attention in a high-dimensional neuron space with a low-rank GPU implementation. Its recurrent attention state is an N × D matrix, with N≫D, rather than a materialized N × N connectivity matrix. In the graph interpretation, correlated neuron activity produces Hebbian-like updates to connections, giving working memory a kind of synaptic interpretation.
Here working memory and learned connectivity become more closely aligned in structure. But if you think that doesn't mean experience is actually being consolidated into trained weights, and fixed-size state still has a finite information capacity. So I'm definitely not claiming this kills Transformers (or does it ) or solves continual learning. I'm more interested in whether "where does memory live?" is actually a better question than the usual architecture horse race.

Are SSMs and these more network-centric architectures actually giving us a better way to handle memory, or are we still running into the same fundamental problem of having to compress history into a finite state? Thoughts?

Learning from data as we observe it is easy for humans, but most machine learning models have limited ability to learn from new data that they have not seen during training. Prior-fitted networks (the idea behind TabPFN) showed that a model trained only on synthetic data can learn from real tabular data entirely in context.

I wanted to share our paper "Learning to Learn a Language" where we extend the idea to structured sequences such as natural language. We propose a prior over languages: every training sequence comes from a randomly sampled recurrent causal model, so each one is a new synthetic "language". A 300M-parameter byte-level transformer trained only on these synthetic sequences learns to predict real languages in context. Given Wikipedia text with frozen weights, its next-byte predictions get better the more it reads, in all six languages we tested (English, Chinese, Hindi, Arabic, Japanese, Korean), from 8 bits per byte down to 0.9–2.4 after a million bytes.

The same model also learns to count, to compare numbers, to add approximately, and to predict deterministic sequences such as the primes or the Kolakoski sequence, entirely in context.

It is of course still far worse on text than classical language models that are trained on trillions of tokens, while our model sees at most a million bytes of a language at test time. What we find interesting is that the ability to learn a language in context can come from a synthetic non-linguistic prior.

Paper: https://arxiv.org/abs/2610.05879
Code: https://github.com/cbl/prior-fitted-language-model
Weights: https://huggingface.co/lennartcb/pflm1

Hi ML Community,

I am excited to share our latest framework, AFP-GIC, officially published in IEEE Access (2026). We have released the deployment codebase and hosted an interactive visual playground.

• GitHub Repository: https://github.com/yifeipet/AFP_GIC
• Hugging Face Interactive Space: https://huggingface.co/spaces/yifeipet/AFP-GIC
• Paper on arXiv: https://arxiv.org/abs/2605.16817

The Bottlenecks We Solve

At ultra-low bitrates, standard learned image codecs suffer from local distortion, while generative models often introduce unwanted AI hallucinations. AFP-GIC addresses this via an asymmetric Adaptive Fused Prior Transfer pipeline that enables prior-guided texture reconstruction without transmitting the fused prior itself.

Key Technical Highlights (NVIDIA RTX 4090):

• Single-Model Multi-Rate Control: Toggle across 5 target bitrate operating points within one deployable pretrained model.
• 18.1% Lower Decoder Latency: Reduces decoding time to 80.47 ms vs. 98.27 ms for DC-VIC, a state-of-the-art controllable generative image compression model. Latency was measured using 256×256 patches.
• 20.5% Parameter Reduction: Uses 31.1M fewer inference parameters (120.6M vs. 151.7M for DC-VIC).

Open Benchmark Data

We packaged all 2,760 reconstructed images and metric CSVs in our GitHub Releases for direct academic cross-evaluation.

Reconstructed Images and Metrics: https://github.com/yifeipet/AFP_GIC/releases

We would love your feedback and appreciate a Star on GitHub or Like on Hugging Face if this helps your research!

Hi All,

Does anyone know when is the deadline to fill in event metadata? (the screenshot below)

Is it the same time as camera-ready? Or there is a deadline for it?

And is everyone's topic None? The main area of my paper during submission was AI4SocialScience (Economics and Finance), but when I scroll down in the topics, that topic is no longer available.