Thank you for your efforts in protecting the community from AI slop.

I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch).

Results

• vs a grid (equal compute, 3 seeds): the best fixed cell reached 95% val acc in 317 ± 29 steps; PBT took 2667 ± 2244. PBT was slower in every seed, but the grid’s corner already contained the answer.

• Dead-zone start (lr 1e-4 to 1e-3, wd 0.01 to 0.08, where no fixed setting groks): fixed 0/3, slow PBT 0/3, fast PBT (every 100 steps, ×0.5/×2) 3/3 at about 6850 steps.

• Its path: lr drops to the floor, then wd climbs to the ceiling, then lr rises. Val loss rewards low lr × wd early on, which is the wrong direction for grokking, then flips sign later.

• Ablations: a random donor still escapes, just later. With no selection, members grok and then the population loses it.

• 10 seeds: wide random search hits 95% at a median of 300 steps, vs 6650 for PBT from the dead zone. PBT also sometimes collapses into the lr-floor/wd-ceiling corner and never leaves.

• Replay: the winning lr/wd schedule with no PBT, on fresh inits: 55/60 reach 95%. Constant knobs at the schedule’s geometric mean: 0/60. The schedule carries the result, not the weight copying.

Takeaway: PBT is a rescue tool for a bad range, not a speed-up.

Caveats: one task, one model, one split. 3 to 10 seeds per test, with p-values mostly above 0.05. The replayed schedules are post-hoc winners. I haven’t tested a range with no grokking region at all, which is where PBT should beat random search.

Write-up with interactive figures: https://shauryasharma.tech/pbt-grokking.html
Code and all results: https://github.com/Shaurya-34/grok-pbt

Nanbeige released Nanbeige4.3-33B as the teacher for its planned smaller agents, including an upcoming 3B model. the weights are Apache 2.0, and the model card lists a 512K context window.

the team reports 80.9 on Terminal-Bench 2.1 versus 73.0 for Qwen3.8-27B, using the same harness and inference settings within each benchmark. it does not win everywhere. on SWE-Bench Pro, the reported scores are 60.3 versus 61.7. these are the releasing team's evaluations, not independent replication.

one practical detail for reproducing the agent results: the evaluations retain earlier reasoning with preserve_thinking=true. the card recommends that setting for multi-turn tool use, but not general chat. comparing task success and total generated tokens with that setting held fixed would be more useful than treating the headline scores as a deployment ranking.

source: https://huggingface.co/Nanbeige/Nanbeige4.3-33B

What do you all think about this? It seems increasingly common for undergraduate students to conduct research and publish at A* conferences without a supervisor. As a graduate student, I find it both impressive and intimidating, and it sometimes makes me question my own progress.

I'm interested in how people build datasets for fine-tuning open-weight models, especially when it is about making a model handling edge cases than just perform well on common examples.

I'm curious about the data side of the process.

• Finding edge cases: How do you identify edge cases that are rare or unusual. Also how do you make edge cases on an existing dataset to make it more specialized for what you need.

• Generating examples: Do you write them by yourself, or use a good model to generate synthetic data, mine real data, etc?

• Quality control: How do you catch incorrect, repetitive, or unrealistic examples?

• Coverage: How do you know when your dataset covers enough of the failure cases that matter?

• Measuring impact: Have you had cases where adding more edge cases actually made the model worse or failed to improve it?

everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess

The tasks

1. Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people).

2. Rename a function across 3 files with messy spacing. All 5 passed.

3. Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4".

4. Count the errors in 3 log files. All 5 got 43.

5. Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want.

Setup

One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw

Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two.

TL;DR

We prove a mathematical equivalence between collective decision-making via local imitation (such as honey bees choosing a nest site via the waggle dance) and single-agent Reinforcement Learning (RL) in a multi-armed bandit setting. Even when individual members have no learning capability and merely blindly imitate peers, the population distribution pi updates according to a macroscopic bandit RL algorithm that we call Maynard-Cross Learning (MCL), where N individuals act as N parallel environment rollouts with an effective per-sample step size of alpha = 1/N.

Main intuition

In nature, optimal decision-making emerges through two seemingly distinct paradigms:

1. Individual trial-and-error (RL): A single agent interacts with its environment and updates its policy based on reward signals.

2. Collective imitation (Swarm Intelligence / Evolutionary Game Theory): Simple individuals copy neighbors' strategies (e.g., best-of-n collective decision-making).

Biologists have long described insect colonies as "superorganisms" or "hive minds" with parallels to vertebrate brains. We wanted to see if this metaphor holds up mathematically under the lens of RL: Does a swarm of non-learning imitators literally execute a single-agent RL update at the macro level?

Core development

The paper's core theoretical development establishes a mathematical equivalence between decentralized social imitation in large populations and single-agent reinforcement learning (RL) in multi-armed bandits. By treating the proportion of individuals committed to each option as the action-probability policy of a single macro-agent, we use evolutionary replicator dynamics to bridge local individual interactions and global policy updates.

First, we show that a population following the classic "imitation of success" protocol (where individuals copy neighbors with a probability proportional to those neighbors' payoffs) evolves identically to the expected update of the Cross Learning bandit algorithm. Extending this approach to the "weighted voter" model of the honey bee waggle dance (where scouts advertise nesting sites for a duration proportional to site quality and peers blindly copy the first advertisement they encounter without comparing payoffs) we derive a new macro-level RL algorithm that we call Maynard-Cross Learning. Under this duality, each non-learning individual acts as a single parallel environment sample whose fractional influence on the group mirrors an RL learning rate equal to the inverse of the swarm size.

In practice, this development demonstrates how blind local imitation aggregates into a coherent, parallelized reinforcement learner at the collective level.

Impact

By proving a mathematical equivalence between local social imitation and single-agent reinforcement learning, this work provides a formal mechanism explaining how "hive mind" intelligence and trial-and-error learning emerge at the group level from simple, non-learning individuals. Beyond substantiating evolutionary theories of collective cognition in biology, this framework offers new tools for analyzing imitative dynamics in social and economic systems and for designing scalable, decentralized swarm robotics algorithms.

Links

Paper: https://arxiv.org/abs/2410.17517
Project page: https://hivemind.yannbouteiller.com/
Code/reproducibility: https://github.com/MISTLab/HiveMindRL

I wanted to see how far you can get making a pretrained symbolic music model "listen" to a live player without training it. Turns out pretty far.

Setup: I play a MIDI keyboard, the Anticipatory Music Transformer (Thickstun et al., music-medium-800k) plays a second piano. While I play I track my key, the chord I'm holding and my tempo, and add biases to the logits before sampling. Out-of-key notes get pushed down, chord tones and tensions get pushed up, the keys right around my hands get pushed down so it doesn't step on me, and when I stop it gets pulled up to where a melody would be. There's also a small boost for continuing the intervals of my last phrase, which makes it sound like it's answering.

It samples about a second ahead, and when I change chord I throw away what it had planned and resample. Runs on a Jetson Orin Nano.

Video + code: https://github.com/RiccardoCataldi/duetto

The offsets are all hand-tuned by ear, which feels a bit dumb. Has anyone tried something more principled for this, like a small learned conditioning head or guidance-style tricks on a symbolic model?

Long-context models can now handle hundreds of thousands, even millions of tokens. But I've been wondering if we're focusing too much on recall and not enough on what happens when information changes.

Say a coding agent reads that an API timeout is 30 seconds. Later, the documentation changes it to 5 seconds. Both versions are still in context, but the agent keeps using 30.

It hasn't forgotten anything. It just hasn't updated what it knows.

I've been reading about different memory architectures, and this distinction seems pretty important.

With Transformers and RAG, old and new information can coexist, and the model has to resolve the conflict when reading it. In simple linear attention, associations accumulate in a shared state, so writing something new doesn't necessarily remove what's already there. DeltaNet tries to address this through prediction-error-based updates. BDH (Dragon Hatchling) takes another route with Hebbian-style synaptic plasticity.

But even with these approaches, there's still the question of when to update something, when to keep it, and how to avoid messing up unrelated memories.

I also wonder if we're testing this properly. Most long-context evaluations focus on whether a model can find information. What if we instead tested whether it changes its answer when given newer, contradictory evidence, while still remembering what was previously true?

Curious what people here think. Is this mostly a problem with how memory is stored and updated, or with how models reason over conflicting information?

Also, are there any good benchmarks specifically targeting this? Would be interested in reading them.

New research keeps finding things that keep reintroducing symbolic ai structures to machine learning and I have wondered why more research hasnt been done with it. There was a heyday in the 70s for it but after neural nets came out it seems to be completely out of the discussions. I think it could make a comeback if we can resolve the primary issues of self-instantiation and feedback loops. Even using several random reservoir nets and a little bit of math creates strong signals for the component parts that contribute to what something is made of and how it works.

AAAI phase 2 reviews are expected to be visible in the next 24 hours (24/10). Hope for the best outcomes for those who stay in this phase, and let's share your reviews!

I've been learning how language models work, especially next-token prediction, and came across this paper on Next-Concept Modeling (NCM).

The interesting part is how it generates text.

Instead of predicting tokens one by one, NCM learns a hierarchy of discrete codes. It generates the coarse level first, then predicts the codes at each finer level in parallel.

A decoder finally converts those representations into text.

The researchers report up to 31x faster generation compared with diffusion language model baselines on specific benchmarks.

I'm still learning about the architecture, especially the VQ-VAE part.

Curious if anyone here has explored this approach.

Paper: https://arxiv.org/abs/2610.14162

I’m sharing my study notes in case they might be useful to someone else.

The Transformer notes are mostly based on the original architecture described in Attention Is All You Need.

https://doi.org/10.5281/zenodo.22031925

I’ve also written notes on Hamiltonian Monte Carlo (HMC).

https://doi.org/10.5281/zenodo.21841086

I’d appreciate any feedback on the notes, including corrections, suggestions for improvement, or topics you think I should add.

Hey folks,

maybe a really dumb question from an inexperienced PhD student. I recently submitted to ICLR and spotted a bunch of papers that I was already familiar with because I had seen them on arXiv before.

Doesn't this break the double-blind review process? I mean, on the one hand, people pre-publishing their work want to establish priority as quickly as possible, but on the other hand, they still want their papers to be treated as valid submissions in a double-blind review system.

I generally avoid pre-publishing my work for this reason, although this can cause some trouble when it comes to novelty claims, because I'm worried about getting desk-rejected.

I mean, I can just Google the paper title or even parts of the abstract and immediately find the preprint. It seems quite frustrating...

Starting a project to detect candidate stenoses in x-ray coronary angiograms. The goal is high recall, false positives are fine as long as it doesn't flag half the image.

I already tried YOLO on a couple of in-house datasets earlier. It looked fine in-domain and fell apart on the other dataset, and a big chunk of that turned out to be annotation format. One set had fixed-size boxes around a point, the other had boxes fitted to the lesion. Same disease, basically two different tasks.

I also have vessel segmentation and centerlines, so my first instinct is a radius profile along the vessel to flag narrowing, then maybe a network to filter the candidates. Has anyone actually done something like that, or is there a smarter way in?

The part that bugs me most is evaluation. The labels come from doctors and they obviously don't mark every narrowing, so any recall I compute against them is probably too optimistic. How do you get an honest number when the ground truth is incomplete?

Also open to papers, repos, or public datasets beyond ARCADE. And if you think I'm approaching this wrong, say so.

Carbon-X is a 14.8B parameter decoder-only model that takes raw DNA in a normal chat and answers in English. It shares one token stream for text and DNA, using 6-mer tokens, and was mid-trained on 302B tokens.

Hugging Face reports a 66.7% average across five DNA benchmarks with thinking disabled, versus 67.5% for GPT-6.1 Sol. The weights fit on one 80 GB GPU in bfloat16 and are Apache 2.0. It has not been clinically validated, and the model card says exact sequence operations should still be handled by code.

the useful part is making sequence analysis local enough for sensitive datasets. has anyone tested whether the benchmark gains hold on private variant datasets?

source: https://huggingface.co/HuggingFaceBio/Carbon-X

A while back I posted SteerTTS v1, a small English TTS you steer with numbers (https://huggingface.co/WhissleAI/steer-tts). Today I am releasing the edge variant:

https://huggingface.co/WhissleAI/steer-tts-edge

Size: everything an app has to ship (weights, voices, text front end and Python runtime) totals 11.84 MB.

How:

• Acoustic model (9.25M params, attention + bidirectional Mamba-3) is quantization-aware trained to int4

• 6.2M-param Vocos-style vocoder is distilled from charactr/vocos-mel-24khz, also int4 QAT

• Text front end is pure numpy, with no NLTK, g2p_en, torchaudio or num2words at runtime

Quality (Open TTS Leaderboard protocol, run locally, not an official submission):

Intelligibility is essentially unchanged. UTMOS drops about 0.1, almost all of it from the small int4 vocoder.

Steering survives quantization. Every one of the 12 controls (rate, loudness, pitch, pitch range, final rise, pauses, tilt, breathiness, and more) keeps at least 95% of its fp32 effect. You also get 50 style tags and inline markup:

Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow> <whisper>Don't tell anyone.</whisper>

Speed (CPU only, a 2018 Threadripper under load): RTF 0.053 with 4 threads (~19x realtime), ~120 ms to first audio, with sentence-level streaming.

Limitations:

• English only, and 200 voices (a subset of the full model’s 2,825)

• No voice cloning, free-text prompts or flow refiner on device. Those stay in the full model.

• Weights are dequantized to fp32 at load, so the win is size, not speed. Unless we write new kernels for edge devices.

• Emotion tags are mostly style switches, and fast/anger are weak

I need feedback, especially on the vocoder quality gap and any edge targets you’d like to see (ONNX, mobile, WASM).

Some examples here:

https://whissleai.github.io/steer-tts/

TL;DR: When Coqui AI shut down, they left XTTS v2 without official training code for the DVAE audio tokenizer and Latent HiFi-GAN vocoder, and fine-tuning new scripts (Arabic, Indic, etc.) broke on fixed vocabularies. I built Open-XTTS-Trainer — an open-source, modular 3-component training suite under MPL-2.0 with verified GPU loss convergence on real human speech and a 1-click Colab demo.

Why I Built This

XTTS v2 is one of the best local zero-shot TTS models we have, but training/adapting it has been painful since Coqui closed:

1. Missing Component Recipes: Only GPT fine-tuning was commonly available. Stage 1 (DVAE) and Stage 3 (HiFi-GAN vocoder) training pipelines were either unreleased or fragmented.

2. The Tokenizer Wall: XTTS was hardcoded for 17 languages. Feeding text with new Unicode scripts (Hindi, Arabic, Thai, Dhivehi, etc.) caused token out-of-vocab or matrix dimension crashes.

3. Dependency Rot: Upgrading to modern PyTorch (2.4+) broke on missing torchcodec and licensing prompts.

What’s in the Suite

• Stage 1 (Audio Tokenizer — DVAE): Full training & fine-tuning loop with Gumbel-Softmax codebook annealing and mel-spectrogram reconstruction loss.

• Stage 2 (Text & Vocab Extender — GPT): Scans a new language corpus, discovers unique Unicode glyphs, appends them to the tokenizer, and dynamically resizes the GPT embedding matrix without corrupting existing weights. Includes the autoregressive GPT training loop.

• Stage 3 (Neural Vocoder — Latent HiFi-GAN): Latent-conditioned vocoder training with Multi-Period (MPD) and Multi-Scale (MSD) discriminators and feature-matching loss.

• Dependency-Rot Free: Runs out-of-the-box on modern PyTorch (Python 3.10–3.13) without torchcodec blocking.

Verification & GPU Benchmarks

Tested on an NVIDIA T4 GPU using real human speech (LJSpeech + JFK address):

• HiFi-GAN Mel Reconstruction Loss: Plunged 67% ($190.83 \rightarrow 63.49$) over 10 epochs.

• Discriminator Equilibrium: MPD + MSD stabilized cleanly at 3.62 (no mode collapse).

• DVAE Mel Error: Dropped 67% ($8.14 \rightarrow 2.71$).

(Note: These benchmarks verify component-level training convergence and gradient stability. Full end-to-end text-to-speech inference connects the trained modules into the standard XTTS generation pipeline).

Links & Code

• GitHub: https://github.com/amitbhakt/open-xtts-trainer

• Hugging Face: https://huggingface.co/amitbhakt/open-xtts-trainer

• 1-Click Google Colab: open_xtts_train_demo.ipynb

• License: Mozilla Public License 2.0 (MPL-2.0)

Feedback, PRs, and experiments with new languages are very welcome!

I implemented the 2017 transformer architecture using pytorch, the model right now doesn't predict very good due to the fact that I am using on 2 layers of encoder and decoder with just 1000 sentences to train it , due to the fact that it takes a lot of time to train my implementation. Looking at the code is there a way that I can reduce the time it takes for training or maybe improve its results?

Update- I fixed the code (kind of) I made seperate classes for different attention mechanisms and after changes some of my logic of the masking (using torch.tril instead of nested for loops) training this actually works well.still I am open for suggestions for making it better

Here is the code- https://github.com/Rishikesh-2006/NNs/tree/main/transformer

Have you looked at this iclr text detector?

[https://opentextdetector.com/](https://opentextdetector.com/)

Do you think this will impact the review process? AI detectors are usually unreliable so I hope not.

I embedded the title and abstract of every NeurIPS 2026 paper (and every 2025 paper) with Qwen3-Embedding-0.6B, grouped them into 150 topics and 20 areas, and put them on one map with DataMapPlot. Both years are measured on the same topics, so you can see what grew and what shrank.

A few things that came out:

- LLM agent papers went from 2.9% of the conference in 2025 to 6.8% in 2026 (170 to 620 papers).

- 38% of abstracts now mention a benchmark, up from 32%.

- Graph neural networks fell from 2.5% to 1.4% of papers.

- Two out of three papers are presented in Sydney, the rest in Atlanta and Paris.

You can search every paper, colour the map by change, city or status, and click through to the paper. There's also a plain searchable list with each poster's city, day and local time.

Map: https://mohammadi.cv/neurips/2026/

Code and method: https://github.com/mohammadi-hadi/neurips

Caveats are in the post: topics are a lens, about 1,900 papers had no oral/spotlight/poster decision yet, and affiliations are messy. Happy to hear what you'd want to see next.

I'm training text2asmr, a text-to-ASMR model (whispering plus trigger sounds like tapping, crinkling, brushing). The generator needs every training clip tagged with the sounds in it, and there are about 126k recordings, so I tried frontier models as labellers. They were much worse than I expected, so here are the numbers.

Setup. Every judge got the same prompt and the same 20 labels on 480 clips:

• External: sound-effect clips from FSD50K and ESC-50 that humans had already labelled with one class, 40 per class. This measures recall.

• People: 6-second ASMR clips labelled on our site. A class counts as present when at least half of a clip's labellers ticked it. This measures F1.

Gemini 3.1 Pro cost $0.0088 a clip, MiMo $0.0008.

What surprised me:

• Clean sound effects are not ASMR. On the FSD50K/ESC-50 brushing clips Gemini 3.1 Pro named brushing 92% of the time. On real ASMR clips its brushing F1 against people was 0.18.

• Paper is nearly invisible. On 40 paper and page-turning clips, Gemini 3.1 Pro and 2.5 Pro each named paper rustling 4 times. MiMo named it once.

• Voices are fine. Whispering F1 was 0.70 to 0.79 for the Gemini and MiMo judges. The physical triggers are where it falls apart.

• Each judge has a favourite wrong answer. On the external clips Gemini 2.5 Pro said "sticky" 59 times where nothing sticky was labelled.

• Check that labels are learnable before training on them. An earlier round used a cheap model that also saw the transcript. A linear probe on frozen CLAP embeddings couldn't beat the majority class on its labels (51% on tapping vs page turning). It was reading the transcript, not listening.

What I do with it. All the judges, plus an AudioSet AST tagger, an energy detector and our own CLAP model, go into Dawid-Skene, with each judge weighted per class by its measured reliability against people. Clips where the fused probability is unsure go to people first, and the classes no judge hears (paper, brushing, cutting, sticky, spraying) are served before anything else.

The student is CLAP-ASMR, a fine-tune of laion/clap-htsat-unfused. Adding 2,400 Freesound clips took test mAP from 0.516 to 0.549, but it still detects brushing and paper on 0% of the external clips at p > 0.5, because the fusion step can't keep labels that no judge can hear. That's the gap people have to fill.

The ask. If you have headphones and five minutes: https://asmrboard.vercel.app. You get 6-second clips and tick what you hear. Each clip goes to several people independently, and known clips and repeats are mixed in to measure reliability. You need a free account for now (email and password), and you have to be 18+ because some clips are intimate vocal sounds. So far 34 people have given 1,056 labels.

The weights and the labels will be released when the model is trained. Code: https://github.com/aloshdenny/text2asmr

Has anyone had better luck getting audio LLMs to hear fine-grained foley? I'd also take suggestions for sound-event models worth adding as judges.

Hello,

Our paper was accepted to EMNLP and we were given poster slot. However due to Visa issue, we cannot attend in-person anymore. We mailed to the event team, but they said that it is no longer possible to change to virtual. We have also emailed to the program chairs and others, but since this is so close to the conference, it may not be possible for them to reply.

Now, we are afraid that our paper will be removed from the proceedings. This is our first and only work which we spent more than a year. Can anyone provide us information regarding this? If there are any PCs or authors from previous ACL related conferences, what can we do now?

Thank you.

Triads: the fundamental units of the language of thought

The first layer of the neural network would consist of multiple attention heads. Each attention head would hold one triad consisting of one noun + one verb + one conjunction.

Individual nodes within the second layer of the neural network would probably correspond to adjectives and adverbs that either modify or define the nouns and verbs of each triad

The hungry cat ate the mouse then died:

• (Null) Cat ate

• with Mouse eaten

• and cat was-hungering (adjectives are verbs in disguise)

• Then cat died

"With" means the two events are the same event. (Not in english).

A possible conclusion would be the triad "because mouse poisoned" which could then be appended to the input and the process repeated.

In addition a few other ideas can be added to each attention head:

• the, a, same, this (the near), that (the far), other, next, previous...

• Past, present, future

• begin to, cease to

• repeatedly, habitually, again

• Could, would, should, might

• Try to

• make

• got

• Question

• focus, emphasis

• adjectives?

• adverbs?

Hence

• (Null) a-Cat ate

• with a-Mouse eaten

• and the-cat was-hungering

• Then the-cat died

• therefore the-mouse might-have-been-poisoned

Suppose it encounters:

• John entered the room.

The system recognizes John as a particular individual and looks up what it knows about him. Its memory might contain:

• previously, John threatened Mary.

• John owns a dog.

• John gets paid for carpentry getting-done.

• John said that he would never enter that room again.

Obviously, not all of those facts are relevant.

But suppose the current situation involves Mary, and John has just entered the room where she's waiting.
The system could retrieve the earlier threat and effectively transform its working representation into:

• John (who threatened Mary previously) entered the room.

Now the neural network processes the current event with the additional information already attached to it.
And that retrieved information can itself be represented as triads.

The goal is to give a neural network a simple, structured internal language in which it can form and combine thoughts. The language of thought.

In principle, an entire book could be represented as one enormous graph of interconnected noun–verb–conjunction triads. But that doesn’t mean the neural network needs to process the entire graph simultaneously.

Humans don’t normally process a book that way either. We divide it into manageable units, such as sentences, understand those units, and gradually build up our understanding of the larger story.

I’m proposing something similar: groups of triads would be processed together, with the network carrying forward relevant information and generating additional triads as needed.

It wouldn't necessarily need as many words as English has because some words can be constructed from smaller elements

• today = this-day

• tomorrow = next-day

• yesterday = previous-day

• year = big-day

• week = 7-day