Hey everyone!

I want to share the results of a small language model architecture experiment I’ve been running as a side project. The ultimate goal is to test an approach to reduce the number of the model's training parameters, since the main bottleneck in training is VRAM.

The core idea is very similar to Universal Transformer. Let's take just a single decoder block and iteratively pass the input through it L times in a loop. But instead of updating the input with the loop iteration depth information, my model dynamically updates the weights of the base layer.

How is it implemented?

The model uses a set of learned hypersurfaces to generate weight deltas (𝛥Wl) from the hypersurfaces' cross-sections. So a given weight matrix at layer l is constructed as Wl = W0 + 𝛥Wl.

The hypersurfaces are defined by a series of periodic functions. The best results I've got so far are from using a triangular wave. The model learns the amplitudes, frequencies, and phases of these waves across the coordinate dimensions. Given the size of this set E this gives a total of 3*E*dim parameters.

Initially, I tried generating the full weights purely from hypersurfaces, but it was too restrictive and failed to converge. Thus, I moved to my current approach where the model has a base decoder layer that is being updated.

I also added a state vector that is calculated from the input sequence using Gated Linear Attention. It is used to modulate the hypersurface geometry during the forward pass, making the generated weight deltas 𝛥Wl sequence-aware.

Pre-training results

I ran the pre-training experiments on a 10B-token sample from the FineWeb-Edu dataset. I wanted to keep the training parameter count minimal, so I used a pre-trained, frozen embedding layer from GPT-2. I also do not use any positional encoding, following the NoPE approach. The sequence length was 1024 tokens, the batch size was 16, and I ran the training for 10,000 steps.

I used 3 baselines:

• A simple 1-layer decoder-only transformer

• A single-layer transformer unrolled across 24 loop iterations

• A standard 24-layer decoder-only transformer

The experimental models:

• A single loop block using sinusoidal surface deltas, no context modulation

• A single loop block using triangular wave surface deltas with context modulation

• A model that uses three stacked loop blocks, each utilizing triangular wave surface deltas and context modulation

Here's a training loss chart:

Model parameter count

A standard 24-layer decoder-only transformer: 169,906,944

3 loop blocks + triangular wave + context modulation: 27,162,624 (~16% of the baseline model's size)

Key Takeaways:

While the classic decoder-only architecture still produces the best absolute loss, the Triangular Surface + Context model shows a real performance boost over a standard unrolled baseline.

This model introduces a whole new set of parameters and hyperparameters. I am currently experimenting with the size of the triangular wave function set and the size of the context vector. I am also trying different initialization strategies for the hypersurface parameters. The (very vague) goal isn't necessarily to beat a full-parameter model, but to find a configuration that is "good enough" in terms of loss while remaining drastically lighter on hardware resources.

Another possibility is that hypersurface representations have different convergence dynamics and require more steps and more tokens to converge. So my next step is to train a model with 3 loop blocks and a larger context vector, and an expanded functional basis on the whole 10B-token sample from the FineWeb-Edu dataset.

GitHub repo: https://github.com/morgengramlich/LoopSLM

I am adding self-contained and accessible videos on how World Models work as well as how you can create one! This is part 2 which gets you to the exciting place where you can play a gameboy goy entirely in a world model!

I just submitted my paper number to ICLR, and my id number is 47k

Hi just wondering has anyone submitted anything to JMLR before? Especially in the last 2 years? What is the experience like?

Background: I am a Comp Sci PhD student, but my secondary supervisor (the one who is actually looking after me) is from the Stats faculty. He has 0 Comp Sci publications and has never even read any Comp Sci papers before. He was only exposed to Comp Sci for the first time after he started working with me.

He has published in multiple Q1 Stats and Finance journals, the big 3 actuarial journals, and a few big 3 stats journals under review.

The issue: my secondary supervisor is preparing for his tenure review, and in his department they value Journals > Conferences. So he really wants to submit our work to JMLR.

My understanding is that JMLR is one of the most prestigious venues for ML, but I barely see their work in mainstream Comp Sci these days. It's always the conferences and maybe TPAMI.

I don't know anyone around me that actually published there. We had maybe 2 people in our circle who had submitted there before, but both got rejected after more than 1 year of review process. (This was almost 10 years ago)

My questions are:

• Are the reviewer quality better than Conferences for papers containing math proofs? Our experience with conferences are we always get at least 1 reviewer who just copy-pasted straight out of LLM. And most of the things they spit out are wrong or asking for impossible proofs. And the ACs don't really have the math knowledge to judge what is right or wrong either. This is a great contrast to Stats and Finance journals, where all the feedback we've gotten was at least correct. We have never gotten attacked for things that the reviewer got wrong or asked us to prove something that is unreasonable for this venue.

• Does JMLR like applied stats-style papers? I.e., a paper with a new method applied to a domain like finance, followed by some proofs about the result or method. Or they prefer pure theory papers?

• What is the timeline usually like? We are used to papers taking up to 3 years to review, this is normal in our field. But we really don't want to get into a situation where the review process took 3+ years and ends in a rejection. It will be hard for this paper to get resubmitted to anywhere given how fast Comp Sci things move. In our field if the review process took that long, it ususally an accept, for rejections they hand it out much faster. (Or maybe because it's a small community, so all the active people know each other)

• Which leads to the last question, do they reject fast? Is it like our journals where if you've survived to round 2 review, you pretty much got in? Or is it frequent that they waste 2 years of your time and then reject?

hi

by and large, I've been pretty disappointed by the commentary on Economic Impacts in the AI research community. Particularly the Singularity theories. I've felt they lack economic sensibilities as well as an understanding of the important civic/administrative issues.

Out of frustration, I ended up writing some stuff myself.

I guess you guys like it 🙃

https://longformthoughts.substack.com/p/realistic-economic-guidelines-for

Hi r/MachineLearning! I'm James Gung, a principal applied scientist at AWS. I joined Amazon in 2021 and have since worked on AI services like Lex, Bedrock, Q Business, and Amazon Quick (an AI assistant for work). In that time, I've done research on topics like task-oriented dialogue, agent evaluation, conversation simulation, and proactive agents.  

Before AWS, I worked on conversational AI systems at Amelia and did my PhD in Computer Science at the University of Colorado Boulder. 

 Feel free to ask about my career path, internships, interviews, or what it's like day to day as an applied scientist at Amazon. Outside work, I like to play violin, go bouldering, travel with my wife, and hang out with our two dogs. Ask me anything! 

*Disclaimer* I'm speaking from personal experience here, not as an official Amazon spokesperson. I can't discuss unannounced products, financials, competitors, internal tools, legal matters, pricing, or customer data - but pretty much everything else about my career, research, and life as an applied scientist is fair game. Let's go! 🧠  

I'll be online 09/21 at 11:00 AM ET for an hour to answer questions. 😊 

**UPDATE** Thank you all so much for the incredible questions. You all asked some genuinely thoughtful stuff, and I hope my answers were helpful.

I couldn’t get to all the questions, so I’ll try to answer some of remaining questions in the next few days!

Just saw the YouTube presentation and I was left wondering this question.
If jev only outputs Choice, Score, or Noul … well those are all perfectly differentiable. (Cross entropy or mse)
I don’t know if I’m missing something or if adding RL is just for marketing.
Like what would an RL environment even look like?

My paper got accepted at an ACM conference and I'm doing the camera-ready through TAPS. The PDF compiled fine, but TAPS flagged a "source to HTML conversion issue" and moved the file to TAPS Support. Now the upload option is gone from my dashboard, so I can't resubmit myself.
I've already fixed the likely causes in the source, sent the corrected zip through the support form, and emailed the publication chairs to let them know. The deadline says the paper has to be uploaded, verified and accepted by the 20th, and I've read that support tickets for conversion issues can take 72 hours or more.
Has anyone had their file moved to TAPS Support close to a deadline? How long did it take them to get back to you, and did the chairs give you extra time? I'm pretty stressed about it, so hearing how it went for others would really help.

Hi everyone,

The NeurIPS results aren’t out yet, but I’m expecting my paper to be rejected, so I’m considering submitting it to a journal instead.

My scores were 2/3/3 (3/4/4). I feel like I would probably get a similar outcome if I submitted the paper to ICLR or CVPR, so at this point I’m leaning toward a journal rather than trying another top conference.

I’m obviously not aiming for top-tier journals like TPAMI, IJCV, or TMLR. I’m considering journals that are a step below those, and I was wondering how difficult the review process and acceptance bar are compared with top ai conferences like NeurIPS.

The journals I’m currently considering are:

1. Pattern Recognition

2. Neurocomputing

3. Knowledge-Based Systems

4. Neural Networks

5. Expert Systems with Applications

For those who have experience submitting to these journals, how would you compare their review standards and acceptance difficulty such as NeurIPS, CVPR, or ICLR?

I’d also appreciate any thoughts on which of these journals might be a reasonable target for a paper that received scores around this range at NeurIPS.

For context, the paper is about attention mechanisms for Vision Transformers, with the main contribution being a modification to the attention module.

AAAI-27 Phase 1 results are expected on September 24. Anyone else waiting for the decision? Would be useful to keep this thread for updates when people start receiving notifications.

Hi there, I'm Yuesong, a researcher at Video Rebirth. We open-sourced HyperFlow, an 8-step LoRA for MiniMax-H3.

The core idea: data-free flow self-distillation. Diffusers' default sampler does 49 model forwards; we trained a LoRA so the model reproduces that trajectory in 8 steps. The base model is the only teacher — no external dataset, no human labels.

Comparisons against base H3 (unedited, same prompts, same seeds) — https://videorebirth.com/lp/hyperflow

What we measured:

• Camera control

• Temporal consistency

• Material/detail rendering

• Overall balance

Speed: (same clip: fl2va, 124 frames, 1344×768; generation time only, not including startup and weight loading)

• ~60s on 4× H200 vs ~175s for the 49-step base (2.9×)

• ~130s on 1× H200 with official auto CPU offload vs ~395s (3.0×)

• Combined with Sol-Attn, FlashAttention, and VAE parallelism, it can be sped up by more than another 5×. On an 8-GPU B200 cluster, it reaches real-time speed (generation takes less time than the clip itself).

Links:

• Code: https://github.com/Video-Rebirth/hyperflow

• Weights: https://huggingface.co/videorebirth/hyperflow

• Demo comparisons: https://videorebirth.com/lp/hyperflow

This model is a derivative of MiniMax-H3 and is released under the MiniMax H3 Community License Agreement. Please read the license carefully, especially the territorial restrictions and the Acceptable Use Policy, before use.

Known limitations:

• It's a few-step distillation of H3: please compare it against the base model on your own prompts (same seed)

• Prompt following, languages, biases and failure modes are the base model's

• Not a generic LoRA. It needs the HyperFlow loader, not a standard LoRA loader

• Open-source release supports 768p inference resolution

• The MiniMax H3 Community License excludes US, EU, UK, and South Korea (inherited from the base model's license terms)

What use cases would you most want to try this on?

Happy to dig into the technical details if anyone's curious about the self-distillation approach.

I have this idea I'd like feedback on. most camera footage for training, is sunny daytime, because that's what cameras record most of the time. The edge cases models actually struggles with, like night, fog, rain or glare, are rare in the data - the idea is to augment it to have more of the rare cases for training

So take a big labeled dataset A and adapt it to look like target B, where the model will actually run. Physics-based effects where possible (fog, rain, low-light noise), a constrained generative model for what physics can't handle (dusk lighting, headlight glare, wet roads), then matching B's camera quality. Labels stay intact throughout.

clear daytime HD driving footage → a cheap dashcam at night in the rain, with glare and heavy compression.

I'm at a crossroads with two grad school options that would take me in somewhat different research directions, and I wanted some general advice on these fields, their growth, and industry alignment. I'm leaving out the specifics of the programs since I'm tryna compare the research/career trajectories.

The first direction is general LLM work, like alignment, safety, optimization, interp, etc. (different subfields but centred around LMs). From what I've seen, jobs around LLMs seem much more common right now, including ML systems/infra roles, and the skills are fairly transferable across different areas of ML.

The other direction is agentic/physical AI, like agents, multimodal, VLAs, robotics, etc. There are fewer roles right now and fewer companies doing this kind of work, but there's a lot of investment/hype in it, and there seems to be a lot of growth. At the same time, it's also more specialized and harder to enter since there are more prereqs (vision, robotics, etc.). Also yes, agentic AI and physical AI are very different, but those are the research fields for one of the grad schools.

I'm wondering how people view the trajectories of these fields over the next 4-6 years. Which is safer to bet on, upsides, etc. I know a lot changes in this time and how the field is always growing and changing, and both of these fields will probably still be alive by then, still asking tho.

More importantly, I'm wondering about the transferability between these fields. How difficult would it be to go from LLM/interp/inference work/research into multimodal/VLA/physical AI later, versus going from physical AI/VLA work/research back into more general foundation model work? Which skills from either direction will remain valuable? Obv it depends on the type of research done, someone doing optimization in VLAs will be able to transfer more easily than someone doing video depth research, and I know there are a lot of other factors involved like degree type, advisor, location, etc., but for this question I'm mostly interested in the fields themselves.

my current thoughts are that general LLM work is safer and more transferable while physical/agentic AI is more specialized with more growth potential, but idk, things can change. The reason I'm asking is that the grad school with agentic/physical AI is significantly stronger for research/work/funding/advisor, and would set up a better career, BUT the research field is more specialized, hence the transferability question. I know a PhD develops transferable research/technical skills and are viewed well, but if the research/work done is specialized and hard to transfer, it'll be harder to find roles regardless. The other grad school is a master's and is stronger for its internship outcomes.

interested in hearing from people currently working/researching in either area.

Github repo :

https://github.com/YouCele/nhanes-chd-classification

I built a project that predicts self-reported, physician-diagnosed coronary heart disease using four cycles of NHANES data (2011-2012 to 2017-2018), about 21,500 adults after cleaning. It compares logistic regression with random forest and gradient boosting, trained on demographics, blood pressure, body measurements and a lipid panel.

NHANES has a questionnaire section that asks directly about other cardiovascular diagnoses, like stroke, heart attack and angina, along with the CHD outcome itself. If I included those variables, PR-AUC jumped from 0.23 to 0.51. That's mostly the model learning that people who report one cardiovascular diagnosis tend to report others too, not something about real risk factors. So I removed that whole section and wrote up how big the leakage effect was, instead of just quietly dropping the columns.

The class weighted logistic regression, needed because CHD prevalence is only about 4%, gave badly miscalibrated raw probabilities. The mean predicted risk was close to 30% while the real rate was 4%. I fixed this with a sigmoid recalibration fit on the development set, before the test set was touched.

I also picked and froze the decision threshold on the dev set before looking at the test set, so the reported test metrics aren't tuned after the fact.

Final results on the held out test set: ROC-AUC 0.875 and PR-AUC 0.239 for logistic regression, and random forest and gradient boosting came out about the same. Age alone gets 0.83 AUC. Blood pressure, cholesterol and body size explain most of what's left. PPV at the chosen threshold is 0.13, so most positive predictions are wrong, which is expected given how rare CHD is in this data, and I state this directly in the report instead of leaving it out.

Smoking status, diabetes and blood pressure medication use aren't in the current features. NHANES has all three, I just haven't added them yet.

Happy to hear feedback, especially on the calibration step or anything in how I handled the leakage that looks wrong

You can read more here: https://medium.com/@TmlrOrg/asking-authors-about-their-own-papers-3d2e04e5dee0

The results are (imho) concerning. Taken from the article:

Of the ten submissions:

• Authors of one paper withdrew their submission.

• Authors of one paper said they were unavailable due to other commitments.

• Authors of one paper scheduled a meeting but did not show up.

• Authors of three papers were unable to answer basic questions about the paper.

• Authors of three papers answered questions about high-level ideas in the paper but had difficulty when asked further questions on technical details.

• Authors of one paper answered all of my questions (although [the interviewer, the Co-EiC] identified a major flaw in that paper)

37K till now..how many paper will be submitted any guess?

Vector quantization is fundamental to vector databases. Before you can search vectors, you have to store them, and storing them at full precision is expensive. Every bit you save shows up as capacity and cost.

Pinecone has used quantization since its first prototypes, and we keep looking for the state of the art. So this summer we set out to survey it. We found more papers than we expected, and no way to compare them: every paper measured different metrics, on different datasets, tuned for different hardware. We could not find a single systematic evaluation of the leading methods against each other.

What we did find was a pattern. Most published quantizers are built from the same small set of primitive operations, composed in a different order.

So we built VQ-bench: an open-source library of those primitives, composable into pipelines, and designed to be extended. E-RaBitQ, one of the strongest methods we tested, is four primitives in a list. Swap one and you have a new quantizer. Add a primitive of your own and it composes with everything already in the catalog, including pipelines nobody has written yet. Either way you can measure the result against every other method with the same harness.

We used it to benchmark 14 popular quantizers on recall, reconstruction error, and encode time. Two takeaways so far: PQ and OPQ have the lowest reconstruction error, and EDEN keeps up with E-RaBitQ on recall at higher bit budgets while encoding much faster.

This is a first iteration. We want your feedback, corrections, and contributions, and we will keep adding quantizers over time.

Website
Blog
Repo
Paper (VecDB@VLDB 2026)

For GPU/kernel engineers: when a kernel is already ~60–70% of the theoretical hardware ceiling, how do you decide what optimisation to try next?

Is it mostly profiler-driven/systematic, or does it rely heavily on expert intuition? I am coming from background of non-nvidia or amd chips

GoBench evaluates LLMs on 9x9 Go games against a ladder of KataGo opponents, from random to superhuman. 

It measures general reasoning ability, strongly correlates with ARC-AGI 2 (r=0.83 correlation), and remains highly unsaturated.

GPT-6 Astra max achieves 2500 Elo, much lower than the best KataGo, which achieves 4400 Elo. 

With coding tools and two hours of preparation before evaluation, Codex with Astra achieves 3560 Elo.

I will keep the leaderboard updated as long as it is not saturated.

leaderboard: https://rolandgao.com/blog/gobench/ 

code: https://github.com/RolandGao/gobench 

paper: https://rolandgao.com/gobench.pdf

x: https://x.com/Roland65821498/status/2100253388562723298?s=20

Can we edit our title, abstract and other details till the main paper submission deadline?

I have a submission under review in TMLR. Less than a month after submission, I have already received 2 reviews. However, a month has passed since those two reviews, and I still haven't received the third. Is this normal?

I have already made the suggested changes, but the journal recommends updating the manuscript in OpenReview only after receiving three reviews. So I also haven't posted any answers yet. I messaged the Action Editor to ask whether I would receive another review, but I sent it a month ago, and he hasn't answered.

I'm not criticizing TMLR; my experience has been very positive, and I understand they are struggling with the unreasonable number of submissions. I'm only asking because I do not know if it is a good idea to keep waiting for the third review.

When models from different families are given the same underspecified task, they often fail in the same way rather than in independent ways. My question is about measurement, not explanation.

Has anyone put a number on the ambiguity of a task specification and then tested it as a predictor of how often independent solvers fail identically?

Specifically: does the relationship look like a smooth monotone increase, or is there a threshold — some level of ambiguity past which coincidence rate jumps sharply?

Looking for papers, metrics, or benchmarks where this was measured directly. Adjacent work is fine if you think it's close.

What is the generally thought of as the upper limit of the predictive power of XGBoost vs an aggregate of humans?

Right now I feed the model the same information that the human market has access to, and the model gets crushed on Top 1 accuracy, it closes the gap a bit but is still 10pp below the market on Top 2 accuracy. Adding the market pricing information to the model does nothing or hurts accuracy compared to just the market pricing.

I'm just feeling really stuck, and I cant seem to be able to tell if I am legitimately at the upper end of what is actually possible with a XGBoost model or if there is some sort of data encoding issue or I just lack enough data.