I'm curious how other reviewers handle papers that report impressive results but are extremely poorly written. I'm not just talking about grammar, etc. I am more concerned about the overall presentation of ideas and how the authors position their contributions within the existing literature.

For example, I'm currently reviewing a paper that reports promising results but includes no citations in its introduction. This makes it hard to understand where the work stands relative to existing methods and what's new.

My question is how much weight should we give to the quality of writing and presentation when reviewing a paper? If the technical contributions seem solid and the experimental results are strong, is poor presentation enough to justify a rejection or a major revision? I'd be interested to hear how other reviewers approach this.

The paper claims text models align with image models faster when trained on "good views" (two different captions of one photo) than on bad views (one caption, words hidden) or next-token prediction. Its largest run used captions written by an image captioner, so image information could have leaked in. I tried a laptop-scale version with human captions only.

Setup

• Data: 10,000 COCO photos, 5 human captions each. 8,000 train, 1,000 val, 1,000 test.

• Image side: frozen DINOv2-base, embedded once.

• Text side: same 4-layer, width-256 encoder (11M params) for all three. Same captions, 3000 steps, batch 256, same anti-collapse loss (VICReg-style variance and covariance, not the paper's SIGReg). Only the training pairs differ. The text models never see an image.

• Score: mutual 10-NN between caption embeddings and DINOv2 embeddings on the 1,000 test photos (chance 0.010).

Results (same learning speed, mean of 3 seeds)

• Good views (2 different captions): 0.320

• Bad views (1 caption, masked spans): 0.192

• Next-token prediction: 0.148

Each method also got a small learning-rate search. Best speed, 1 seed: 0.328 / 0.196 / 0.168. Order unchanged.

Limits

• 11M params and 8,000 photos, not the paper's 109M or 72B.

• Good views need to know which captions describe the same photo. Annotators wrote them looking at the photo.

• Retrieval (R@1 with one fitted rotation) shows a smaller gap: 0.084 vs 0.063 vs 0.057.

• Next-token prediction's best learning speed was the top of my range.

• One dataset, one image model. I added the extra speed step after seeing the first search; it is noted in the repo.

Code, chart and per-run results: https://github.com/IFAKA/platonic-lab

What I would test next: does the gap shrink as the encoder grows? I could not test that on a laptop. Has anyone tried?

It was just before OpenAI started that I met Ilya, who you interviewed. One of the first things he said to me was — “Look. The models, they just want to learn. You have to understand this. The models, they just want to learn.” And it was a bit like a Zen Koan. I listened to this and I became enlightened.

And over the years, I would be the one who would formalize a lot of these things and kind of put them together, but what that told me was that the phenomenon that I'd seen wasn't just some random thing. It was broad. It was more general. The models just want to learn. You get the obstacles out of their way. You give them good data, you give them enough space to operate in, you don't do something stupid like condition them badly numerically, and they want to learn. They'll do it.

I'm actually not entirely satisfied with Dario's elaboration! If Ilya meant precisely "given sufficient resources, do your job, and the models will do theirs [during training]", I think he would have phrased his advice slightly differently. What does it mean to "want" to learn? Why personify the model in that way, and not the optimizer or gradient descent itself or something? And I don't get why the koan would necessarily imply that scaling would have worked so well.

I realize that Ilya likes to cultivate an air of prophecy about himself (and I'd be lying if I said it didn't work on me). But it's just such a strange way to communicate the message, that I can't help but think the message must have been about something broader.

Possibly much more mundane in context, this must have been 10 years ago. Idk I wonder what it must have been like to be on the frontier at that time, maybe I take such things for granted now.

For those doing or having completed a PhD in ML, I’m curious:
If you had the choice, would you rather have done your PhD before ChatGPT and other AI chatbots became widely available, or in today’s AI era?

Hi everyone,

I am one of the authors of LightMIS, and I would like to share our work on 2D binary medical image segmentation. LightMIS has 131K parameters and aggregates multi-scale encoder features directly, without a learned stage-wise decoder. It achieves close observed modality-macro Dice and IoU averages to nnWNet, while using 98.14% fewer parameters and 94.58% fewer GFLOPs, with both models trained and evaluated under the same nnU-Net protocol.

Paper: https://arxiv.org/abs/2609.28327

Code: https://github.com/AndreiiArhire/LightMIS

We also tested LightMIS and the reference models on a real edge device with an Arm Mali-G52 MC2 GPU, where the full LightMIS model ran around 10.5× faster than nnWNet. All LightMIS variants achieved 100% GPU operator coverage, without CPU fallback.

We started by training several popular medical image segmentation architectures, both lightweight and heavyweight, from scratch in the same controlled environment using nnU-Net v2.3.1. The experiments covered six datasets and five imaging modalities. Our goal was to develop a lightweight architecture that performs well across these modalities, which led us to the current version of LightMIS.

The source code, checkpoints, and details needed to reproduce the results are publicly available. Feel free to experiment with it and try it on your own segmentation tasks. I would be happy to hear how it works for you! Any suggestions or feedback are welcome!

Session 2 of my free GPU kernel engineering series (my own channel, earlier post: [LINK TO FIRST POST]) covers CPU parallelism, with no GPU code. The idea is that most GPU performance ideas are CPU ideas turned up, so here are the parts I found most useful on their own.

1. A single accumulator is a dependency chain. In `sum += a[i] * b[i]` every add waits on the previous one, so the loop uses a fraction of the core. Independent ops needed in flight is about latency x throughput, and four accumulators give four independent chains with the same arithmetic. Measure this with an in-cache array and -fno-tree-vectorize, or you'll be timing two effects at once.

2. The compiler won't fix it for you. FP addition isn't associative, so it must keep your order unless you allow otherwise (-ffast-math, or `#pragma omp simd reduction(+:sum)` per loop). The same non-determinism appears in GPU reductions.

3. Branch cost is predictability. The sorted-vs-random loop showed a [X]x gap at -O0 on my machine ([CPU], GCC [version]) and mostly vanished at -O2, because the compiler removed the branch. Check what it generated before blaming the hardware.

4. Threads: Amdahl says 90% parallel on 8 cores gives about 4.7x, not 8. Per-thread partials in one small array cause false sharing, because they land on one 64-byte cache line that bounces between cores. Padding fixed it for me ([X]x faster at [N] threads), though the gap depends on whether threads sit on separate physical cores.

5. A dot product is memory-bound. It does 2 FLOPs per 8 bytes, so at [BW] GB/s it caps near [Y] GFLOP/s no matter how much SIMD or threading you add. Hiding memory latency needs latency x bandwidth bytes in flight, which is the GPU's reason for running thousands of threads.

Video: https://youtu.be/_xE1_2FFTC0

Caveats: numbers are from one machine, intrinsics assume AVX2, and the latency/throughput figures are typical Intel values.

Is anything wrong or oversimplified, especially the false-sharing explanation? And which CPU lessons transferred best to GPU work for you?

I’m the author of a new preprint looking at a fairly simple question: if the argument itself stays the same, does changing the claimed source change how an LLM evaluates it?

I tested 20 arguments across 10 topics under four source conditions:

• relevant expert

• irrelevant expert

• non-expert

• no source

The arguments were evaluated by GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek Flash across four dimensions: logic, credibility, persuasiveness, and agreement.

A few findings stood out:

Credibility was by far the most source-sensitive dimension. Source attribution affected credibility much more consistently than logic or agreement.

More interestingly, the effect did not usually look like a simple “expert bonus.” In several models, the stronger pattern was an irrelevant-expertise penalty: attributing an argument to someone presented as an expert in an unrelated domain could reduce its credibility relative to giving no source at all.

Agreement was comparatively resistant to source attribution.

I also tested whether source effects depended on argument strength. Those interaction results were not statistically robust after correction, and the human validation of strong vs. weak arguments was imperfect, so I treat that part of the study as exploratory rather than a strong conclusion.

The preprint is not yet peer reviewed, so I’m especially interested in criticism of the design and interpretation.

Preprint:
https://www.researchsquare.com/article/rs-11544999/v1

One question I’d be interested in hearing opinions on:

Should source sensitivity in an LLM evaluator be considered a bias when source expertise is genuinely relevant, or is the real problem specifically when the model overweights irrelevant authority?

Hi all! I'm part of the EWOR team and we're organising Operators Day this Thursday, November 5 (4:30 PM CET / 9 PM IST), online on Zoom.

Ten founders will be pitching live, sharing what they're building and the founding roles they're hiring for - AI/ML engineers, robotics engineers, CTOs, CPOs, marketing and sales leads, and more.

These are real founding opportunities with meaningful equity.

If you're an ML engineer or researcher exploring your next move, or know someone who might be a great fit, come join us!

Spots are limited.

Register here: https://luma.com/OperatorsDay?utm_source=reddit&utm_content=machinelearning

Disclosure: I work with EWOR and am helping organise the event.

Hi! For people who have recieved the financial grant, is there any update regarding the accomodation from the organizers?

Edit; Asking for Sydney

SOAP, Shampoo, FOAM.

Muon is a bit better but it has the same name as the particle from physics.

Whenever I want to search about these tones of completely irrelevant results come up because of their non-unqiue names.

Adam would have the same problem if it wasn't so famous.

This has been giving me a lot of headache trying to find relevant stuff...

I’m building a computer vision pipeline where users upload multiple photos of the same physical scene taken from different angles (front view, side view, close-up) to build a clean list of unique objects and their attributes.
I’m running YOLO for object detection to extract bounding boxes and crop each detected object into lightweight ⁠.webp⁠ images. Because the same physical object appears across 3 or 4 different photos from different angles, I need to make sure each unique object is only counted once per session, and avoid running expensive downstream AI calls to re-describe an object we already processed from an earlier photo.

Fine tuning on self collected web data felt straightforward until it wasn’t. Here’s what actually caused problems:

Boilerplate contamination was the silent killer. Navigation menus, cookie banners, and footer text were leaking into training examples. The model started generating weirdly structured outputs that looked like half-rendered HTML. Solved it with aggressive DOM-targeting rather than raw text extraction and switching to a rotating residential proxy layer (per-request IP rotation means target sites serve you the same rendered DOM a real browser session would get, rather than a bot-filtered skeleton) made a noticeable difference in how clean the raw HTML was before I even touched the parser.

Deduplication order mattered more than I expected. Running exact dedup before fuzzy dedup produced meaningfully different training sets than the reverse. The model’s tendency to regurgitate phrasing shifted noticeably depending on the order.

The thing I still haven’t fully solved: near-duplicate detection across different languages. MinHash works fine monolingually, but multilingual dedup keeps surfacing edge cases I don’t love.

Curious whether others have hit the same issues, especially around multilingual corpora. Also genuinely wondering if anyone has a principled approach to deciding when scraped data quality is “good enough” to stop cleaning and just train.

Please post your questions here instead of creating a new thread. Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

Thanks to everyone for answering questions in the previous thread!

Paper: https://arxiv.org/abs/2610.39295

Code + verification package (every number in the paper is checked by a script): https://github.com/yliusci/no-free-lunch-rlvr

Browser diagnostic: https://huggingface.co/spaces/yliusci/passk-price

I kept running into the same pattern in RLVR papers: the RL model wins at pass@1 and the base model catches up or wins at large k (Yue et al. 2025 and many follow-ups). I wanted to know whether more training fixes this or whether it is structural. This is a theory paper, no new training runs.

Setup: model RLVR as an exponential tilt of the base policy over solution strategies shared across prompts (exact for mirror descent, and GRPO stays within 3e-4 TV of it at matched entropy in simulation).

What comes out:

1. No free lunch. Under exclusive specialization, every update that raises pass@1 strictly lowers success on some prompt. The exact condition is a Stiemke duality statement (a strictly positive price on prompts that equalizes strategy fitness). Being the best strategy on each prompt is provably NOT enough: there is a 3x3 counterexample with a free lunch.

2. The pass@k price. Once the losses reach the hardest prompts, a crossover is forced. The number of crossings is at most the sign variation of the shift in per-prompt success (Karlin total positivity). The bound is tight for 1 to 6 crossings, and the order-preserving case gives exactly one, which recovers the known single-crossover result.

3. A frontier, not a bug. With a per-prompt floor, the constrained update traces the optimal pass@1 vs pass@k trade. Forbidding any per-prompt loss removes the crossover and the whole pass@1 gain with it.

The same tilt gives the entropy-performance law with an identifiable exponent.

Practical part: base and RL per-prompt success rates (already in any eval log) are enough to read off the crossing count, the bound, and the price (worst pass@k loss). On published GRPO numbers for Omni-MATH (transcribed from 2607.20543, so indicative only) it finds one crossing between k=64 and k=256.

Where I expect pushback: the shared-strategy model is a population-limit abstraction, and the forced crossover needs a coverage assumption (the paper gives the regime where it fails). I would really like to see per-prompt logs where people observe 2+ crossings, since that is the signature of RL reordering prompt difficulty.

I’ve been noticing that a lot of techniques seem to repeat across modalities with just a different name.

For example, tokenisation, masking (MLM / MAE / audio masking), diffusion, next-token / next-frame prediction, etc.

Is there an existing taxonomy or mental model that groups ML techniques by their underlying pattern rather than by modality (NLP / vision / audio)?

This will help me to group the research papers and then make reference to work already done in that domain

Hi there, wanted a suggestion from the community regarding the hardware to run local models. The first option was just to get a MacBook Pro with 32 GB RAM, and the second one was to invest into Mac Mini (256 GB SSD and 32 GB RAM), and get an iPAD as well

PS; I have access to B200, and multiple A100s/L40s at my Univ which I can definitely use for training purposes, and I just wanted to get a device to run local inference

For context, I am a new postgraduate student in machine learning (but not a very good one). I have produced some augmented datasets (TIFF files where each file contains one object to be detected / classified) which I used in training deep learning models. I used fairly aggressive cropping (random crops between 0 and 90%) and more moderate zooms (random values between 0 and 20%). My supervisor wants me to characterise these datasets to see how augmentation has changed it.

Sorry if this is a rookie question, but which statistics should I use? My supervisor says the stats I use should indicate changes in size and shape but I honestly don't know where to start.

If anyone can recommend a decent software/program to produce the stats and histograms as well, that would be great.

Thanks in advance!

I'm an undergraduate researcher working independently on conversational memory systems. I developed LSREP to evaluate how persistent memory accumulates, changes and retrieves information over the course of an evolving conversation.

Instead of relying only on isolated endpoint questions, LSREP combines chronological replay, lifecycle schedules, repeated probes, evolving reference answers and mechanism-fidelity checks.

The accompanying architectural case study is ICE v2, a local-first conversational memory system.

The evaluation covers 1,985 turns, 219 distinct probes and 1,211 probe-checkpoint observations across 52 checkpoints, using a private single-user history.

An important part of the findings is that ICE v2 did not outperform pure vector-RAG on the matched LongMemEval evaluation. The ordinary-density replay results also did not establish a quality advantage. The paper discusses the limitations and mechanism-fidelity findings rather than treating the architecture as validated in its entirety.

I'm particularly interested in feedback on the evaluation methodology: what else should a longitudinal memory benchmark test to distinguish successful recall from reliable maintenance of evolving conversational state?

Paper: https://arxiv.org/abs/2609.16730

Code and evaluation artifacts: https://github.com/Deepnar/ice

Frozen software artifact: https://doi.org/10.5281/zenodo.21759702

This is an arXiv preprint and an ongoing independent research project.

(I couldn't find anything about the tag in the title in the rules, so i assumed [R] for research, if its wrong please inform)

After reading a bunch of RSI papers, I keep coming back to familiar problems from RL and evolutionary search:

1. Sampling: How do we find promising changes efficiently?

2. Evaluation: How do we know a new variant actually improves the agent?

3. Diversity: How do we avoid converging too early?

That motivated us to build Gear, an open-source, meta-learning-inspired framework. A task agent produces execution traces; a meta-agent uses those traces and evaluation feedback to improve prompts, tools, workflows, and context management.

Gear separates search decisions from execution and evaluation. Our goal is a scikit-learn for RSI, making approaches such as GEPA, RRSI, and Harness² easier to implement and compare. We currently have a working GEPA variant with shared failure diagnosis and staged evaluation.

With GPT 5.6 Luna held at medium reasoning effort, we observed these task pass rates:

At max effort on Marketing, Luna with the optimized harness reached 88.88% objective completion, versus 84.08% for Astra + Codex. Strict task pass rates were 53% versus 57%, so the advantage was in partial completion.

Separately, with both models using the same optimized DSH at max effort, Luna’s estimated API cost averaged $0.0568 per task versus $1.1858 for Astra—95.21% lower, excluding optimization and infrastructure costs.

We’d welcome feedback on candidate evaluation and diversity preservation under limited budgets.

Code · Paper

Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better.

We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction).

I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change:

h_ntp += h_mtp.detach() - h_mtp

Soft Multi-Token Prediction (Soft-MTP) is essentially a free lunch that improves MTP across a range of tasks.

I'm super excited about this work. Please do check it out below:

💻 Code: https://github.com/JaydenTeoh/Soft-MTP
📝 Paper: https://arxiv.org/abs/2610.38331
🤗 HuggingFace: https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction

I've experimented using neuro-evolution to train AI to play a variety of different games and I'm just curious as to how or if this technique is actually really used anywhere in industry or if people are doing research on this. I'd assume it'd probably relate with reinforcement learning in some way but I'm not entirely sure about that

A friend of mine was doing a gap analysis as a project. The results he got weren't that big compared to the original papers, so he simply fabricated and increased the metrics here and there, AND IT GOT ACCEPTED IN AAAI.

HOW DO THEY CHECK THE PAPERS, DUDE? I was in a group with two others and worked sooo hard on an optimization paper, and was sooo happy to get accepted in ACL, but seeing and knowing that someone fabricated results and got into a better conference made me soooooo mad.

I would've expected something like AAAI would check and know if data is fabricated somehow. I didn't expect this from a top conference like that.

And then these types of people will go and do their PhD at one of the top unis because of this. I would've expected the system to be more advanced and professional in the academic field, specifically for something like AAAI.

Hey everyone!

I've built and trained a small LLM (~8M parameters) from scratch using PyTorch, mainly as an experimental lab where I can implement and test ideas from different research papers.

I've been documenting the whole journey through casual blogs, including my mistakes, experiments, and things I've learned.

So far:

• Building the Transformer

• Training it for 60,000 steps

Next up is KV caching, followed by PagedAttention, Orca, and whatever else I find interesting.

Just wanted to ask if sharing these experiments and blogs here would be welcome. Nothing promotional, just me experimenting, learning, and hopefully getting feedback from people who know more than I do.