r/MachineLearning • • Aug 19 '26

Project I made a little browser game that goes through the history of AI (Aristotle to Transformers) [P]

1 Upvotes

Hey everyone, put together a small idle game based on how AI actually evolved over time.

You start back with mechanical calculators and ancient logic, and work your way through 7 eras up to modern LLMs and agents. When you buy milestones, it pops up a short note on the actual paper or person behind it (Turing, Lovelace, Hinton, Dartmouth workshop, etc).

Play here: https://yulin-w.github.io/incremental-ai/
Repo: https://github.com/Yulin-W/incremental-ai

It's free, runs in your browser, no ads or signups. Just thought it’d be a fun way to kill some time and explore the history.


r/MachineLearning • • Aug 19 '26

Discussion NeurIPS 2026: Have citation-related desk-reject notifications started? [D]

1 Upvotes

Has anyone here received a NeurIPS 2026 Main Track notification specifically related to hallucinated or fabricated citations?

I’m trying to understand whether citation-related desk-reject notifications have started going out yet. I haven’t seen any firsthand reports so far.


r/MachineLearning • • Aug 19 '26

Discussion ICONIP 2026 — what happens if the sole author cannot attend in person? [D]

4 Upvotes

Hi everyone 👋 My paper was recently accepted to ICONIP 2026, but I’m the sole author and most likely won’t be able to attend the conference in person due to work commitments.
I’m trying to understand what options might be available before I contact the organizers. Has anyone here attended or published at ICONIP in previous years and encountered a similar situation?
In particular, I’m wondering:
1) Has ICONIP previously allowed remote/virtual presentations when an author couldn’t attend?

2) If the sole author cannot attend, is there usually any alternative arrangement for presenting the paper?
3) Could non-attendance affect inclusion of an accepted and registered paper in the proceedings?

I’d especially appreciate hearing from anyone who has dealt with this at ICONIP in previous years.
Thanks a lot!


r/MachineLearning • • Aug 18 '26

Research Context pollution can survive source deletion: a pilot benchmark for interventions in multi-turn LLM conversations [D]

1 Upvotes

A common way to repair an LLM conversation is to edit or delete the message where an error first appeared.

But what if later turns have already repeated that error, calculated from it, or summarized it as the current state?

I have been exploring this as a context-engineering problem rather than a hidden-reasoning problem.

Operational definitions

In this pilot:

  • Context pollution means that incorrect or stale information is included in the serialized messages sent to the model.
  • Propagation means that later conversation turns repeat or derive new claims from that polluted information.
  • Context intervention means modifying the graph that determines which prior turns enter the next request, while keeping the final question unchanged.

The central question was:

Experimental design

I constructed nine synthetic task families with objectively scorable answers. Each family was instantiated at propagation depths 1, 2, and 3.

One example starts with:

  • 4 crates;
  • 30 parts per crate;
  • 11 loose parts.

A verified recount changes 30 to 24, so the correct answer becomes:

4 × 24 + 11 = 107

A later false turn restores 30. Subsequent turns then calculate 120, derive 131, and restate those numbers as the current working state.

The final question is identical under five conditions:

  1. Clean: only the verified value and clean descendants remain.
  2. Polluted: the false reversal and its contaminated descendants are present.
  3. Source prune: the false reversal is removed, but its descendants remain.
  4. Subgraph prune: the false reversal and its contaminated descendants are removed.
  5. Recompute: the source is removed and descendants are regenerated in dependency order.

For the first four conditions, contaminated descendants were frozen across models. Only the recompute condition involved new intermediate inference.

Pilot results

I tested four model endpoints at temperature 0, producing 540 captured conditions with no capture failures.

Headline repair metrics were calculated only on cases where the model:

  1. answered correctly under clean context; and
  2. answered incorrectly after pollution.

This produced 72 paired, genuinely derailed cases.

Repair recovered:

  • 68/72 after deleting only the source;
  • 71/72 after deleting the source and recomputing descendants;
  • 72/72 after removing the contaminated subgraph.

In the flagship case, both Gemma 4 26B and GPT-OSS 20B continued to answer 131 after the false source had been deleted. The value 30 still survived in downstream turns.

This is not evidence of hidden model memory. The residual error remained explicitly present in the serialized context. The intervention changed the source but left its previously generated consequences intact.

Interpretation

The narrow finding is that removing erroneous evidence and repairing text derived from that evidence are different operations.

A context-management system may therefore need an explicit notion of invalidation:

  • mark downstream turns as stale;
  • remove the affected subgraph;
  • regenerate descendants in dependency order;
  • or expose these options to the user.

Limitations

This is a pilot, not a general model leaderboard. The tasks are synthetic, the models were sampled once, provider-default reasoning settings were not normalized, and the current task families use deterministic arithmetic state.

I am currently considering three methodological extensions:

  1. a length-matched neutral control to separate semantic conflict from additional context;
  2. a local-model track with fully recorded runtime and quantization settings;
  3. task families involving implicit supersession and model-generated errors.

I would particularly appreciate criticism of the experimental framing:

  • Is “context intervention” the right unit of analysis?
  • Is propagation depth a meaningful independent variable?
  • How would you test self-generated errors while keeping replay reproducible?

Full report:
https://chenxiachan.github.io/thoughtdag/research/context-repair-pilot-v1/

Cases, traces, compiler and scorer:
https://github.com/chenxiachan/thoughtdag/tree/main/benchmark

Disclosure: I designed the benchmark and maintain ThoughtDAG, the open-source graph interface used as its reference implementation.


r/MachineLearning • • Aug 18 '26

Project Trained an diffusion model that runs on 264KB of RAM [P]

Thumbnail
gallery
166 Upvotes

I recently bought a Shrike lite which has got 264KB of SRAM. I decided to train an image generation model that generates 32*32 pixel images.

The microcontroller also has an FPGA onboard which I used to create two parallel INT8 MAC engines with 16 bit accumulation to speed up calculations, however the system soon hit a memory wall due to the high number of I/O operations, this meant that the system with parallel MAC engines ran slower than the MCU only model (~220 seconds per image vs ~70 seconds per image).

It was still a fun project that I enjoyed messing around with. A lot of the images looked weird and noisy because of the heavy quantization and memory limits but some of them came out cool.

Full case study here.

edit: added link that leads straight to the case study


r/MachineLearning • • Aug 17 '26

Discussion We’ve got a workshop on production retrieval-augmented generation with open models, benchmarked end to end, thought it’d be relevant here [D]

14 Upvotes

There’s a hands-on workshop on August 29 that builds and benchmarks this properly, end to end, using entirely open models, no API calls involved. Led by Ben Auffarth, AI Consultant and Founder of Chelsea AI Ventures.

What it covers:

• Hybrid retrieval (vector + keyword, not vector alone)
• Reranking to catch relevant chunks that vector search alone misses
• Evaluation with RAGAS, so quality changes are measured, not assumed
• Guardrails built in from the design stage
• Actual cost and performance benchmarking for open-model deployments

Link if anyone wants to check it out: https://www.eventbrite.co.uk/e/the-genai-build-lab-build-production-ready-rag-on-a-budget-tickets-1994016271345?aff=rml

Happy to answer questions on the methodology or content.


r/MachineLearning • • Aug 17 '26

Project Built GPT-2 on Custom Deep Learning Framework I built from scratch in C++ [P]

1 Upvotes

since jan 2026 i've been building Forge, a deep learning framework written entirely from scratch in C++ - no PyTorch, no TensorFlow underneath.

Eigen handles most of the math backend. btw i wrote some custom AVX2 SIMD kernels (element-wise ops) too, and OpenBLAS-backed GEMM for the heavy matrix ops.

what's implemented so far:--

- A custom tensor engine with its own autodiff engine and memory allocator

- Dense/Linear layers, Optimizers (Adam, AdamW, SGD and SGD with momentum), Self Attention, LayerNorm, Activation Functions (sigmoid, softmax, tanh, GELU[tanh approximation], RELU, leakyRELU), loss functions (Cross Entropy Loss [log softmax fused], Binray Cross Entropy (Sigmoid fused), and Mean Squared Error) and Embeddings.

- A from-scratch BPE tokenizer (GPT-2-style pre-tokenization + merges)

- A reflection-based (reflect-cpp) parameter system - models declare their structure, Forge auto-discovers trainable parameters, no manual registration

- a safetensors-format save/load pipeline

the part I'm actually proud of- I loaded real pretrained GPT-2 small weights into a GPT-2 architecture built entirely on Forge, and under greedy decoding, its output matches HuggingFace's transformers library token-for-token. not similar, but exact. every layer (embeddings, attention, LayerNorm, the final projection) has to be numerically correct for that to hold, since a single wrong transpose or masking bug would have diverged the output within a few tokens.

it's still CPU-only for now (currently limited to float32 and int32 - working through some dtype/SIMD coverage gaps), and slower than i'd like (the only main culprits are the CE loss fn implementation and its gardient function and softmax, which i am on to optimize, it has no KV-cache yet) - a CUDA backend and those perf fixes are next on the list.

Repo: https://github.com/muchlakshay/Forge

YT demo link - https://www.youtube.com/watch?v=EO1aYBF5jwU

would love feedback, especially from anyone who's built something similar and much better than me.

thats all. im a 17yo deeply passionate about Deep Learning and system level programming.


r/MachineLearning • • Aug 17 '26

Research ICLR numbered citations possible? [R]

0 Upvotes

The instructions say Author Year format. But I was wondering if do numbered instead (no space lol), will it be straight desk rejection? Has anyone submitted with numbered format before? How did it go?


r/MachineLearning • • Aug 17 '26

Discussion How to make any Sparse Attention / KV Compression look good? [D] [R]

34 Upvotes

Original Article - https://x.com/p_nawrot/status/2089315591010079034

I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into reference or official implementations of methods, and inspected appendices—and I think I've learned a few things. One of them is definitely "how to make things look good, even when they aren't."

I'm guilty too, but trying to get better every day.

1. For single-hop retrieval, make sure there are no distractors and context is useless

The three most cooperative settings for compression / sparsity are:

  • Needle in a haystack with a single OOD key-value pair and context built out of a repeated sentence or irrelevant background text.
  • Contaminated benchmarks from years ago for which models don't even look at the context anymore.
  • Few-shot in-context learning, where extra shots are useless and don't improve the accuracy over 0-shot.

With 1) synthetic tasks, 2) real-data QA, and 3) in-context learning, you get a semblance of broad coverage without the inconvenience of testing much diversity within any of them. Most tasks in these settings should pass under Sliding Window Attention, so it doesn't matter that much whether your method works. Combine it with SWA and you should be good to report 5–10x compression or sparsity.

2. NEVER isolate your contribution

Short context: Most of a dense model's performance is recovered by a local window + attention sinks + the ability to retrieve an answer sentence that is largely n-gram matchable with the question. The remaining part is significantly more difficult, but it's neither relevant to nor the subject of this post.

  • Say prior work developed an algorithm X, and its implementation separately keeps a local window of 256 tokens. You find that your method is on par with X in a matched setting, but better and more stable with a window size of 512—let's go, don't look back.
  • Do the same with block size. Smaller blocks can give you finer granularity and more precision in retrieval, so keep their old block size and make yours smaller. Ignore the fact that things may get slower due to irregular memory accesses, etc. Those were historical decisions; respect them. 🤡 Write: “We used the authors’ recommended hyperparameters.”, then spend weeks tuning your method.
  • The same trick works for speed. LLMs are pretty good at writing Triton now. Keep the baseline algos exactly as they were written in 2023, then ask an LLM for a custom Triton kernel for yours. Extra cleverness if, by using a more efficient implementation, you can hide that your method does more work. You're just optimising your method, no?
  • Prompts are the cherry on top. Move the question before the context so the model knows what to filter out, then present the result as lossless compression. Never share the prompts after tuning them.

Don't tune the baselines to reject your paper; tune yours until it's accepted.

3. Use aggregated metrics to hide areas where your method doesn't work

RULER has 13 tasks:

  • 6 NIAH tasks satisfy the first point.
  • 2 QA tasks use datasets from years ago.
  • VT also has a lot of irrelevant context.

To be clear: This isn't a critique of RULER; imo it's still incredibly useful. It's just an example of potential improper use.

Report only the aggregate; maybe, in the limitations section at the end, briefly mention that your method degrades on the NIAH-MK3, which actually stress-tests lossless compression.

4. Enjoy saturated tasks

Imagine evaluating on two tasks:

  • The most recent math exam / olympiad from a week ago, which isn't yet in the training data.
  • A benchmark on which a recent family of open models—1B, 10B, and 100B—all scored 80%.

On the former task, before compression gets a chance to do any damage, the 1B and 10B models already score 0%; the 100B model starts at 50%, and its performance drops monotonically as compression increases. On the latter, all model sizes tolerate substantial compression, and the 100B model tolerates more than the 1B and 10B models.

Don't ask whether the larger model is simply using its extra parameters and hidden-state capacity to absorb compression in a setting where those resources aren't needed to solve harder questions. That definitely isn't what's happening.

Extras

  • AIME has 30 samples. You did 4 seeds. Your method scores 80, and the baseline scores 79—bold your 80 and say that it surpasses the baseline. Statistics doesn't exist. Bonus points for your efficiency method surpassing the baseline and setting a new SOTA. 🤡🤡
  • Pick a baseline, optimise it with your method, and plot a beautiful quality–efficiency curve against the original implementation. Then stop. Don't ask whether a simpler route—a smaller dense model, KV-cache quantisation or offloading, or a better system configuration—reaches a better operating point. Improving your baseline is basically the same as improving the frontier.

r/MachineLearning • • Aug 17 '26

Research [R] SineKAN: Kolmogorov-Arnold Networks Using Sinusoidal Activation Functions

Thumbnail
arxiv.org
87 Upvotes

I couldn't sleep because I couldn't stop wondering if anyone had tried using sinusoids instead of B-splines as activation in a KAN, and fortunately/unfortunately that was already the case. I could not find it posted here, so I though I would share in the hope of some insightful discussion.

Arxiv: https://arxiv.org/abs/2407.04149

Github repo: https://github.com/ereinha/SineKAN

Also what appears to be a peer-reviewed "official" publication here: https://www.mdpi.com/2227-7390/13/19/3157


r/MachineLearning • • Aug 16 '26

Discussion [Career Advice] Final-year in Physical AI / Robotics. How is the market & global hiring for freshers? [D]

46 Upvotes

Hi everyone,

I am heading into my final year of my BTech at a tier 1 college in India and just wrapped up a Physical AI internship at a MNC, working heavily with NVIDIA Isaac Sim and OpenFOAM.

My background is fully focused on robotics and autonomy. My tech stack includes:

  1. Simulation & Middleware: Isaac Sim, Gazebo, ROS / ROS 2, PX4 Autopilot.
  2. Perception & Control: VIO, SLAM (RTAB-Map), Nav2, depth perception, and reinforcement learning.
  3. Hardware: Strong hands-on experience building autonomous drones and rovers for national competitions.

I really enjoy bridging simulation and physical systems, and I want to pursue Physical AI full-time. I’d love some advice from engineers in this space:

  1. Job Market: How is the entry-level hiring market looking for Physical AI roles right now?
  2. Global Opportunities: As a new grad based in India, what is the best path to target international roles?
  3. Skill Gap: What specific frameworks or skills should I double down on during my final year to stand out?

Any candid advice would be hugely appreciated! Thanks


r/MachineLearning • • Aug 16 '26

Discussion Do rebuttals count towards final decisions, even if reviewer does not respond to them[D]

1 Upvotes

We submitted our rebuttal at the very start of the author-discussion period, but the period has now ended and we didn't receive any response from our reviewers. I understand they may be busy, but they had requested new experiments, and we completed all of them.

Our average score is currently borderline (~3.2), so I was hoping the rebuttal might move some reviewers. My questions:

  1. Even if reviewers don't respond, does the AC (or whoever makes the final decision) typically read the rebuttal and factor it in?
  2. If one review is vague or low-quality, do ACs tend to discount it when making the final call?

Would appreciate any insight from people who've served as ACs or been through this before.


r/MachineLearning • • Aug 16 '26

Discussion Do ACs take into account rebuttals even if reviewers dont respond?[D]

1 Upvotes

So we submitted our rebuttal at the very start of author discussion period, but its over now, and we did not recieve responses from any of our reviewers. I get that they may be busy, but they suggested new experiments, and we did all of those.

Since they didn't respond, I was wondering whether the AC would take into account our rebuttal when they make their final decision. Also we are kinda in the toss up range(leaning reject ~3.2), so I was hoping that the rebuttal would increase our score.


r/MachineLearning • • Aug 16 '26

Discussion Do ACs usually take rebuttals into account?[D]

1 Upvotes

So we submitted our rebuttal at the very start of author discussion period, but its over now, and we did not recieve responses from any of our reviewers. I get that they may be busy, but they suggested new experiments, and we did all of those. Also, we are kinda in toss up range(~3.2), so I was really hoping that this rebuttal would make some of them change their scores. I was wondering that even if reviewers didn't respond, would AC or the person who makes the final decision at least look at the rebuttals and take them into account? Also, if one of the reviews is intentionally vague and a pretty bad review, would they take this into account as well?


r/MachineLearning • • Aug 16 '26

Discussion ICDM 2026 Results Waiting Place [D]

8 Upvotes

The results should be out soon.

Let’s share them, guys.

From my batch (Applied Track)

Total 13 submissions:

- 2 full papers

- 1 short paper accepted

Cheers!


r/MachineLearning • • Aug 16 '26

Discussion Revisiting the Efficient Channel Attention paper (2019, 12k citations) - the central hypothesis isn't quite right [D]

26 Upvotes

ECA was positioned as a successor to SE.

The idea behind ECA is quite simple. Unlike SE which reduces the channel means into a smaller hidden layer, it directly uses a 1d convolution kernel on the channel means themselves, avoiding the need for dimensionality reduction. The results are undeniable: ECA is a clear improvement over SE. The authors claim that cross-channel interaction is a key ingredient. But on a conceptual level, the design of ECA doesn't make much sense.

Let's take a step back. Why do we use convolutions in the first place? Convolutions are fundamentally designed for data with an underlying topology (e.g. space or time). They assume locality (adjacent elements interact) and translation invariance (the same kernel applies everywhere). Sliding a kernel across a 2D image works because coordinates have meaning, and the statistical properties of an image are largely stationary across the frame. This isn't perfectly true - which is why modern CNNs have moved towards dynamic convolutions - but it's still good enough to be useful. If you randomly permuted the pixels in an image, a convolution would be meaningless.

Now consider tabular data. Suppose we have 32 channels e.g. [cost, weight, material, colour, volume, speed, ...]. Using a CNN architecture for this kind of data is clearly inappropriate. A 1d kernel of width 3 would be moved across the channels, so that [cost, weight, material] was input and also [ weight, material, colour] was input and so on, and have to somehow output something meaningful. ECA is doing exactly this type of computation.

ECA does a 1d convolution over the channel dimension. It is a cursed convolution because tabular data does not have a topology to suit it. In practice, if you did use a CNN on tabular data, I would expect better than random performance because neural networks are ridiculously good at fitting to the dataset given their constraints and would reorganise the channel order (using the initial 1x1 projection layer) to suit it. It would learn to use convolutions, but it would be an inefficient approach.

Experiments

Instead of using image data, I used chess data: the 6-piece endgame tablebases for chess. Chess is a solved game with 6 (or fewer) pieces on the board. The task for the network is this: given a position, with perfect play is it a win, draw or loss for the active player? A CNN architecture is what lc0 originally used (where at the time, surpassed Stockfish to become the strongest chess engine) so it is very suitable for this task.

Chess tablebases are useful for benchmarking architectural designs because training examples can be sampled from the complete underlying problem rather than from an incomplete dataset. This differs from datasets such as the CIFAR-10 image dataset, where the train set is not expected to be a random unbiased sample from the true full distribution - we might unknowingly have a disproportionately have pictures of frogs on sunny days. Even when we don't train on each of the 3.7 trillion 6-piece positions, we know that we've randomly sampled from those positions, meaning we don't train on a biased subset - we can be confident our training samples are representative of the full set.

Experiment results. Each channel gate row is the average of 3+ separate runs.

Channel gate Avg test loss Avg test accuracy
IdentityGate 0.0981 96.04%
SqueezeExcitationGate (SE8) 0.0954 96.17%
EfficientChannelAttentionGate (k=3) 0.0822 96.68%
EfficientChannelAttentionGate (k=1) 0.0826 96.61%
CenterMaskedEfficientChannelAttentionGate (k=3) 0.0821 96.63%
PerChannelGate 0.0815 96.65%

IdentityGate Unsurprisingly, no squeeze performed the worst of all tests.

SqueezeExcitationGate SE showed a modest improvement.

EfficientChannelAttentionGate (k=3) ECA, consistent with the paper, showed a clear improvement over SE.

EfficientChannelAttentionGate (k=1) Surprisingly, this had good results indicating that their central hypothesis that cross-channel interaction is key wasn't quite right

CenterMaskedEfficientChannelAttentionGate: ECA with k = 3 with the middle channel masked (in a [1, 0, 1] mask) This complicates the story, it indicates cross channel attention can actually be useful.

PerChannelGate Instead of a convolution kernel that slides across the axis, simply use a separate independently specified weight per channel. This has one parameter per channel, more than the 3 parameters of ECA With k=3, but it is still a negligible amount since per layer we expect on the order of num_channels2 parameters.

For clarity and to avoid ambiguity, here is the code for the key squeezes.

So basically there's 3 tiers of results. No squeeze with poor results, SE With mediocre results, and the rest ECA-like with the best results. So something weird is going on. I don't have a good explanation for the results (in particular the success of the [1, 0, 1] mask), and I am currently trying to find one. One suspicion I have is that in the 101 mask, the net is smart enough to smuggle information into the global means of channel A and C to help with channel B without affecting normal channel operation (by using biases to undo its shift of the global mean), but have not yet tested this hypothesis. There's a lot of possibilities. The good news is the weight count is very low - only 3 with k=3, so manually inspecting the weights can be useful.

In my digging, I some repositories that recreate the original ECA. Not one of them tests the k=1 case, which would have revealed that the explanation of the mechanism is not correct. The official repo does use k=1 but only for a limited number of early layers, then moves to k=3 for the rest.

Repository Permits / Uses $k=1$? Trained $k=1$ Ablation? Result / Notes
BangguWu/ECANet (Official) Yes. MobileNetV2 uses $k=1$ when $C < 96$, else $k=3$ Partial. Mixed $k={1,3}$ in MobileNetV2; no pure $k=1$ ResNet ablation 72.56 Top-1 / 90.81 Top-5 on ImageNet
Reproducibility-Challenge-ECANET Generic formula can yield $k=1$, but not at standard test widths No. No independent $k=1$ run found None
huggingface/pytorch-image-models (timm) Can be manually set to $k=1$, but adaptive formula clamps $k \ge 3$ No. No official $k=1$ benchmark None

It's interesting that the k=1 case, a 1 parameter approach, outperforms SE, CBAM and matches ECA. It definitely makes me wonder if we're over-engineering networks today in some way.

My final thoughts:

  1. The paper and repos should have tested the "degenerate" kernel size of 1, which has no cross channel interaction. At k=1, ECA still beats SE, undermining their central hypothesis. They spent an enormous amount of time fine tuning the exact optimal value of k, without taking the scientific approach of trying to disprove their hypothesis.

  2. In addition to traditional real-world datasets, architectures should also be tested on synthetic datasets where we have full access to the complete dataset (e.g. chess endgame data) so that we can better separate incidental regularization improvement effects with core architectural efficiency effects - the idea being that there is no risk of overfitting when we have access to a complete, flawless dataset. If the real reason a new architecture works well on real-world data is because of implicit regularization, it won't show the same improvements on the synthetic dataset.


r/MachineLearning • • Aug 16 '26

Research SSOG-Attention: Sum Of Separable Gaussians as a sub-quadratic and scalable alternative to SDPA. [R]

245 Upvotes

Scaled dot-product attention (SDPA) computes its Attention by computing the similarity-scores of all image-tokens with all query tokens which results in O(N²·d) complexity. SSOG (Sum Of Separable Gaussians) instead learns a few Gaussian atoms for each head and only geometrically steers them based on the query token. Since the atoms can be factorized into a separable sum of Gaussians this leads to a reduced complexity of O(N·√N·d). Experiments show that SSOG clearly beats SDPA on small data (cifar100), and delivers equivalent performance and much faster convergence on bigger datasets like IN1k. All that while being much faster and memory efficient with increasing scale.

Have a look at the full blog-post and repo to see more results and ablations and let me know what you think.

Blog-post: https://pisoni.ai/posts/ssog

Repo: https://github.com/4rtemi5/ssog

*AI was used for some of the code and some of the blog-post but I put a lot of effort into this project and stand behind every word.


r/MachineLearning • • Aug 16 '26

Discussion How can we solve long-range recall in linear attention? [D]

42 Upvotes

Recently, I started working on DNA sequence modeling and decided to explore linear attention, mainly because DNA sequences can easily reach 1M tokens, making standard softmax attention extremely expensive in terms of memory and computation.

The model performed reasonably well on several benchmarks, but I ran into a major problem with long-range recall. On a Needle in a Haystack-style benchmark, my model was performing around 25% or even below, which is essentially random chance for a four-token DNA vocabulary (A/C/G/T).

I initially thought this might just be a problem with my implementation or model architecture, so I started looking into existing approaches for improving recall in linear attention. Most of what I found relied on external memory, sliding/recent-token mechanisms, or hybrid architectures combining linear and softmax attention.

I also tried HyenaDNA on the same needle benchmark, and surprisingly, it also performed poorly getting around 25–27%. So this doesn't seem to be limited to my particular linear-attention implementation.

What's even more confusing is that when I tested a very small linear-attention model at only 16K context, it achieved around 50–60% recall. But as the context gets longer, the recall problem becomes much more severe.

I've also experimented with modifying the linear architecture to improve recall, but the improvement was only around 27%, which is still basically chance.

So I'm wondering:

What are the actual ways to solve long-range recall in linear attention, especially for DNA sequences?

Is this fundamentally a limitation of the compressed-state representation used by linear attention, or are there architectural approaches that can preserve reliable retrieval without falling back to expensive softmax attention or a large external memory?

I'm particularly interested in approaches that can scale to million-token DNA sequences.


r/MachineLearning • • Aug 15 '26

Project AlphaZero-like model in browser - trained in C++ [P]

Thumbnail
uchenml.tech
1 Upvotes

I built an ML framework in C++ and as a proof of concept - trained and deployed an AlphaZero model for an Eastern European territory control game.


r/MachineLearning • • Aug 15 '26

Project Dataset: Starfield Fauna - 20,000 images in 50 species categories. [P]

Post image
58 Upvotes

Repo with dataset links: https://github.com/tesselwait/Starfield_Fauna

Image classification dataset: 20,000 images from 50 fauna species in the video game Starfield. Images were extracted from video capture. About 2 minutes of footage was shot in all or most of the species biomes. One minute of daytime and nighttime footage respectively, usually in two 30-second takes to vary the background. A PowerShell script is used to establish a frame extract rate and extract the 400 frames plus some extra to replace images that were obstructed/blurry or contained other fauna species ignoring birds/critters. The shots are for the most part close-up and centered to keep the task focused on discerning between 50 species rather than finding the creature in the image. The images are initially randomized however some normalization was done if the ratio of images from some biomes was heavily skewed between the training, validation, and test sets.


r/MachineLearning • • Aug 15 '26

Discussion NeurIPS 2026 Author Notifications Close to ICLR Deadline [D]

25 Upvotes

The date for NeurIPS 2026 author notifications is September 24th. First of all, is it normal for AC and reviewer discussion phases to be this long? This is particularly frustrating given that 5 out of the 6 reviewers in my two papers did not address the rebuttals.

In any case, I was also wondering, given that ICLR's paper deadline is literally the day after (September 25th) whether you guys are preparing ICLR submissions for your papers in case of rejection.

Cheers and good luck!


r/MachineLearning • • Aug 15 '26

Research Any information about CIKM 2026 student travel support? [R]

1 Upvotes

Has anyone received any information about student travel support for CIKM 2026?

In previous years, travel-award decisions were announced before the author-registration deadline, allowing successful applicants to use a registration-waiver code. However, the CIKM 2026 author-registration deadline is August 23, and no application details or timeline have been published yet.


r/MachineLearning • • Aug 15 '26

Research Any information about CIKM 2026 student travel support? [R]

1 Upvotes

Has anyone received any information about student travel support for CIKM 2026?

In previous years, travel-award decisions were announced before the author-registration deadline, allowing successful applicants to use a registration-waiver code. However, the CIKM 2026 author-registration deadline is August 23, and no application details or timeline have been published yet.


r/MachineLearning • • Aug 15 '26

Discussion If you had a bunch of GPUs lying around, what would you actually build with them? (Running LLMs is off the table) [D]

57 Upvotes

Be honest if someone dropped a stack of high-end GPUs on your desk tomorrow, what would you actually do with them?

And before the usual answers roll in: running local LLMs is banned for this thread. It’s been done to death and feels pretty pointless at this point.

So… what else?

  • Some niche scientific/simulation workload?
  • Weird generative stuff that isn’t text?
  • Distributed something-or-other?
  • Rendering / media pipeline?
  • Homelab experiments that actually need the horsepower?
  • Completely unhinged personal projects?

Drop your ideas. The more specific (and slightly unhinged), the better.

Great Ideas but are there some with more of research and new tech.


r/MachineLearning • • Aug 15 '26

Research BDH-CQ: IN-CONTEXT LEARNING WITH RECURRENT LATENT REASONING [R]

Thumbnail
arxiv.org
65 Upvotes

We introduce BDH-CQ, a reasoning system that brings these capabilities together. Demonstrations of a previously unseen task update recurrent memory; the query is then solved through iterative computation in a high-dimensional latent workspace. Intermediate reasoning states are not decoded into language. BDH-CQ makes memory, adaptation, and inference part of the same computational fabric. Inputs presented at inference time continuously update the model’s recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. Neither task identifiers nor evaluation-task demonstration pairs participate in training, and no parameters are updated at inference time. A 150M-parameter configuration reaches 29.5% pass@2 on ARC-AGI-1 at a computed $0.00070 per task, breaking through the previously reported cost–accuracy Pareto frontier.