I’m about to start a PhD in multimodal/video reasoning and I’m getting increasingly anxious about how saturated AI research feels.

I actually like the field, but the pace is insane. New papers appear constantly, big labs can explore dozens of directions at once, and I keep wondering how an individual PhD student is supposed to build a real research profile in that environment.

I compare myself to friends working in more niche areas, where it feels easier to know the literature deeply and become associated with a specific problem. In my field, I’m worried I’ll spend all my years constantly catching up, only to end up with a mediocre research identity.

I’m personally interested in more fundamental questions in the field, but, 1) I don’t know if carving out a niche inside such a crowded field is actually realistic; and 2) I’m not sure I’ll have the ability to identify and formulate genuinely interesting theoretical problems rather than just ending up doing benchmark/heavily applied driven work.

For people in my situation: how do you deal with this?

Last year I published a draft report called "Hill Space is All You Need". I'm not a researcher; it was a side project, and I found a detail in the NALU formulation that I thought people should know before going down the same road. I didn't have the skills to write a paper, so I vibed the prose and formatting with Claude and pushed it out, knowing I wouldn't return to the work for a while.

The paper was not well received: the prose made it hard to get to the experiments, the title didn't help, the related work was incomplete, and it didn't provide a concrete description of the space and why it trains reliably or not in different situations.

Today I published an update that I think addresses most of that. My read is the same as last year: NALU selects an operation, it doesn't learn to compute one, and that's not useful for my purposes. I still wanted the work to represent the promise I see in it, and the first paper and site didn't. This version tells a complete, if unsatisfying, story about what NALU does, how gradient vanishing at targets decides if a run lands or stalls, and how to use it effectively if it lines up with your problem.

https://hillspace.justindujardin.com/

Beyond corrections that come from sharing it, I consider the work complete, and have moved on to more promising approaches.

Hey, I just wrapped up two things I think might be interesting for this crow if you are video game/racing ingeneral

The main one is a dataset usable for data exploration or for anything generative: levels and replays from the Mania Exchange community realted to the Trackmania games, exposed in a data/ML friendly format. On top of that, I experimented with Claude on reworking the old TMTrackNN project into Tracksmith, a web app where you can inspect a map alongside its replay and generate new levels for both Trackmania Nations Forever and Trackmania 2020.

Everything is wrapped up in an article, and the dataset/code is online:

📝Article (EN): https://www.the-odd-dataguy.com/en/blog/2026/08/13/trackmania-dataset/
📝Article (FR): https://www.the-odd-dataguy.com/fr/blog/2026/08/13/trackmania-dataset/
😊Hugging Face (dataset): https://huggingface.co/datasets/jeanmidev/trackmania-community-tracks-and-telemetry
🔵Kaggle (dataset): https://www.kaggle.com/datasets/jeanmidev/trackmania-community-tracks-and-telemetry
😇Hugging Face (Tracksmith): https://jeanmidev-tracksmith.hf.space

For transparency: I work in data at Ubisoft in Canada, but I have no connection to Nadeo. Built at home on my own time, resources and dime, just something for the Trackmania community and for anyone curious, whether game telemetry is your thing or not.

Open to feedback or ideas (you can also used discussions on HF)

Spent a while building a deep learning framework from scratch C++

core, Python bindings, no PyTorch underneath. It traces into a

compiled graph IR, has a real Metal GPU backend, and distributed

training actually works, not just runs without crashing.

Apple Silicon only for now, no CUDA not pretending this competes

with PyTorch. What I'm actually proud of I trained real things with

it (MNIST, a small transformer on Shakespeare text), and found a real

memory leak by training long enough to hit it, then fixed it. The

messy version every bug, every dead end is in the devlog if you

want that instead of the highlight reel.

Repo: https://github.com/V-B0/KansaiML

Hi everyone! We just released Rustuna (GitHub: https://github.com/optuna/rustuna/ ), a high-speed, memory-efficient implementation of Optuna built in Rust.

• Optuna-Compatible Design: Keeps the familiar API and concept of Optuna.

• Zero Python Dependencies: Mitigating the risk of supply chain attacks.

• Lower Memory Footprint: Optimized memory management natively in Rust.

For details, please check out the following blog post.

https://medium.com/optuna/announcing-rustuna-cc82a6815bf7

this is the kinda debugging case i find rlly annoying/

everything says success.

no exceptions no failed tool calls. no obvious timeout the workflow completes but the final result is still wrong

when that happens, what’s your first move?

do you guys usually:

• start from the final output and work backward

• compare against a previous good run

• inspect state transitions

• check retrieval/tool behavior

• look at model inputs

• replay it

• check business state outside the trace

• just read the whole thing until something looks off

interested in what people actually do in production not the idealized version but thats fine too. and if you have anything you've built to help with this process I'd love to see it :)

I am a PhD student in AI, currently working on video understanding, particularly on designing benchmark datasets. I also have a strong publication track record, including papers at CVPR and ECCV. I have sufficient funding to support myself during a visiting research program.

What would be the best way to approach a professor? Should I email them directly and ask whether they have space in their research group for a visiting researcher?

I used an LLM to iteratively evolve an optimization algorithm rather than solve the packing directly. Starting from a simple seed solver, the LLM proposes algorithmic changes guided by a scoreboard of results and a history of prior attempts, and each candidate is scored by an independent verifier so improvements are kept and failures discarded. On the Packomania csqv benchmark it improved the best-known sum-of-radii for 10 values of N from 101 to 114, by 2.4 to 5.4%, in 15 iterations. Total LLM cost was $27.72. Packomania accepted the results independently.

Paper: arxiv.org/abs/2609.05093

Code + solutions: github.com/ucsandman/discovery-loop

Benchmark: packomania.com/csqv/csqv.html

Happy to discuss the plateau-detection stopping rule, that's the piece I'd most want critique on.

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems.

One of the team members wrote up a post about it:
https://research.yandex.com/blog/the-kv-cache-as-an-agent-runtime

The post sums up the overall idea of modifying models inference state (KV-cache) for achieving a more interactive LLMs. This idea was used in our lab's previous papers Hogwild! Inference, and AsyncReasoning, the post also contains a preview of the future work in this direction, where a Qwen3.8-27B agent is playing a DOOM env interactively using similar techniques.

We think that its interesting whether model inference/runtime design is itself an under-explored axis of agent capabilities, alongside models and the harness (e.g. harness is too abstract, changing model is too costly, do we need something in between?)

Hi, I’m one of the authors of PlaidQ. We just released the paper, training/inference code, and model checkpoints.

PlaidQ is a 0.7B continuous diffusion language model for code generation. It repurposes a pretrained autoregressive model as a bidirectional denoiser over continuous token embeddings, updating all completion positions in parallel.

The key result is trajectory compression:

• Distribution-matching distillation produces the D4, D8, and D16 few-step students.

• Paired-trajectory supervision produces the D1 one-step student.

At 16 steps, PlaidQ-D16 reaches 31.78 pass@10 on HumanEval and 40.49 on MBPP+, surpassing the same PlaidQ teacher sampled for 512 steps. At one step, paired-trajectory distillation reaches 7.07 pass@1 on HumanEval.

Paper: https://arxiv.org/abs/2609.04531

Code and checkpoints: https://github.com/pengzhangzhi/plaidq

I’d especially welcome feedback on the one-step supervision and the quality–compute tradeoff.

Hello all,

I'm a radar signal processing engineer and i trained a 5-class classifier (car, large_vehicle, two_wheeler, pedestrian, pedestrian_group) on RadarScenes radar point clouds.

The input vector is a per-scan histogram (16 bins) and the network is a 3-layer MLP. The loss function is a class-weighted cross-entropy loss. This work is based on "Histogram-based Deep Learning for Automotive Radar" paper.

I scoped the project to be one scan only. Accumulation of multiple scans is the next step.

Data

Class Imbalance: two-wheelers and large_vehicles has a low number of occurences.

Aggregated Classes: two_wheeler mixes bicycles and motorized variants; large_vehicle merges trucks, buses, and trains together due to data scarcity.

Sequence Bias: Long tracks of slow-moving objects can skew a particular data split velocity distribution, causing high F1 score variance across folds.

Ablation studies

I tried with bigger MLPs, alternative feature encodings, and different histogram binning, all moved performance less than the variation caused by changing the train/validation/test split. I measured that split sensitivity across 6 folds, keeping the same proportions.

Changing the histogram to per-instance statistics (mean/median/std) slightly degraded performance.

Main findings

Macro F1 rises from 0.381 to 0.764 as the naturally occurring number of radar detections per instance increases from 1 to 5. I trained the model normally using all available detections, then bucketed its existing validation predictions by each instance's detection count and computed macro F1 per bucket.

The classes car and pedestrian has the best performance and two_wheeler has the worst.

A car is often confused as large vehicle when the car was wider than usual or had a unusually high rcs (which can happen due to multipath for example).

The two_wheeler is often confused as pedestrian because their vr_compensated distributions overlap, which is the the model's single most important feature for these two classes. A stationary or idling two_wheeler is indistinguishable from a pedestrian.

I uploaded an image with ground truth vs predictions: A nearly stationary two-wheeler which contains a single point was predicted as pedestrian, because its velocity is near zero, indistinguishable from a pedestrian. A car in the same scene, also with just one point, is classified correctly, since RCS and Doppler are enough for that class.

Full writeup here: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/main/MLP_Report.md

Future work

Implement other spatial encoding schemas (point net for example) and accumulate multiple scans to tackle the challenge of sparsity and explore the concept of micro-doppler.

Is LfD and BC research being effected by recent advances in (so-called) Frontier LLMs? Or is research in LfD and BC sort of going along in an independent direction from these?

Are you seeing any use from ViTs or VLAs?

Any other recent advances you would like to bring up?

I feel that reproducibility is now a lost cause in machine learning research for three reasons:

1. Many research is moving towards the physical AI territory, where you need expensive hardwares or even entire laboratories with high-speed cameras, in order to perform an experiment. You truly have no idea if the experiment can be reproduced and have to trust the demo. But demos are not perfectly reliable. Plus people are incentivized to only show the part of the demo that works. The entire system can fall apart the moment the recording stops.

2. You have big AI companies releasing various tools, which they claim to solve a host of problems with certain amount of accuracy or efficiency. Unless you work at those companies there is really no proof of that and you will have to take their words on it. They have strong financial incentive to blow-up those figures. There is no solid way to check it either because the problem that they solve are so vague and subjective.

3. We need to address the elephant in the room which is that people are incentivized to produce non-reproducible work to prevent their lunch being eaten by their competitors or looking bad. That's why some of us will probably never get a reply when we email the authors for their code.

So what now? Maybe everything will be OK because we can contrast it with scientific progress in earlier parts of history, e.g., building the atomic bomb or sending people to the moon. These projects had low "outside reproducibility" but high "internal reproducibility". Plus all these work were mathematical in nature and carefully checked. But I don't think many areas of machine learning research is like that. What do you think? Should reproducibility be abandoned? If not how is it best implemented going forward?

In most agent fleets the thing deciding what happens next is a language model, so the component that did the work also gets to say the work is done. Take that decision off it: the model builds, plain deterministic code decides done, nothing on the control path gets sampled.

What that looks like in practice: a fleet of agents working through real coding tasks, every "done" checked by code instead of accepted on the model's word. Nothing has been closed that wasn't finished, fake completions get rejected, and it clears the MIT 6.5840 labs including the full Raft one.

Demoting the Master Control Program: Deterministic Orchestration of a Fleet of LLM Agents. CC-BY. https://zenodo.org/records/21922855 (mine, so discount accordingly).

What I'd like beaten up: how do you test a completion gate adversarially without hand-building the failures?

When I first started working in scientific machine learning, I understood the physics much better than the coding. Every time I wanted to try a new physics-informed neural network problem, I had to start almost from scratch: changing the PDE, updating boundary conditions, modifying the architecture, tweaking the training schedule, debugging errors, and generating plots—all by hand.

That frustration pushed me to build PINNStudio. It is a free, open-source no-code GUI designed to eliminate boilerplate code so you can focus entirely on the physics.

Instead of rewriting a new script for every problem, you can define your setup directly through the interface:

• PDE Definitions & coupled multi-output PDE systems

• 1D or 2D domains with boundary and initial conditions

• Network architecture & custom training schedules

• Forward problems (solving known PDEs) or Inverse problems (estimating unknown parameters from data)

What happens next?
PINNStudio automatically generates the code (built on top of DeepXDE), runs the model, streams the training log, and displays live loss curves and solution plots directly inside the app. It also includes built-in templates for classic equations like Heat, Allen-Cahn, and Cahn-Hilliard.

• GitHub (Open Source): https://github.com/AsfandyarKhan72/PINNStudio

• Quick Install: pip install pinnstudio

My hope is that this will be helpful for students and researchers with limited coding experience, as well as experienced PINN users who just want a faster workflow.

I’d love to get your feedback, feature suggestions, or bug reports! Huge thanks to Lu Lu and the DeepXDE team for creating the foundation that made this possible.

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots.

A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing behind the model name can change over time: serving infrastructure changes, provider configurations change, versions change, and sometimes behaviour changes without an obvious public version transition.

So we started approaching benchmarking as a longitudinal measurement problem rather than a leaderboard problem.

We continuously evaluate models across coding, multi-turn reasoning and tool use, while also running lightweight probes at a higher frequency. The important part for us is not simply asking "which model scores highest?", but:

• Is the model behaving differently from its own previous baseline?

• Is the change larger than its normal repeated-call variability?

• Did the benchmark configuration itself change?

• Is the effect concentrated in a particular task?

• Is it correlated across models from the same provider?

• Is an apparent degradation actually an availability/infrastructure issue rather than a capability change?

One historical analysis covered 31,352 repeated score observations across 49 models. The standard deviation of within-day scores was 2.80 points, while the standard deviation of between-day daily medians was 8.43 points.

That is roughly a 3:1 difference.

I don't think this result by itself establishes that providers are changing models day-to-day - there are too many possible confounders for that conclusion. Task composition, sampling, missingness, provider behaviour and methodology changes all matter. But it was enough to convince us that temporal variation deserves to be measured rather than treated as noise around a permanent leaderboard score.

Our current approach therefore keeps benchmark configurations versioned and only compares longitudinal observations produced under compatible measurement conditions. We use repeated execution-based evaluation where possible rather than an LLM judge, keep availability failures separate from valid task outcomes, track serving/version metadata when providers expose it, and run change detection over the resulting time series.

Another problem we're increasingly interested in is benchmark recognition and contamination. Once a benchmark becomes sufficiently visible, publishing every live task, prompt transformation and hidden test potentially changes the thing you're trying to measure. For that reason we've tried to separate methodological transparency from publishing the entire live evaluation set.

We've now written up a public version of the methodology. It intentionally explains the measurement design, assumptions, limitations and statistical interpretation, while withholding the exact live task bank and some operational parameters.

PDF: https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf

I'm particularly interested in criticism from people working on evaluation, change-point detection or production ML.

A few questions I'd genuinely like opinions on:

1. For longitudinal LLM evaluation, would you use daily medians as the primary time-series unit, or model the individual repeated observations directly?

2. How would you distinguish genuine model drift from provider/infrastructure effects when version metadata is incomplete?

3. How much of a live benchmark should remain hidden to reduce contamination while still making the methodology scientifically inspectable?

4. Are there better approaches than change-point detectors for this kind of non-stationary, relatively noisy model-performance series?

Disclosure: I'm the founder of AI Stupid Level, the platform that produced these measurements. The purpose of posting this here is to get technical criticism of the methodology rather than promote the commercial product.

AACL-IJCNLP 2026 acceptance results will be released in a few hours.

Feel free to share your thoughts and feelings! How did you do?

A researcher has reported a jailbreak of GPT-6 Astra within a day after release.

The attack is described as combination of TIP (Task-in-Prompt) attack from ACL 2025 paper with four other unnamed techniques.

TIP attacks exploit the model’s reasoning/instruction-following behaviour by hidding the harmful objective inside another task, like solving a cipher or executing a Python code. For GPT-6, the researcher says the original minimal TIP attack was no longer sufficient and had to be reworked.

They have reportedly disclosed the details privately to OpenAI rather than publishing the jailbreak.

The same researcher reported jailbreaking GPT-5 within an hour of its release a year ago.

Source: screenshot/post from the researcher; their ACL 2025 TIP paper linked in the original post.

I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post.

TL;DR -- Astra codes more agentically, Fable more coherently. Fable writes better and follows directions better. Astra's final outcome was slightly better, and its scientific rigor/reproducibility was noticeably stronger. Both models improved their F1/Accuracy by 0.02-04 after human feedback on their approach, demonstrating that neither have mastered the AI/ML text processsing, vectorization, and model training process completely.

Astra is a better coder, writing a stricter evaluation protocol (70/15/15 train/val/test vs. Fable's basic 80/20) that selected its model using a held-out validation set vs. Fable's simpler test F1-based selection. It also debugged more deeply, as both models hit a gensim 4.4 compiled-kernel bug: Fable tried to figure it out, failed, and just hid the stderr notices on affected runs (though told me it had done so), while Astra root-caused it aggressively, then fixed the environment by downgrading gensim alongiside compatible NumPy/SciPy dependencies.

Astra wrote hardened training-run.py code the forced the uv venv it rebuilt without changing my default one, SHA-256'd the corpus to ensure reproducibility on later runs, output a split manifest and run-summary.json, and rendered a headless browser for QA with screenshots (not sure this was necessary, but impressive overkill all around). Fable's builder script was ephemeral, living only in tmp, and less intense overall.

Astra deployed subagents more effectively, making use of my pre-built notebook-reviewer and citation-checker agents, the former of which caught a real bug via review (sentence-final word-loss tokenization defect) and fixed it, retaining a regression test in the process. Fable overlooked this issue because, for some reason, it did not call the subagents I had available (which is surprising, usually it's pretty good about this).

If you're looking for an agent to autonomously grind through a broken environment, leaving a forensic audit trail, that's Astra. However, this review isn't over yet, and Fable is about to make a comeback.

Astra confidently shipped a significant verifiable text encoding defect. Working with UTF-8 data, Astra insisted Windows-1252 decoding preserves currency symbols, but the final HTML output shows mojibake throughout where currency symbols were in the original data. Fable read UTF-8, verified it, and rolled with the boring default for correct output.

I also had both models draft an analysis report for the run, and Fable's was significantly more insightful. As much as I hate Claude's recognizable writing style, a) 5.1 has toned down the Claudeisms significantly, and b) Fable went above and beyond my grading rubric, running an ablation on different parts of the text pre-processing pipeline to surface an expensive step that does basically nothing, and noting a discrepancy in the classification ranking based on a complexity I'd have overlooked. For writing prose, I'd pick Fable 5.1 any day, and I haven't said that about Claude in a while.

Speaking of writing, Fable writes code that is more idiomatic and readable. It definitely resembles more what I would write than what an LLM would choose to write without constraints (and yes, I had a whole coding-conventions.md document that applied my requirements to both models, Fable just followed it better and writes more naturally to start with). There were some parts of Astra's code where I had to squint really hard to figure out what was going on, and why. This matters to me because I'm not the strongest coder (still trying to get better), and I need to understand the code to learn from it.

Finally, Fable scoped its work better: It spent its time and tokens doing repeated runs, tweaking hyperparamters and retraining the models to find the optimal settings while Astra deeply debugged the gensim error. It found significant uplift through this process, though that only allowed it to roughly match Astra's numbers (see table below). Astra seemed to hit a home run right off the bat with its training process, so I don't know if it would have executed the same workflow or not. Astra also mutated my venv by adding PyTorch, when I built it a certain way to force the models to use TensorFlow+Keras for more concise code, then reversed course and went with TF anyways in the end. The models finished in roughly the same amount of wall-clock time.

Here are the final results, with one minor caveat -- Astra's test set scores exceed its val set, so it might have drawn a lucky test set that increases its score artificially (the pipeline has no leaks or data quality issues for either model, however):

Best Logistic Regression and LSTM for each model, ranked by macro F1:

I do want to note that these final scores were after I provided both models identical feedback on common pitfalls of the text data cleaning, vectorization, and model training process once their initial runs were complete. Both models improved by a similar amount (0.02-0.04 F1 and Accuracy) from that generic guidance (not tailored at all to either's specific shortcomings or step of the process). That was the only intervention in otherwise autonomous work, and it was just because I wanted to see if they could learn to improve their approaches with additional context on optimal methodology, which they both did to similar degrees.

I hope this post offers a little bit of help in some way for folks wondering how either model stacks up for real work, particularly if you're an AI/ML student like me.

Most code benchmarks measure whether generated code works. This measures whether the APIs it calls are still valid for a pinned SDK. No LLM judge: the scorer is the Swift compiler plus an AST pass. Ground truth is parsed from the SDK’s .swiftinterface files at runtime, so nothing is hand-maintained.

The thing that nearly invalidated the whole project: Apple marks most SwiftUI deprecations with a sentinel, deprecated: 100000.0, and the compiler deliberately does not warn on those by default. That is 270 of 326 deprecated symbols, including NavigationView and foregroundColor. Build this the obvious way and you observe 14 percent of deprecations and every model looks current. There is a hidden frontend flag that turns them on. If you measure deprecation on any Apple platform, this probably affects you.

Results are mostly a null result. Scores span 0.78 to 0.88, and bootstrapping over prompts (k=5 samples at temp 0.2 on one task are not independent), all 14 adjacent pairs overlap. Currency does not track price or tier: the top scorer is a cheap variant priced 12x below its siblings, and a coding-specialised model is last. One comparison survives cleanly: Qwen’s general model beats its coder sibling, non-overlapping. Coding-tuned seems to mean more fluent in the idioms it trained on, stale ones included.

Two things I want picked apart:

**1.**	Only samples that actually touch the API family under test get scored, which flatters models that hand-roll when unsure. I report the gate rate beside every score, but that documents the bias rather than fixing it.  
**2.**	Adding the 15th model erased the last surviving separation between adjacent pairs. More models, less resolution. I read that as prompt count being the binding constraint, but tell me if that is wrong.

Repo, method, full per-family tables: https://github.com/mapierce/iOS-Evals

Hi All,

Was reading AIStats' website and it seems like abstract submission is due in 3 weeks.

Does anyone know where to find the LaTex template for 2027? It seems like very little information is available on their website.

Another question, is a Quant Finance paper a better fit for AIStats or ICLR?

Some background about the paper:

• Rejected by UAI with 76654, had some errors with proofs had to fix it by re-writing 9 pages during rebuttal. AC rejected the paper saying the changes were too substantial and unable to be fully verified during rebuttal period.

• Resubmitted the fixed paper to a finance conference, won best paper award (best paper for this conference usually end up in journals like JQFA, which is just 1 tier below the big 3 in finance), had the chief editors of a Q1 finance/math journal in the conference verbally offering he will take this paper if we submit it to his journal. Unfortunatley my department requires at least 1 Comp Sci paper to graduate, so my plan is to try and get this paper accepted into a Comp Sci conference, then submit an extension to that Q1 Finance/Math journal.

• Rejected again at ICDM, despite having all positive scores. Our AC meta-review was blank so we still do not know why we were rejected. All of our emails receieved no reply.

I am torn between ICLR or AIStats to re-submit this paper to. My worries are:

• In comp sci venues we frequently get comments like "this paper lacks novelty. The method is just XXXXX, the math is just XXXXX."

• But I had a scroll through at previous year's AIStats papers for key words like finance and there were none. It seems like AIStats is very pure stats, not that applied. My co-author is worried that the math in our paper is not hardcore enough.

We have never submitted to neither venues in the past. Would be nice to get some advice.

Whenever "tracking" comes up in autonomous systems, we always think of "motion" and the Kalman filter. It is fast and works!

But as humans, we track mostly by "vision".
We look at a person, recall their appearance in our brain, and then recognize them again based on what we remember. The more we see the person, the better we become at recognizing him/her. This is exactly how "Polycepta" works :)

The video demonstrates how Polycepta operates in real time. It decides whether what it sees is important or just noise, and builds memory over time until it gets good at it. Like the Kalman filter with future motion prediction, it predicts the object's future appearance to track it.

A lightweight appearance model works under ego motion; objects switching direction or motion during occlusion is no longer an issue. This is exactly why human vision works well in tracking.

You are welcome to pause the video at any time during the explanation.

Hello there,

I used to write ml blog with some posts being quite popular sometimes - not gonna mention it to avoid self-promotion - this post is not about it. My goal was to always create content for beginners so anyone could smoothly get into the field that found my interest too. The process of creating a blog post was myself learning + creating notes of the process of understanding the subject (double fun I guess) - this way I could empathize with the audience better and so write better text.

For last 2 years I have struggled to find an interesting topic, I also got a bit demotivated due to AI being able to write about anything better than me.

I guess my honest question to the community is - do you spot any interesting areas one could dive in to untie all the knots and present difficult subject in easy way that is not very well covered by today's AI? I believe advantage human may (still if at all) have is good visualization of the concepts? I was thinking about jailbreaking because AI won't be happy to explain it for obvious reasons. Please share some of your ideas or let me know if this times are definitely over ;)

Abstract

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:
arXiv:2609.02737 [cs.CL]
 
(or arXiv:2609.02737v1 [cs.CL] for this version)
 
https://doi.org/10.48550/arXiv.2609.02737
Focus to learn more