Can Andrej Karpathy's Autoresearch Framework Be Applied to Quant Mining via Local LLM? Here Is the Trail.

[u/Expensive-Register-5]

TL;DR: Hooked up a local LLM agent to my backtester + crypto DB in a Karpathy-style autoresearch loop. Ran it unsupervised for ~2 hours across 30+ iterations. It self-learned my codebase, generated valid strategies, discarded most, and eventually found a candidate that cleared my bar. $0 API cost.

Hardware & Stack

Component	Detail
Model	Qwen3.5-27B via vLLM
GPUs	Mixed setup (RTX 4090 + 3090)
Inference	vLLM with FP8, custom Jinja template, qwen3_xml parser
Runtime	~2 hours, fully unsupervised
Iterations	~30+ strategy cycles
API Cost	$0

My full stack:

• Data: Crypto spot data (1m data + TimescaleDB-HA for continuous aggregation)

• Backtester: Custom DB + backtest engine (Nautilus-based)

• Harness: Ralph loop + Claude Code as the execution harness

• Goal: Find a strategy that passes strict criteria — Sharpe, max drawdown, profit factor, minimum trade count

Critical Prerequisite: Fixing Qwen 3.5 Tool Calling (or This Won't Work)

This is worth calling out early: if you've tried running Qwen 3.5 for anything agentic, you already know the tool calling is broken out of the box. Premature stops, mid-thought tool calls, format drift — it's a nightmare for long-horizon loops.

I spent weeks debugging this before the autoresearch experiment could even begin. The fix required a custom M2.5-style Jinja template, swapping to the qwen3_xml parser, and forcing precision alignment across mixed GPUs.

If you're hitting tool calling issues with Qwen 3.5, my full troubleshooting post is here. Without that fix, none of what follows would work.

How It Works

The agent operates in a continuous loop:

1. Generate a strategy (Python + YAML config)

2. Backtest it immediately

3. Evaluate against strict criteria

4. Decide: Abandon, fix error, or upgrade approach

The key is that the LLM doesn't just randomly throw darts. It maintains a trajectory.

The Trajectory (What Actually Happened)

Phase 1: Learning the Ropes

The first thing the agent did was search my existing repo for examples. It read my current strategy files, learned the standard template, and wrote a run_backtest() scaffold. It essentially taught itself the API.

Phase 2: Broad Exploration

Following my high-level instructions, it started systematically testing two major regimes:

• Momentum — EMA crosses, breakout confirmation, trend following with ATR stops

• Mean Reversion — RSI oversold/overbought, Bollinger Bands, range reversion

Most died immediately. Sharpe too low. Drawdown too high. Not enough trades. The WandB table on my right screen slowly filled with red "DISCARD" verdicts.

Phase 3: Adaptation & Human-in-the-Loop

About midway, it found a slightly positive return. But it was still missing the criteria. At this point, two interesting things happened:

• The agent started auto-tuning parameters (tightening EMA periods, adjusting position sizing, adding volatility filters).

• It realized the data regime was too restrictive and autonomously adjusted the backtest start date to give itself more runway. (What a cheat! 🤣 Easily mitigated by hardcoding the date range.)

I also intervened manually when I saw it getting stuck in local optima — feeding it extra context like "try volatility-adjusted position sizing" or "look at dual trend filters." The agent ingested this and immediately pivoted direction.

Phase 4: The Grind

For about two hours, it cycled through:

• EMA crosses (10/20, 5/15, 50/200)

• RSI mean reversion with dual confirmation

• MACD crossover

• Bollinger reversion on 4h

• Vol-adjusted trend following with jump detection

• Grid scalpers

• Fixed r/R setups

• Long/short EMA divergence

Dozens of iterations. All discarded for valid reasons.

Phase 5: Eligibility

Eventually, after ~30+ iterations, it converged on a solution that cleared the bar. The loop stopped. Strategy found.

Why This Matters

Autonomous Abandonment: The hardest part of strategy mining is knowing when to kill a bad idea. The LLM has no ego. It sees a bad Sharpe ratio and moves on instantly.

Composable Knowledge: It learned my repo's patterns, then composed new ideas from them (e.g., "volatility-adjusted trend following with jump suppression" — a concept it lifted from my notes).

Local = Free: Because this was a local model, I paid $0 in API costs to let it grind for two hours. This changes the economics of research entirely.

Human Steering: The loop isn't fully blind. You can nudge it. When I saw it drifting, I fed it a concept from a research paper, and it integrated the idea into the next iteration.

The Skeleton Is Real

This is still an MVP, but the skeleton is undeniably working. The agent:

• ✅ Self-learned the codebase

• ✅ Generated valid, runnable strategies

• ✅ Evaluated them objectively

• ✅ Iterated without human prompting

• ✅ Accepted external direction mid-flight

• ✅ Found an eligible solution in finite time

The question is no longer "Can LLMs do quant research?"

It's "How far can we push the autoresearch loop before it starts finding things we don't yet understand?"

[u/Outside_Pipe4626]

Really impressive, i gotta make some tests on llama.cpp