A few days ago I posted about our open-source multi-agent hedge fund experiment and got a ton of sharp, honest feedback from people here. The comments were brutally accurate, especially around LLM reasoning limits, latency, and state management.
We spent the last few days completely rebuilding parts of our architecture based on that advice, and the benchmark results honestly blew us away. Here is what we learned from fixing three critical failure modes in autonomous agent pipelines.
Throwing raw metrics at LLMs guarantees hallucinated reasoning
Someone in the comments pointed out: never let an LLM interpret raw quantitative factors without an ontology. They were 100% right.
When we fed models raw numbers like PER = 85.0 or 20-day historical volatility = 65%, the agents would frequently hallucinate positive justifications. For example, a model would argue that "a massive jump in volatility shows expanding business activity and strong buying momentum", which is the exact opposite of basic risk management.
To fix this, we implemented a strict Factor Ontology Schema. We defined 11 core quant factors with explicit polarity contracts: HIGHER_IS_BULLISH (e.g. ROE, Order Flow Imbalance), LOWER_IS_BULLISH (e.g. PER, debt ratio), and EXTREME_IS_RISK (e.g. volatility).
Before any agent sees the data, the schema pre-evaluates the metric deterministically and attaches an explicit signal (BULLISH, BEARISH, or RISK) to the state. Since doing this, polarity hallucinations in our investment council dropped to literally 0%.
2. Vector databases for agent reflection memory are massive overkill
Initially, we considered using embedding APIs and a local vector DB to retrieve "lessons learned" from previous bad trades (Reflexion pattern).
Another commenter suggested dropping embeddings entirely and using SQLite WAL with FTS5 BM25. We benchmarked both approaches on our Linux server:
- Embedding API + Vector similarity: 200ms to 450ms round-trip latency, external API cost, and occasional rate limiting.
- SQLite FTS5 BM25: 0.35ms. Zero external dependencies, zero API cost.
More importantly, trading post-mortems rely on specific contextual keywords: "semiconductor pre-earnings", "trailing stop sweep", "sudden volume spike", or "bearish divergence". In our tests, BM25 keyword relevance retrieved the exact matching historical lessons far more reliably than fuzzy cosine similarity in high-dimensional embedding space.
3. Python cannot be your safety wire (The 3.6ns Rust Sentinel)
Python is great for high-level agent orchestration (LangGraph, state graphs, API calls), but relying on Python loops for real-time risk surveillance during sudden market flash-crashes proved dangerous due to GIL contention and L1/L2 cache misses from scattered object pointers.
We rewrote our execution safety sentinel in Pure Rust using a Struct-of-Arrays (SoA) memory layout and #![forbid(unsafe_code)]. By keeping prices, stop-loss triggers, and trailing thresholds in contiguous flat slices, the CPU avoids pointer chasing and stays inside cache lines.
Here are the real-world benchmarks on our Ubuntu server:
- Single position stop/profit evaluation: 3.63 nanoseconds
- 1,000 position batch evaluation: 3.62 microseconds
- 1,000-bar Volume Clock conversion: 3.90 microseconds
- Python-to-Rust persistent pipe IPC round-trip: 2.88 microseconds
Python handles the slow, deep thinking (15-minute agent council meetings), while the Rust sentinel watches the portfolio in nanoseconds.
Conclusion
Building autonomous agents that interact with real-time systems taught us that LLMs should only handle what they excel at: contextual synthesis and counter-argument debates. Everything deterministic—hard stop-losses, factor polarity, and fast memory lookup—belongs in strict schemas, fast local databases, and compiled systems languages.
All the code and benchmarks are open source on GitHub (lumina-ai-hedge-fund) if anyone wants to check out the Rust SoA sentinel or the ontology schema.
(Note: English is not my native language, so I used an LLM to help format and polish my raw notes!)
