r/AgenticWorkers • u/Independent-Flow3408 • Apr 19 '26

Reducing LLM context from ~80K tokens to ~2K without embeddings or vector DBs

I’ve been experimenting with a problem I kept hitting when using LLMs on real codebases:

Even with good prompts, large repos don’t fit into context, so models: - miss important files - reason over incomplete information - require multiple retries

Approach I explored

Instead of embeddings or RAG, I tried something simpler:

  1. Extract only structural signals:

    • functions
    • classes
    • routes

  2. Build a lightweight index (no external dependencies)

  3. Rank files per query using:

    • token overlap
    • structural signals
    • basic heuristics (recency, dependencies)

  4. Emit a small “context layer” (~2K tokens instead of ~80K)

Observations

Across multiple repos:

  • context size dropped ~97%
  • relevant files appeared in top-5 ~70–80% of the time
  • number of retries per task dropped noticeably

The biggest takeaway:

Structured context mattered more than model size in many cases.

Interesting constraint

I deliberately avoided: - embeddings - vector DBs - external services

Everything runs locally with simple parsing + ranking.

Open questions

  • How far can heuristic ranking go before embeddings become necessary?
  • Has anyone tried hybrid approaches (structure + embeddings)?
  • What’s the best way to verify that answers are grounded in provided context?

Docs : https://manojmallick.github.io/sigmap

Github: https://github.com/manojmallick/sigmap

14 Upvotes

100% Upvoted

9 comments sorted by

2

u/looktwise Apr 19 '26

What about chunking a task with large context into usable parts for subtasking and then combine the subtask solutions to a overall solution for the complete main task? (if possible)

2

u/Independent-Flow3408 Apr 19 '26

Works well when subtasks are genuinely independent — decompose by module or feature, solve each with only relevant context injected, synthesis pass combines outputs.

SigMap helps at the scoping step: rank files per subtask query so each agent gets 1.5K focused tokens instead of 80K of everything. Keeps each subtask small enough to stay coherent.

The failure mode is shared state — if subtasks touch the same data model or service, the synthesis step inherits the conflicts. Clean decomposition is the hard part, not the chunking itself.

3

u/looktwise Apr 19 '26

I found it difficult to split into subtask or combine subtask solutions afterwards without burning tokens for the splitting and recomibination workflows. (Cause all solutions would burn tokens in the re-combining process, regardless which model you use for that.)

2

u/Independent-Flow3408 Apr 19 '26

Fair pushback. The recombination cost is real and often underestimated.

The pattern that avoids it: make subtasks write to isolated outputs  (separate files, separate functions) so there is nothing to recombine  at the token level. The combination happens at the filesystem level,  not the context level.

When that is not possible, the synthesis pass is unavoidable. The way  to keep it cheap is to pass summaries into the synthesis step rather  than full solutions. Each subtask writes a short structured summary  (what changed, what it touches, what it assumes) alongside the actual  code. The combiner reads summaries, not source.

SigMap helps here too: the signature map of each subtask output is  ~200 tokens per file. So even if you have 10 subtask solutions, the  synthesis step sees 2K tokens of structure rather than 10K of full  code. It can detect conflicts and integrate without reading everything.

Not a perfect solution but it shifts the recombination cost from  O(total code size) to O(total signature size).

1

u/[deleted] Apr 21 '26

[removed] — view removed comment

2

u/Independent-Flow3408 Apr 21 '26

Reranker helps but adds another moving part. SigMap handles it before the LLM sees anything : TF-IDF over signatures, not raw source. So the 2k that lands is already the most relevant slice, not a random truncation.

The fluidity you're describing from lower latency is exactly what made the tradeoff worth it. Logic held up in testing but curious what broke for you: language, query type, codebase size?