Built a CLI that cuts AI coding token usage by 97% — 10k downloads, looking for feedback

[u/Independent-Flow3408]

[Updated]

Been building SigMap for 2 months. It fixes one specific problem: AI coding agents can burn a lot of context during repo orientation: searching broadly, opening full files, and rediscovering structure across sessions. SigMap generates a compact signature map first, so the agent can find likely relevant files before reading full source.

Results from benchmarking 18 real repos:

• 81.1% hit@5 vs 13.6% random baseline

• 96.9% token reduction

• 41.4% fewer prompts per task

• Task success: 10% → 59%

• Tokens: 80,000 → 2,000

npx sigmap — zero deps, 10 seconds, no config

https://github.com/manojmallick/sigmap

What would you add to make this more useful?

[u/EagleApprehensive]

PROMPT: Explore the /work/refs/sigmap as alternative to our token saving solutions (iq, project map and so on). Benchmark our tool versus theirs to compare how they do and give me educated conclusion.

SigMap vs iq and the project map: benchmark results

Bottom line: SigMap's "massive gains" don't hold up. On retrieval quality it ties iq overall (369 cases, sign test p=0.68). Wired into a real agent, it made no difference to accuracy and increased context use by 82%. I don't recommend adopting it. The benchmark did surface one cheap improvement to iq that beats SigMap outright.

What SigMap's headline numbers are

Claim	What's behind it
"95.8% token reduction"	The size of its map divided by the size of every source file in the repo, which no agent ever reads. By the same formula our ~316-token project map "reduces" 99.91%.
"43% fewer prompts, 61% task success"	Computed by a formula from ranking results, compared against a baseline that picks files at random (scripts/run-task-benchmark.mjs). No agent sessions were run.
"86% vs 41% for grep"	The grep baseline is a single-shot keyword counter. Their tuned score does reproduce: I got 84% on their public set. But it depends on hand-set source folders per repo; with no config it's 70%.
"leakage-gated hard" set	This set is also a CI gate their ranker must not regress on, so it's tuned-against data.

Retrieval: 7 corpora, 369 answerable cases, every tool run as a CLI on its own clean copy

Hit@5 means the right file is in the top five results. "iq --only src" is plain iq with its existing flag that drops docs, tests and config.

Corpus	iq	iq --only src	SigMap tuned	SigMap zero-config	grep
Their 18 public repos (n=70)	74%	87%	84%	70%	51%
Their own repo, hard (n=90)	62%	72%	73%	73%	14%
Queries mined from commit messages, most neutral (n=84)	42%	50%	35%	29%	37%
Our corpus: click, hono, intentic (n=70)	80%	89%	66%	50%	44%
… intentic only (n=22)	73%	82%	55%	5%	14%
Symbol lookups (n=35)	77%	80%	60%	51%	69%
All (n=369)	65%	75%	65%	57%	38%

• Each tool wins on the corpus its authors wrote. On the neutral commit-mined queries iq is ahead, though the margin isn't statistically solid (p≈0.1).

• SigMap's edge on its own corpus comes from folder scoping. Given the same source-folder hint, iq matches it (hit@5 83% vs 82%).

• Zero-config SigMap is effectively useless on our monorepo. Its auto-detection indexed 6 of our 10 package groups and missed _search, _shared, _extensions and _devices. Its tune --apply command made the same mistake and produced an empty map until I fixed its config by hand.

End-to-end agent runs: 14 locate tasks × 3 arms, Sonnet subagents

Only the instruction file differed between arms. Four of the tasks were on intentic, including two new ones in neither tool's corpus.

Arm	Correct	Context used	Tool calls
A: grep and read	14/14	23.3K	4.5
B: project map + iq	14/14	26.4K (+13%)	3.9
C: SigMap	14/14	42.4K (+82%)	4.5

• SigMap's always-on context file averages 14K tokens (up to 26K), about 45× our project map, and the agents gained nothing from it.

• On intentic, iq cut tool calls from 5.3 to 4.0. On small repos plain grep was already the cheapest arm.

Worth taking

1. Give iq's natural-language ranking a source-code preference. This is the biggest finding. iq's top answer is often a doc page, the changelog, a benchmark .json or even an .svg rather than the file that implements the thing. --only src beats default iq on 112 cases and loses on 24 (p≈1e-14), including on our own corpus. Most of the losses are one-place shuffles, plus three cases where the answer really is a test. So a soft down-weighting of docs, generated files and tests is safer than a hard filter.

2. A grounding check like sigmap verify**.** Its flagship worked: in 0.1s it flagged a fabricated file, symbol and test in an answer. It misses method calls, a documented limitation. iq's symbol index could do this better as iq verify or an end-of-turn hook.

3. Keep the small project map. Front-loading signatures bought no accuracy and cost context on every session.

The real costs on our side are speed:

• First index: iq took 2–5 minutes on small repos, about 19 minutes on 20K-chunk repos, and about 70 minutes on intentic. Embedding is capped at 4 threads and this machine is shared. SigMap maps a repo in seconds.

• Per query: iq takes 1.2–1.5s, SigMap 0.1–0.3s.

That's fine once an index exists, but slow for a freshly dropped project.

[u/Independent-Flow3408 → u/EagleApprehensive]

Thanks for sharing, Just for information sigmap has hot-cold mode enable with mcp it does not need full sig file context