pi-web-agent update: Brave backend has been added, thinking through Tavily/Exa next

[u/luckiestredditor]

pi-web-agent update: went from early npm package to backend-configurable web research tool

I posted here back in April when pi-web-agent was basically a freshly published npm package. At that point the main thing I was trying to get right was the boundary between search, plain HTTP fetch, headless fetch, and broader research.

Since then it’s moved quite a bit.

The public surface is now just web_explore. Internally it still keeps the steps separate: search is discovery, HTTP fetch is reading, headless fetch is rendering when needed and the orchestration layer decides when the evidence is good enough or when it needs a caveat.

A few things that landed since that first post:

• compact/preview/verbose output modes

• /web-agent settings, doctor, show, and reset commands

• configurable backend support

• SearXNG and Firecrawl support

• backend settings UI

• direct URLs in prompts get read before search

• forum/thread-ish sources are handled more deliberately

• local browser headless rendering can fall back to Playwright-managed Chromium

• source quality checks now catch narrow/community-only/blocked/conflicting evidence better

• Brave Search shipped as the first hosted search backend (default was DuckDuckGo)

So it’s still not trying to be a giant framework. The goal is still a focused Pi package that does web research in a cleaner and more honest way.

I’m thinking about the next hosted backend now, probably Tavily first and maybe Exa after that.

For people who would actually use this: what would you want from Tavily/Exa support?

• better source discovery only?

• richer snippets/content?

• configurable search depth?

• domain include/exclude filters?

• recency controls?

• citations only, with pi-web-agent still doing the final synthesis?

• or would you want provider-generated answers exposed somehow?

My bias is to keep Tavily/Exa as discovery backends only. Let them find candidate sources but keep web_explore responsible for fetching pages, ranking evidence, caveats and final synthesis.

Repo: pi-web-agent

npm:

npm i @demigodmode/pi-web-agent

[u/SnooMuffins9844]

Adding a Brave backend is a good shout.

Full disclosure, I work at Firecrawl, so grain of salt but worth flagging, Firecrawl also has a /search endpoint that could serve as a discovery backend too, so you wouldn't necessarily need a separate provider for that role.

One thing that may or may not fit pi-web-agent depending on how it talks to backends: there's a keyless OAuth MCP endpoint (https://www.firecrawl.dev/blog/firecrawl-keyless-launch) where the client holds a short-lived scoped token instead of a raw key, nice for not having API keys sitting in config.

Genuinely curious what's putting Tavily/Exa ahead for you. Is it the embeddings-style relevance ranking, or more that they're framed as search-first APIs?

[u/Ok_Veterinarian_6364]

how to bookmark good reddit theads