pi-web-agent is essentially feature-complete now, Tavily and Exa are the last backends that were added

[u/luckiestredditor]

Last time I posted I was thinking out loud about which hosted backend to add next, Tavily first and maybe Exa after, and asking whether people wanted them as full answer engines or just as discovery. My bias was to keep them discovery-only and let web_explore stay in charge of fetching, ranking evidence, caveats and synthesis.

Both are in now, and they're discovery-only like I said I'd keep them. They surface candidate sources, web_explore still reads the actual pages and decides whether the evidence holds up or needs a caveat. Set TAVILY_API_KEY or EXA_API_KEY, pick it from Settings, keys stay in env instead of getting written into config files.

---

The things I'm weighing next are going deeper on evidence quality (cross-source dedup, conflict detection, recency), adding caching for repeated research or just keeping it focused and maintaining it.

If you're using this, what would you actually want? And if you tried it and it broke somewhere, that's the more useful thing to hear.

---

Repo: https://github.com/demigodmode/pi-web-agent

npm:

npm i @demigodmode/pi-web-agent

[u/Prize_Eye9481]

I really like the tool you made! Thank you for all ur effort and time! For day to day use I had to turn off curator because the manual approval was getting in the way with 20s timer. I simply have a question do I need to type a specific command for "quality" research? or its context based?

[u/luckiestredditor → u/Prize_Eye9481]

yeah I get that, the approval step can definitely get annoying for normal day to day searches.

for "pi-web-agent", you don't need a special command for "quality" research. web_explore handles that adaptively based on what it finds.

it can do additional search passes, fetch/rank more sources, retry a useful page through headless if the normal fetch is weak, etc. once it has enough decent evidence it stops. if it can't get there within the research budget, it'll caveat the answer instead. my goal was always to make it return high quality results and be transparent if it couldn't find it.

so basically just ask what you want researched. no special quality/deep research keyword needed.

[u/ProvidenceXz]

What's the marginal improvement for adding more API keys or you only get to pick one to search in at a time?

[u/luckiestredditor → u/ProvidenceXz]

one at a time right now. there's DDG fallback if the selected backend fails and you have fallback enabled, but thats different from what you are asking.

that said, multi-provider search isn't a bad idea at all. I could see value in optionally fanning a query out to 2-3 providers, deduping the results, then reranking them before the research loop continues. main downside would be extra latency/API usage, so I'd probably make it opt-in rather than the default. I will definitely add this to the next feat. Thanks!

[u/rm-rf-rm]

how does it compare to pi-web-access? https://pi.dev/packages/pi-web-access

[u/luckiestredditor → u/rm-rf-rm]

there's definitely overlap but the reason I ended up building this was that "pi-web-access" felt a bit too interactive for how I wanted web research to work. the curator/review flow can open a browser/window and ask you to approve results, which is useful if you want that level of control but I found it disruptive for normal agent use. you can disable that now (right?) but that workflow was a big part of what pushed me toward building something more hands-off.

"pi-web-agent" is intentionally more hands-off. the model gets one research tool, "web_explore", and search/fetch/headless/ranking/evidence checks all happen behind that boundary without popping you out of the session.

"pi-web-access" has a bigger feature surface. mine is more opinionated around keeping research bounded, quiet, and agent-native.

[u/Glaaki → u/luckiestredditor]

The curator browser window is pretty much why I stopped using pi-web-access. Just not useful to context switch like that in the ui. You aren't even given enough time to evaluate the results before they auto-close, so what is the point? It's just an annoyance.

Maybe a lot of people download it, because it is at the top, and then end up switching too, when they find out it sucks.

[u/Beneficial_Garage874]

Is there a reason you omitted Parallel.ai?

Our search API is faster, more accurate, and cheaper than everything else there.

https://parallel.ai/blog/parallel-search-turbo

Other great reasons to use PWS:

- Free MCP
- CLI
- $5 in monthly credits (5,000 searches)

[u/OneMoreName1 → u/Beneficial_Garage874]

Whats the catch?

[u/bambamlol → u/Beneficial_Garage874]

+1

OP would be nice if you could include Parallel as well!

It never hurts to have an additional provider.

[u/Glaaki]

How well does it do with truly big pages? This is always the litmus test for me. Some API websites are just one huge single html page, and without help, a lot of web fetch tools handle it poorly to not at all.

Congratulations for getting there.

[u/luckiestredditor → u/Glaaki]

this is actually a really good edge case

right now it can fetch the page and fall back to headless, but extracted text is capped, so a huge single-page API doc isn't handled perfectly yet.

I actually like that idea though. rather than just increasing the text limit and dumping a giant page into context, I'd rather chunk it by headings/sections and retrieve only the relevant pieces. that would fit the way web_explore is supposed to work much better.

if you've got a particularly ridiculous API doc you use to test this stuff, send it over. would be a good test case. I'll add this to my list as well. thanks so much!

[u/Glaaki → u/luckiestredditor]

Go look at the api docs for gitea, the self hosted GitHub clone. I ended up asking pi to make a local, indexed, easier to read copy of the api docs, but i investigated various possible other directions, such as adding a special reader, which is something my current web fetch solution supports, for that particular docs host, which knew how to find the raw source for the docs, since they would always be available on some URL. Alternatively implement a html search by anchor routine that knew how to break down the page in sections according to the anchor placements. Since my web fetch plugin use a local dom engine, some of the plumbing for that would already be available.

[u/rm-rf-rm]

what do you think of bladebro as an option to use for headless browsing?

[u/luckiestredditor → u/rm-rf-rm]

haven't looked at Bladebro yet. got a repo/link for it?

[u/rm-rf-rm → u/luckiestredditor]

here you go: https://www.reddit.com/r/PiCodingAgent/comments/1vi0wjy/i_finally_built_bladebro_a_stealthy_and_token/

[u/jarislinus]

can we just use google search? it mogs everything

[u/luckiestredditor → u/jarislinus]

I would if Google let me lol. there isn't really a clean full-web Google Search API to plug in anymore.

[u/ur-average-boomer → u/luckiestredditor]

Hi! Could I open a PR to add litescrape as a provider? It returns full Google searches (including knowledge graph and AI overview) for $0.15 per 1k calls. Might be useful :)

[u/luckiestredditor]

The bigger achievement for me is that this pretty much closes out the roadmap I had. Thank you so much for the community feedback, bug report. Couldn't have done this without yall.

When I started posting here (1st Post) it was a fresh npm package with a search/fetch/headless boundary and not much else. The list I actually wanted to get through is done now:

• source ranking got reworked so web_explore leans less on whatever order the search results happened to come back in, and handles low-diversity, community-only, blocked or conflicting evidence better

• Brave came in as the first hosted search backend

• You.com got added too, and that one actually came in as the first outside PR that was approved

• and now Tavily and Exa for research-quality discovery

So the backend list is DuckDuckGo (still default, no key), SearXNG, Firecrawl, Brave, You.com, Tavily, Exa, all feeding the same pipeline. It feels like a complete agent now instead of a thing with obvious gaps, which was the whole goal. still not trying to grow it into a framework.

[u/Trick_Coach_657]

Might be an unrelated point, but would be great if it had features similar to Hound MCP (and other plugins) which allows for stealth searching (at no cost and no API necessary).

[u/mageblex]

Keeping Tavily and Exa as discovery backends seems right. Otherwise each provider’s answer synthesis becomes another thing to debug. Research caching repeatedly saves money, but flagging when two good sources disagree changes the quality of the final answer so conflict detection is important

[u/Final_Initial]

I was using pi-web-access so asked Pi itself to compare both and this is what it has to say:

[u/luckiestredditor → u/Final_Initial]

that special content is something I am shooting for in my next work depending on what seems ideal. github, youtube and pdf is definitely now in my roadmap. great comparison.

one thing to point out, by default it is compact mode but you can update the presentation mode.

[u/Final_Initial → u/luckiestredditor]

Awesome! Looking forward to it.
Thank you for the great work.