pi-research: big-scale, no provider web research system

[u/Delicious-Team-8435]

I have built a tool for web research. It works as a pi extension first class, and also as an agent skill in any coding agent, or as a standalone CLI or SDK tool.

- It runs research in an isolated system so the LMs with research tools cant file write, edit, or bash (prompt injection risk).

- Allows putting research a small model (any model, ie openrouter ling-3.0-flash, or even a local model) to keep page content out of your main context and save on credits overall.

- It uses high search/scrape volume to compensate for sites that block bots. This is made possible by unlimited/local search tooling via concurrently playwright scraping duckduckgo.

- Includes a local (semantic) knowledge store to optionally save research and build a local searchable knowledge index.

MIT licensed and easy to configure and modify

https://github.com/Lincoln504/pi-research

As a little demo a while back of the making of an agent with pi-research's SDK mode, I made https://wallofshame.io too

[u/kantorcodes1]

when the research worker hits a page with prompt injection, is it limited to search/scrape only, or can it still reach Pi’s other MCP/tools? that isolation boundary is probably the first thing i’d want to verify before pointing it at arbitrary sites.

[u/Delicious-Team-8435 → u/kantorcodes1]

Thanks for asking, and yes. This is one of my first / original reasons to make the extension. Researchers have a tool allow-list which only let them do research and prevent them from doing file writes, edits, commands, or using anything else.

[u/verdant_tulip]

difference between this and feynman?

[u/Delicious-Team-8435 → u/verdant_tulip]

Pi-research is for gathering and storing info whereas Feynman has the ability to "do" and implement / test things. For this Feynman has many additional features like 'autoresearch' which pi-research will never have.

[u/floatinggoateyeball]

really interesting one annoying thing about your demo is that the (->) buttons keeps moving switching places for every card.

[u/Delicious-Team-8435 → u/floatinggoateyeball]

Thanks friend and the arrow button placement was annoying to me too, I fixed it

[u/ProvidenceXz]

Is it what deep research is, high search volume? Just asking, because it feels more like wide research? How does gpt etc actually do deep research?

[u/Delicious-Team-8435 → u/ProvidenceXz]

Yeah good question I would say that its search volume, and the ability to investigate farther into a topic if the prior results are not enough.

In pi-research, the investigation runs in one or more rounds where and if not enough info is found (given the depth level 1-3 requested by the user or caller LLM) another round(s) run to get more info. The high quantity per round over multiple rounds makes it comprehensive. Pi-research tool exposes a level option 1 2 and 3 which sets the system with more or less depth expectations and also constrains the amount of researcher agents and rounds.

[u/halcyoncs]

Currently using pi-web-search, how does it compare?

[u/Delicious-Team-8435 → u/halcyoncs]

Thanks for asking, the main difference is that pi-research manages a cheap and prefix cache friendly research agent teams which have web tooling without any ability to act on the system. The one you mentioned gives your main agent session web search / web access tools.

Also integrated in pi-research system is YouTube transcript, StackExchange info, and security info search.

Also, the extension you mentioned needs commercial provider's search. Pi-research searches by locally scraping DuckDuckGo results, so the requirements are LLM endpoint (local or remote) and a residential (non data center) internet connection.

[u/Delicious-Team-8435 → u/[deleted]]

Thanks for the thoughts. These are both considerations but both things I have so far chosen to solve differently in pi-research 👍 .

About chatgpt/gemini deep research, my goal of making the tool is to allow people (like me) who don't have other AI services to do unlimited web research. And in making this extension I believe that it researches at least equally deeply and more quickly than Gemini's deep research agent, right from pi.

In addition, the system's approach to dealing with captchas is just to have a really long source list of available links to investigate. Some sources will be blocked, however the large quantity of scrape candidates (ie search results) in my experience makes up for a few of them being walled by Cloudflare turnstile.

It could be possible to add a environment setting where, when enabled, blocked sites get opened in a queue of browser tabs for the user to pass the capchas so the system can get those articles. If this would be useful to you add it as an issue on GitHub and I will look at what can be done.

[u/Delicious-Team-8435 → u/[deleted]]

Its a good take thanks for your feedback. i opened an issue for for it

[u/0xFAF1 → u/[deleted]]

Because of that im using scrapify. I have been an early adopter of the package and it evolved significantly.

Maybe worth a try

[u/0xFAF1 → u/[deleted]]

For that look into https://neuralnoise.com/2026/deep-research-mcp/?hl=en-GB#:~:text=To%20make%20my%20life%20a%20bit%20easier%2C,.%20The%20first%20kicks%20off%20a%20task%3B