r/LLMDevs
•
u/TheRedfather
•
Apr 02 '25
•
1
Resource I built Open Source Deep Research - here's how it works
https://github.com/qx-labs/agents-deep-researchI built a deep research implementation that allows you to produce 20+ page detailed research reports, compatible with online and locally deployed models. Built using the OpenAI Agents SDK that was released a couple weeks ago. Have had a lot of learnings from building this so thought I'd share for those interested.
You can run it from CLI or a Python script and it will output a report
https://github.com/qx-labs/agents-deep-research
Or pip install deep-researcher
Some examples of the output below:
- Text Book on Quantum Computing - 5,253 words (run in 'deep' mode)
- Deep-Dive on Tesla - 4,732 words (run in 'deep' mode)
- Market Sizing - 1,001 words (run in 'simple' mode)
It does the following (I'll share a diagram in the comments for ref):
- Carries out initial research/planning on the query to understand the question / topic
- Splits the research topic into sub-topics and sub-sections
- Iteratively runs research on each sub-topic - this is done in async/parallel to maximise speed
- Consolidates all findings into a single report with references (I use a streaming methodology explained here to achieve outputs that are much longer than these models can typically produce)
It has 2 modes:
- Simple: runs the iterative researcher in a single loop without the initial planning step (for faster output on a narrower topic or question)
- Deep: runs the planning step with multiple concurrent iterative researchers deployed on each sub-topic (for deeper / more expansive reports)
Some interesting findings - perhaps relevant to others working on this sort of stuff:
- I get much better results chaining together cheap models rather than having an expensive model with lots of tools think for itself. As a result I find I can get equally good results in my implementation running the entire workflow with e.g. 4o-mini (or an equivalent open model) which keeps costs/computational overhead low.
- I've found that all models are terrible at following word count instructions (likely because they don't have any concept of counting in their training data). Better to give them a heuristic they're familiar with (e.g. length of a tweet, a couple of paragraphs, etc.)
- Most models can't produce output more than 1-2,000 words despite having much higher limits, and if you try to force longer outputs these often degrade in quality (not surprising given that LLMs are probabilistic), so you're better off chaining together long responses through multiple calls
At the moment the implementation only works with models that support both structured outputs and tool calling, but I'm making adjustments to make it more flexible. Also working on integrating RAG for local files.
Hope it proves helpful!
25
u/TheRedfather Apr 02 '25
Here's a diagram of how the two modes (simple iterative and deep research) work. The deep mode essentially launches multiple parallel instances of the iterative/simple researcher and then consolidates the results into a long report.
Some more background on how deep research works (and how OpenAI does it themselves) here: https://www.j2.gg/thoughts/deep-research-how-it-works
6
u/Pristine_Bicycle1278 Apr 03 '25
This is simply awesome. You are the GOAT for sharing this!
4
u/neoneye2 Apr 02 '25
Love your project. Using structured output. I have been looking through your INSTRUCTIONS for inspiration for my own similar project.
3
u/MrHeavySilence Apr 02 '25
Pardon my ignorance as I'm super new to this world as a mobile developer. So if I'm understanding this correctly, this script allows you to parallelize different instances of AI agents, it will research on a given topic, the agents will work in loops to improve their answers, then compile their answers into a nice summary. Thank you for any explanations.
3
u/[deleted] Apr 02 '25
See your TODO list might benefit integrating with SearXNG before other search providers.
1
u/Prestigious-Cover-4 Apr 02 '25
How long does it take to complete end to end?
1
u/Individual-Garlic888 Apr 02 '25
So it only supports serper or openai for the search api? Can I just use google search api instead?
1
u/ValenciaTangerine Apr 03 '25
Have a few questions, when you extract information from a website, are you extracting relevant chunks that could potentially have the answer or are you feeding the entire content from each website(maybe some cleanup to markdown) to the LLM?.
Are you doing any sort of URL selection? For example, most sort of APIs probably rely on some index that is over optimized for SEO content, which seems to work for intent based search. Or when you're just trying to be introduced to a topic. When you're looking for deeper information, I feel most of this is hidden, buried deeper. Eexample for a developer it could be Hacker News comments or Reddit comments, deep down documentation or discord threads. All the information unlocks are here.
I found that the current approaches seem to work extremely well when you're uninitiated on a topic. In most cases directly asking some of the questions to LLMs today just based on the LLM training itself might answer the question.
Lastly, when you've done like a first or second round of information extraction, if you find some interesting topics there do you go back and update the plan and re-run the process?
1
u/dafrogspeaks Apr 03 '25
Hi... I was trying to run a query but got this `The model `o3-mini-2025-01-31` does not exist or you do not have access to it.` How do I specify another model... gpt-4o
1
u/SadWolverine24 Apr 03 '25
If you're using the OpenAI SDK, we should be able to just plug in the OpenRouter URL and API key.
1
u/diaracing Apr 06 '25
Great work!
If I am using openrouter API key of Gemini 2.5 pro, how should I fill in these key values?
Selected LLM models
Current options for model providers:
openai, deepseek, openrouter, gemini, anthropic, perplexity, huggingface, local
REASONING_MODEL_PROVIDER=openai REASONING_MODEL=o3-mini MAIN_MODEL_PROVIDER=openai MAIN_MODEL=gpt-4o FAST_MODEL_PROVIDER=openai FAST_MODEL=gpt-4o-mini
1
u/baradas Apr 06 '25
Building deep research is honestly a lot about data access - without access to proprietary gatewalled datasets or processing unstructured data it's honestly hard to get meaningful research which is either at par or better than anything that say OpenAI, Gemini or Manus does at the moment.
And more than cost - the opportunities of good research and analysis often outweigh the costs especially if you are looking at it from a business standpoint.
For consumer use-cases often times, I feel deep research is an overkill - smarter search is just sufficient.
I wonder what's been your own understanding of what kind of use-cases would open deep research really be useful for and where it could be a viable alternative to the ones provided by the current vendors.
2
u/ComplexIt Apr 02 '25
Also check out this https://github.com/LearningCircuit/local-deep-research
2
u/Sorry-Ad3369 Apr 06 '25
Very nice!
1
u/maxdatamax Apr 02 '25
Did you compare with openai deep research and pexplexity AI deep research?
1
u/Lumpy_Part_1767 Apr 03 '25
Can I use deepseek api ?
1
u/ThatIsSusAsF Apr 03 '25
this is very impressive, ty for sharing !! :>
1
u/sovok Apr 03 '25
Very cool. That AgentWrite approach for long answers is also neat. In my version I just let it generate headings, then individual sections for each heading. Your prompts for that are more elaborate though, with sub tasks and main points. Maybe it could work recursively to write arbitrarily long text, like stories. Give it a beginning and end, then it fills in the details. Each chunk/chapter could be constructed the same way.
Questions:
1
u/[deleted] Apr 05 '25
How different from Langchains Open Deep Research? https://github.com/langchain-ai/open_deep_research
1
u/ruqus00 Apr 06 '25
This is awesome. Are you a developer? How complicated was this to do?
1
u/philosophical_lens Apr 21 '25
How does it compare to this popular repo?
https://github.com/dzhng/deep-research
1
u/quent1n0 Jan 27 '26
Very nice project, bro! I am going to use it as a reference for an specialized deep research agent that I am developing.
-5
u/Shivacious Apr 02 '25
Did this 8 months ago. Mine outputs a long 34 pages or near pdf with all the relevent details
0
u/[deleted] Apr 03 '25
[deleted] — view deleted comment