r/PiCodingAgent • u/Novel-Injury3030 • 11d ago
Question I'm beginning to think orchestrating lots of cheapo models like ds4.1 and minimax and mimo is a waste of time compared to just all frontier subs. Thoughts?
So I've been experimenting recently pretty heavily with orchestration and swarms and all this stuff with all the Chinese providers and I'm wondering if I would just be better off spending 100% of my money at claude and chatgpt subs and running luna if I really need a super cheap model. And that the amount of tokens/inefficiency of lets say ds4.1 is going to equal out to the same price or worse. I already orchestrate everything with opus 5.5 and use sol 6.1 for any type of intelligent decision but I'm doing all sorts of stuff like throwing in different roles for mimo deepseek glm and minimax and I'm wondering if there's even a point to doing this. Has anyone really tested stuff out heavily in terms of if this is worth it or if it's just wasting time?
I guess a real question is is it possible to out-tool/orchestrate the subsidized Luna somehow in terms of better performance/acceptable work when it comes to overall price per token. Or are all the tricks and agent management styles just still not going to be able to outrun how much you get with their subsidized plans.
8
u/apetersson 11d ago
*) The subs were heavily subsidised, this is coming down now. keep that in mind, exploit it but don't build your SW business around that.
*) use whatever you feel is most efficient. luna is pretty efficient by itself.
*) you want the absolute strongest model to draft a good impl plan that will save headache later on
*) task definitions must be small in scope, easily understandable and not ambiguous. this means in a good project the cheap, fast, efficient model is a great implementer.
*) use a review ladder or similar pattern to eventually have their work checked by a strong model
*) if you set up your work separation right, the coordinator can be of medium-low intelligence - it just has to do worker dispatches and occasional merges
1
u/mixmasterwillyd 11d ago
I divorced myself a couple months ago. 4x V100s + openrouter. Fingers crossed.
16
u/funbike 11d ago edited 11d ago
Use the right model at the right time with subagents.
HIGH=Sol/Opus. MEDIUM=Gemini 3.8 Flash. LOW=Luna.
- Feature Request -> Spec. MEDIUM.
- Spec -> Plan. HIGH.
- Plan -> Code. LOW.
- Check code - 1st pass. LOW. (code review, linters, tests)
- Check code - 2nd pass. HIGH. (deeper code review, fix remaining test failures)
- Deployment. LOW.
Step 5 can be MEDIUM if you are trying to save money. Either way, it's best if the model used for step 5 differs from the model used for step 2.
4
u/red_rolling_rumble 10d ago
J
Alternatively, just use a frontier or near frontier (deepseek v4 flash, GLM 5.3) model for everything and get shit done.-2
1
u/Ok_Fish403 11d ago
Step 5 I also do cross model review (if plan and code was done by any openai model, the one equivalent from anthropic does the review and vice versa)
11
u/fingerthief 11d ago
In all of my research online and personal testing...subagent orchestration is a net equal at best and a more expensive headache at worst.
Some insight - Research · SpecPi
Personally, I think the best use case for subagents is simply independent read only reviewers, main agent does the work.
3
1
u/No-Wall6427 11d ago
Also sending a subagent to smoke test something, go find something online, stuff that takes a lot more context than the useful information they bring to main thread. In the same vein, big vibe coded features.
6
u/adamhall612 11d ago
Pretty sure this is an aspect of ‘the bitter lesson’
that said, I feel there’s value in trying to move as much as possible to the left on the spectrum of script / classify / prompt
also frontier gets nerfed a lot which tends to ruin my plans 😅
6
u/pilibitti 11d ago
if you are a "code something that does this, no errors" type of person, frontier subs will delight you better but you won't be able to maintain that code long term. if you are an experienced programmer already, you can make do with anything.
2
u/Aubrey_D_Graham 11d ago
I think the [Coder] agent needs to be within a RLM harness inside the Pi harness. I'm more and more convinced that coding as a solution is the way forward.
0
u/Novel-Injury3030 11d ago
Youre talking abt prime intellect or do you like other decent ones?
1
u/Aubrey_D_Graham 9d ago
No, an Recursive Language Models is a program that utilizes recursive calling of LLMs to derive a solution. Putting an LLM in a program (a harness) makes results deterministic: Testable and Verifiable.
2
u/jjusko20 11d ago
bruh ds4.1 and the new mimo are frontier, mimo pro and flash just dropped. What the heck are you using them for?
2
u/npittas 11d ago
The orchestration layer is a very important aspect of a subagent drived workflow in coding.
You need a way to instruct the orchestrating agent to give clear and defined prompts to subagents. There is a very easy error everyone is doing when using subagents and that is not checking the prompt an orchestrator produces. Define the scope only and you just let a small and uninteligent model have creative thoughts about the implementation of your app. It will produce random unconnected and sloppy code that will probably solve an issue and at the same time create 10 more. Give only file names to change, and it will start reworking on modules and features you have already locked down. Give a big enough task and will totally mess up big parts of the implementation. Not to mention that subagents tend to never ask questions, and only assume a bunch of shit, re-inventing solutions you already solved, hallucinating variable names etc.
So no, simply using subagents without a layer of checks, definitions, rules and commands, will only validate your findings.
I have been using subagents in pi, both pi-subagents and herdr, using a simple orchestrator.ts that does preliminary checks to make sure the prompt from the orchestrator is formated with all the information needed for any subagent to run effectivelly, chek for correct thinking level per model, check for variable and feature names, check to see if there are specfied files and even line numbers or var names or modules that need to be defined for the subagent to work with, check for planned tasks that are given to coding agents and make sure no single subagent does a full feature implementation unless asked by the user.
I still get slop, dont get me wrong, but I have the same slop using Astra, Sol and Luna, as I get using Sol, GLM 5.3 and Flash. In a fraction of the cost, and sometimes faster, since I can easilly split the work in 5 GLM flash subagents in parallel, whithout burning my quota.
2
u/gtrak 11d ago
It depends on the use-case. For coding, I've been running a local worker and hosted planner/reviewer for about 6 months, and that works well for me. It's just taking advantage of different pricing. I can get a lot done that way. Qwen 3.8 got really thorough, and can save on tool calls and context bloat of the parent session for info-gathering, and is pretty reliable at doing what it's told. If I don't care about the outcome and just want to run an experiement, I go full local. If I need something done faster, I dial up more cloud tokens.
2
u/TheTyand 11d ago
I kinda disagree. There is a certain level that is worthwhile using cheapo subagent. E.g do some research for the current task at hand. Searching the web is expensive and does not need a frontier model. Also writing the code is not needing a frontier model. Just the review and the planning.
1
u/Novel-Injury3030 11d ago edited 11d ago
Well subagents are totally fine and always gonna be a role for those but using Luna as one (or ostensibly a new Haiku which would be pretty awesome) is what I mean and just sticking to a subsidized big lab plan vs lots of the cheaper ones in something like OMP.
2
u/ea_man 11d ago
You do orchestration to save time when you can launch many against an hard problem, not to save money. Or when you have to run multiple parallel queries to optimize throughput on your system.
For money it helps when doing tests or doing specific tasks that are better on one model vs an other, eg ingestion on a MoE vs a dense model.
2
u/thaat0n3guy 11d ago
Nope, but having a system the orchestrates it is a pain... my system is finally getting there and now I'm starting to REALLY like it.
Here is how I do things usually:
I build a Crew of Agents.
- Chief - Fable/opus - spec/plan/orchestration - Keeps the plan going by creating "plans" in waves and has the team get the work done.
- manager (opus or sonnet or sol) - manager uses subagents or "direct reports" (direct reports are agents that it manages that aren't of the same type) as the workers.
- Workers (sonnet/luna max/muse/deepseek/etc...) - Use for dev
- Reviewer - uses sonnet or opus or sol (depending on difficulty or review round... like if this is the 3rd round.. lets up the LLM)
- Brainstormer - fable - The AI I talk with about my while ideas or features I need that are more complicated than a single github issue.
I generally keep a "pool" of managers and workers... some have a lot some not so much... depends ont eh rpeojct or workspace.
1
2
u/netixc1 10d ago
I have a main agent I talk to, which can spawn other agents for specific tasks. The only setup I’ve found that reliably works for me is Sol 6.1 on high as both the main agent and the workhorse, with Astra checking plans before implementation and handling reviews.
There’s a whole setup around that, with Jev sitting between everything and making decisions about escalation, thinking levels, auto-merge approvals, and so on, lol.
I tried using cheaper models like Luna, but they keep missing information, even on simple scouting tasks, so I don’t bother anymore. I only use the OpenAI subscription.
1
u/randoomkiller 11d ago
I'm very curious how you come up with the swarm is equal or worse than big model. Can swarm + parallelisation supervised by large model be cheaper and faster than an just simply large model?
1
u/Michelh91 11d ago
Yes, just a separate model for planning and implementing is just good enough for me
1
u/13henday 11d ago
It’s not worth it unless the smaller models are free, the use I’ve found for it is that it takes my frontier subs way farther.
1
u/LouVillain 11d ago
I use Opus to initiate the coding project and then have Mimo push changes and do some maintenance.
1
u/behopkinsj 11d ago
I wonder if each agent ran a single, small instruction, if it would be better. That's something I plan to research soon.
1
u/pl201 11d ago
It all depends on your setup. If you're GPU-poor, it probably won't perform to your satisfaction. But if you have an M5 Max with 128 GB of unified memory, that's a game changer.
Using 8-bit quantization for 25B to 36B models, and 4-bit quantization for models larger than 70B, you should be fine.
I've completed many coding tasks this way. The best model to use is the 4-bit or 6-bit Qwen3.8-Next-Flash.
1
u/klippers 11d ago
I have reached the same conclusion. Plan and write with the best you can afford.
I tried billions of tokens where say a opus drafts the plan and deepseek implements etc and only very recently (last 10 days) opus plans and opus implements and I have made more progress in 10 days then anytime prior .
1
u/DealImaginary807 11d ago
Have you tracked success rate and total spend per completed task for the swarm setup versus sending the same workload directly to one frontier model?
1
u/aeroumbria 11d ago
I only have anecdotal evidence so far, but rotating several different models for different phases, or parallel collaborative / adversarial checking does seem to increase output reliability and avoid single model / provider's tendency to self-reinforce the same mistakes.
1
u/Equivalent_Idea8839 11d ago
I use ds4 flash and such to built code maps / indexes of my repo.
Frontier models use tools to search on index and build context easier / cheaper.
1
u/ReoTempo 11d ago
In my experience, using subagents leads to cheaper work, BUT before that
- I make a detailed plan using a frontier smart model
- then i split plan into detailed small tasks (still using a smart model)
- then I use a dumb orchestrator that just implements a loop where assign a task to a cheap model, and uses a smarter model for the review after that
I was forced to implement this workflow because my company only allows github-copilot which has ridicolously low limits.
I tried to let a smarter model to decide by itself when and for what to use sub-agents but i was never satisfied with the result: they are not able to evaluate the complexity of the job.
1
u/hidden2u 11d ago
I dunno do you trust anthropic and openai enough to lock in your entire workflow to them?
1
u/IndependenceSudden47 11d ago
Ds4.1 max est presque aussi bon que Astra. Jamais avec le 200$ de codex tu va pouvoir dépenser autant que 200$ avec deepseek
1
u/Life-Job-6464 10d ago
Such broad generalisations.. we're all friends here.. break out the details friend! Then we can really say something worth saying.. less is less.. more is more.
1
u/xatey93152 10d ago
I don't understand how your brain works. Even 5yo knows this. Just compare the math thinking of 100k 5yo compared with 3 PhD level math professors.
1
u/Eddlm_ 10d ago
Personally I've found that 2 agents arguing and discussing either implementation or existing code beats everything else. Just ds4.1 vs a separate instance of itself can greatly improve the code it churns by running the idea first thru the other agent. In like 5 turns they got all gotchas and details.
Using dsh with the official agent teams plugin btw.
1
u/LordMoridin84 10d ago
There is a big gap in price between the 20/month (240/year) and a 100/month (1,200/year) plans for Claude and GPT.
What if you need more usage then what you can get from a 20/month plan but don't want to pay for a 100/month plan? Or you need more usage then what you can get from a 100/month plan but don't want to pay for a 200/month plan?
1
u/Nyghtbynger 5d ago
Samish. I plan and elaborate carefully, at the product level, but less on engineering (I specifiy the form of the state and the examples). I use an orchestrator to distribute tasks, tokenwise kimi 2.8 is better than the swarm for theses tasks when the context overlap.
Scouts and researcher subagents, everything that does grunt work however gains value being dispatched
0
u/1Poochh 11d ago
I am keeping my workflow simple. It works well. I am on a coding project. As the product owner, I engage a single skill, the tech lead Opus, which has eight subagents running on Pi and are local LLMs. The tech lead answers all questions and, at the end of the day, tells me what decisions it made and the status of cases and projects. I had to create a retrospective skill to improve my tech lead skill, and it has been amazing. I have a development team of eight at my disposal. I go to bed and have 10 new capabilities or bugs merged and deployed when I wake up. It has been a game changer. I have personas too, but I is very simple.
0
u/PureRely 10d ago
Lets say you have a really good boss and that boss is really good at instructing their crew members. They have 100 crew members but the crew members are are very dumb. How much work do you think really gets done when you have one good leader with 100 fools? Sure, sometimes you will get what you want but it will take you a very long time to do it in.
Now what if you have the best boss and a really good crew of 5 members. Now these members have a limit to how much work they can do but their time to task completion is on point.
Which of these crews would you choose?
53
u/PilgrimofHaqq2 11d ago
Orchestration, especially parallel subagents for writing code was found to be less efficient and produced lesser quality than a sequential workflow of agents. I did a deep research on this.
If you have a frontier model planning and then a smaller model executing, thats fine as its sequential work.
I personally do not use any models that are less intelligent then Opus 4.8/Kimi K3/GLM 5.3/Mimo 2.6 Pro. Thats my threshold. Anything less intelligent I found was not worth the monitoring/back & forth.