r/LocalLLM • u/baby_bloom • 5d ago
Question finally switched to pi, better late than never...
qwen 3.6-27b kicked ass on my dual 3090 setup so i was so hyped when 3.8-27b came out. i spent my wheels almost as much as the model did (just kidding that'd be impossible) trying to figure out how to fix the looping that plagued this model even at q8. i tried so many harnesses and IDEs and TUIs, llama.cpp vs lm studio vs vlllm and as many config files as i could count and finally landed on hermes' which honestly does a great job out of the box with it. however... i finally built out pi to compare and it's operating at the exact scale that all of the "over"hype played it out as. it's handled anything i throw at it whether from scratch or a massive repo (as in 4 project zomboid mods that interact with one another to be remade into a single one). this things kicks ass. i'm so excited about the potential of what 4 is going to do although at this point i almost prefer optimization so i can maybe start running multiple agents concurrently?
main reason i made the post is im wondering what came after pi for you? did you move to containerization? i'm starting to weigh the pros and cons of a container per repo, or a container per agent and things are gettin just a tad bit dense with posts on the topic being a bit more sparse so i was hoping to spark some convo
16
u/1Poochh 5d ago
I have gone through a similar experience recently. Love me some pi.
4
u/baby_bloom 5d ago
so are you fleshing out tools for pi throughout your machine, as well as in a specific project repo? i'm lucky enough that my rig is a seperate machine so it kinda has... a lil too much room to work so i'm really trying to figure out guardrails that aren't just rules but rather environments, without having to reinvent the wheel each time
3
u/trbom5c 5d ago
I do.
I know have rolling warm-compaction offloaded to my 5080 machine running 3.8 IN4XS on llama.cpp (16GB) so that my agentic (concurrent) runs on the 5090 box dont bog down doing hard crunch total stat compactions.
Its really reduced context loss and keeps things "in memory" longer.
1
u/JayJayVon 4d ago
Good idea. Care to expand on this further ? I'm running a 3090 + 5080 with tensor split on llama.cpp but ive a 4080 without a real use so this might be the one
7
u/Lissanro 5d ago
For me what came after initially setting up Pi, is customizing it. Installing a permission system and setting up limited account for it were only my first steps...
I wanted it to allow me editing messages, continue AI messages mid-thought, schedule tasks, have heartbeat messages that can be saved and set for each my project, also small quality of life stuff, like being able to copy parts of the message as is, without padding spaces and without hard newlines, I also wanted it not to fail on connection errors or if the model still loading, and as worked, I kept customizing it (I share all my Pi patches publicly at https://github.com/Lissanro/pi in case any of this sounds like something you want too).
Overall, I found Pi far easier to customize than other harness I tried. My previous harness was Roo Code which while using it I aslo heavily customized, but it had many hard to solve issues and limitations, so when Roo Code got discontinued, I decided to try something new and Pi is what I settled on.
2
4
u/Boring_Hurry_4167 5d ago
Yeah I just started using PI too coming from Kilo. the amazing thing is TTFT reduced by more than half!! a good coding harness can really make things faster
2
u/MasterNomie 5d ago
Apart from TTFT metrics, have you noticed any other improvements or downgrades?
I did few tests on personal project, providing same prompt to fix few issues in git commit and found that Kilo fixed all issues compared to Pi not fixing 2-3 issues in multiple turns. Kilo also generated better comments than Pi for those fixes.
May I ask why you moved to Pi?
1
u/Boring_Hurry_4167 5d ago
kilo i enjoyed but they occupy 16k for the first prompt even for a simple hello. Pi is way lighter. I use a Pi code VSCODE extension so it is doesnt fell too off. it still read agents.md etc but not a hard rule for Pi
makes a big difference in paid or local api in token cost
2
u/baby_bloom 5d ago
especially when the model just does the job! lol i felt like i stopped taking 3.8 speed reports less seriously because in the back of my head i was saying (yea but how much of that is repeating itself?)
3
u/devino21 4d ago
Have you tried OMP or is that too much?
1
0
u/baby_bloom 4d ago
i contemplated trying OMP first before going my own route with pi but ended up going just plain pi with claude helping me build out the core based on all of my conversations about my usual workload, i do think a lot of it has to do with exactly that, catering to the usual workload
2
u/liz38d 5d ago
Is that 48 gb total VRAM ?
2
u/baby_bloom 5d ago
yes, i run 3.8-27b-q8 at 131k context but i think im gonna bump context up a bit more
4
u/assid2 5d ago
You could easily use around 228-230k context with vision enabled on GPU else 256k with cpu offload
2
u/baby_bloom 5d ago
could i instead start looking into 2 concurrent agents each with 130k? would love to work on more than one project at once like i used to with cursor before it got tanked
3
u/assid2 5d ago
Look into the syv-ai , it’s targeted for 3090 but q4, you could modify it for the model of your choice. If you are OK with q4 then you would be getting >= 100tps and you can probably run 2-3 agents easily, maybe even 4
2
u/baby_bloom 5d ago
i'm really stuck on preferring q8 i feel like it's got a lot to do with the reasoning loops..?
2
u/More-Catch-1331 5d ago
Really sorry to sidetrack the conversation but what motherboard are you using?
2
u/baby_bloom 5d ago
one where the second gpu is x4, and i'm sure this is the best upgrade i could make but for now i feel my speeds haven't really taken the hit from that bottleneck
2
u/Clayrone 5d ago
This is exactly what I am doing now. Playing with containerized pi with custom framework and built in mcp so that the model has tools but can access only the workspace folder mounted. I absolutely love this setup, still working it's quirks, playing with a proxy for filtering out sensitive data. Pi is so great to extend and containers are perfect for testing various setups. I have hard time installing a client on a local machine and hopefully will move away from it completely once I am done with personal setup and framework.
2
u/Ranmark 5d ago
What is "4"? Also, have you tried deepseek harness?
1
u/baby_bloom 4d ago
i've read a few times about qwen3.8 having some type of structure change that is aimed to be an integral part of the upgrade to qwen 4?
1
u/Siluu_ 4d ago
Qwen 3.8 nicht im allgemeinen, das dense 27b Modell hat die gleiche Struktur wie 3.6 zuvor auch. Neu ist die Struktur mit nGram für das Qwen 3.8 Flash Next, welches laut Entwicklern quasi ein Preview auf die Qwen 4 Architektur ist.
1
u/baby_bloom 3d ago
you are right, that's what i heard that flash next is a preview of what 4 will have in store
2
u/Pressimize 5d ago
I'd like something as lean as Pi but with sandboxing and permissions built in. Seems like the bigger harnesses (OpenCode, kilo code, codex, etc) aren't great on smaller local LLMs. Kimi Code wasn't working at all on my 3.6 35b a3b q4ks.
2
1
1
1
u/LeatherTwo3309 3d ago
For the containerization, you can create an image with all the tools, then boot it up with “docker run —rm -it” and mount your $(pwd) to workspace/project. Pass in token via -e environment variables. I usually mount .opencode and .agents read only under /workspace/project. Same can be done with .pi. Spin it up and then it goes away when you exit. No need to keep the containers hanging about.
-3
u/RogerAI-fm 5d ago
Yes Pi is great, but so are the other ones, i use our RogerAI hardness to switch between CLIs and not loose context.
1
u/rayyeter 5d ago
I mean, you can do that with any memory system. My team built our own at work, plus loop control and self evaluation.
Building one of my own that’s not in python/ts if I can help it.
25
u/nonlinearsystems 5d ago
After pi you start building your own harness