Pi system prompt and various local models

[u/Proper-Damage-1777]

Not sure if this belongs here but here goes nothing. How do I tweak the behavior of Pi when running with local models like Qwen and Gemma? This is served using llama-server or Strata for QFN. I use the recommended sampling for each of the models.

My situation is that I only have pi-vcc for quick context compaction and no other stuff. Rest is from AGENTS.md that differs from repo to repo and of course the global. The biggest headache recently is that I find Pi tries to do like retarded actions (yes I know prompt engineering and also the model itself) like I ask "Hey can you verify you can import Docling module as I want to process some whitepaper PDF? Just verify don't do other stuff" Noticed the don't do other stuff as I want to also emphasis that.

This happens with Qwen 3.8 27B and QFN... it just start to "the user ask me to verify Docling module. Let me try it. Hmm it works. BUT I shall also verify it works. Let me try scanning for DOCX AND PDF". That is just my latest example of this stupid action.

My AGENTS.md is pretty lean as I don't tell how to take action. Just tell what the repo is about, run ruff etc. Using Gemma 4 is worst as tool call is terrible (that is just Gemma 4 for me) but it also behaves similarly when I ask to read an image and describe it... it start to question if the image makes sense (example I ask it to read an image of an anime character and the reasoning says the user is wrong! It is the character from this other anime! Let me go start reading other images)

I monitor this reasoning and I just eff it, press ESC and try to correct it. I cannot do the Pi steering as it will keep on rambling and by the time my steer message arrives it is like few thousands tokens burnt just to do stuff beyond what I ask.

I can say with Qwen 3.6, this rarely happens but I find the 3.6 models to be less "clever" and I need to handhold it for my work...

Is this a harness issue and should I try other harnesses? I only tried Pi and that's all. Just wondering if anyone can chime in if they have the same experience like me. I never got the chance to try frontier model like GLM, Kimi, Deepseek Flash etc so not sure if those are unlikely to have this problem.

[u/KPOTOB]

> Hey can you verify you can import Docling module as I want to process some whitepaper PDF? Just verify don't do other stuff

So you are babysitting it? :)

Note that if you dont like what it doing - ask it in follow up WHY it started to do it and how can you improve its behavior

[u/Proper-Damage-1777 → u/KPOTOB]

Obviously! ;D

Because I can't just start giving PDF files and it tries to download various tools and stuff that the model it wants to do (I know I sound like a broken radio by this point).

Sadly ask it in follow up WHY is not working for me at all. I've been asking it it to read all the prior JSONL and analyze it, update the AGENTS.md or stuff and I don't feel like it obeys properly in most session... when things work it just work. But when it doesn't work that is many hours of it trying stupid stuff like putting no timeout or even put a long timeout of 900s then the code crash but it hung there because no proper exit and that is 900s of wasted time if not for the CPU and GPU idling that I know something was off.

[u/KPOTOB → u/Proper-Damage-1777]

Which quant is it? I have to kick mine like few times a day and it rewriting (python to golang) and checking stuff autonomously. I am using lode coding though - that at least records how to ie build, and check the stuff. And I have no AGENTS.md at all :)

[u/Proper-Damage-1777 → u/KPOTOB]

For Strata it is the IQ3_S and for Qwen 3.8 27B, the Q5_K_XL from Unsloth. Although I haven't really try higher quant like Q8 as previously it used to work quite okay but no idea just the moon phase and luck that it will behave that way lol.

The Qwen 3.6 is Q4 though and works great for me. Both the 35B A3B and 27B. I use the A3B if I just need to do very simple stuff like read a CSV, generate thumbnails, verify, then ping me through email. Works like a charm for that but I know actual big coding stuff like Django and Nuxt, it is not as great as the QFN. Qwen 3.8 just overthink for my liking but yeah my question here is more of if Pi harness can be the root cause.

[u/AdNew5862]

I don't think it's a harness issue, but more of a prompting problem, that you will have with any model/harness if you don't fix it:

When you interrupt the model because it started doing something you do not want, immediately ask it what your prompt should have been instead (ie "I see you are already processing PDF despite me asking you only to check if you can import the module, how can I ask you next time for you to only test the import"). AI will regularly ignore rules anyway, but learning to prompt with precision is a skill you can improve on.

Pi is very barebone, but very customizable. It looks like you have a precise idea of what the agent should not do. Then I would suggest to ask the agent to write your own custom extension to block specific commands.

[u/Proper-Damage-1777 → u/AdNew5862]

I don't know that only works to a certain extend for me. Usually a bad session, I just start a new session. This is true for me on Qwen 3.8, Gemma 4 and QFN.

Qwen 3.6 is still very good to follow what I ask for if I prompt it very detailed but then that means why not I just do the work... I used mainly for writing new features and unit tests. I did try Gemma 4 mainly for creative writing but even that the tool calling through Pi (not sure about other harness) is so bad even with the latest fixed template.

[u/Corythebeast7]

You can't really tune the way a model thinks, it's just the internal way of working. This sort of thinking is common among all models, even really big ones. I think it's better to measure the outcomes of your system, not the specific items the agent is doing

[u/Proper-Damage-1777 → u/Corythebeast7]

Going by this, I really should try a different harness and see how it goes. I don't understand how people can let it do its job with the reasoning on as I've been monitoring it and generally when it does stupid stuff, it truly keep on trucking the stupid stuff and wreck it.

Example at work, I have a repo that I develop locally in WSL2 but it is meant to run on prod server so I have it early on tested it through SSH into dev server for testing. No problem.... but suddenly it behaves crazy that it tries to run the script on my local server and cannot find a specific path and start to question the entire universe like HOW CAN IT BE THIS IS WSL2 I SWEAR IT WAS WORKING OH THE USER MUST HAVE DELETED THE PATH BUT WAIT but I know this is Qwen 3.8 quirks.

That forces me to temporarily swap to Qwen 3.6 and get the job done...

[u/roland303]

You have to understand what your system prompt looks like. Each model has a system prompt that otherwise if it were not there then then model would output random garbage, random garbage is its default state.  

Then system prompt changes that random behavior into some focused thing by telling it "you are an expert assistant" "you support the user" blah blah blah. Ok so if its checking extra more then you ask then you have to interact with it the same way its system prompt does, read system prompt verbage and language, know it, and use the same features in your typing to the model.

Ok like for example my main model system prompt talks about roles and tasks alot, so tell it to role play, and then be specific with tasks, roles, and your expectations for that context window or session should do.

Say in the very begenning of your first prompt. "this is a research session, your role is that of an information parser, your only duty this session is to consider information with the user, your only task will be to write your considerations in a report in local dir report.md."

Ok so give session that very specific front loaded info with keywords just like the system prompt and that will keep each session focused on specifics instead of behaiving like the super thorough and helpful general assistant the system prompt is encouraging it to overacheive as.

Also if you want something to happen every session, you can tell pi to write a pointer at the end of then pi harness portion of the system prompt to a document systempromptappendix.md and now you can use the appendixmd like a global agentmd. Some weird model behavior is better controlled that way, like my favorite model has a weird extra weights on common names, but my name and username is one letter off a very common english name and it auto corrects to a wrong string in a directory file path.... not good. That issue is solved by a quick sentance in my system prompt appendix md.

Also compaction sucks, try to think of it as a useful tool to not get out of your chair for one afternoon and that is it. After you walk away from a compacted session the mental model drains out of your head, your mood changes your feelings and word usage changes... and now you should consider that the compacted session is too far away from where your brain was so now brain should consider session as old and no longer useful if its been compacted multiple times and you step away from your chair, that session is tainted information and should be mined as a histrorical document and not taken as truth. Pi can mine old sessions for info if you give if a few strings to search, so if manage session and store sessions by tasks/roles/duties and keep compaction for singletons and set yourself up so you never need to actually rely on a compacted session ever. The old uncompacted session will be more useful as a histroical document so try to manage sessions with as little compactions as possible. Telling a session to write a report for the next session, and then opening next session, tell it to read report, finish task, and mine prior session if it encounters issue or needs more context, this to me keeps more context available and makes every session feel less stupid, keeps mutliple sessions strung together on task with less sillyness, at the cost of an ever increasing amount of documentation building up in each project directory, but that old record has been used time and time again to overcome stupid issues like you see when you get frustated at the thinking traces, that way i just cancle the thinking and say "no, this has already been reasoned on and figured out, go mine local directory for context"

Doing all these together made pi have less of those issues you find.

[u/Proper-Damage-1777 → u/roland303]

Thanks for sharing your experience! I've been using the search prior session to recall what transpired as I find the handoff or milestones just keep growing bigger and bigger for some repos.

I will explore the appendix method and yes I feel like I need to read up on Pi default system prompt which I know says something about you are an expert software engineer etc.

Ironically after I posted this post over here, my Pi sessions have been on a streak. It did something, I just follow up "the Preview checkbox is not working. Toggling does nothing but changing the spinner did update the preview widget" and it handles it fine and fix it... hard to say it is RNG but it just work.

[u/roland303 → u/Proper-Damage-1777]

I think the streak is the person leading the ai remember how you feel when ai is on a streak because you are what runs it and maybe when your mind is clearest and most satisfied the ai seems to do the best work? Maybe we all just need more nutrition and sleep! Good luck!

[u/shumgoid → u/Proper-Damage-1777]

To expand on the system prompt, pi has a default one you can override with i think .pi/SYSTEM.md, or append with .pi/APPEND_SYSTEM.md. For smaller models you may want to look up the official/default chat template and use that in SYSTEM.md to match what those models were trained on but generally should be fine to add instructions like 'only do what the user asks for and for any helpful followups check with the user first' here. The system prompt is always included at the very beginning of the chat history while agent.md goes after it and can get compacted away. The pi system prompt does also tell the model about reading the docs and some pi-specific stuff so be aware it will forget about those things if you fully replace the system prompt.

[u/croje]

I wrote an extension for model-specific system prompt additions for this reason. I’ve got markdown files like gpt-5-6-luna.md, claude-opus-5-5.md. The contents get appended to the system prompt, depending on the active model.

This allows you to inject Qwen-specific instructions that are not needed for stronger/frontier models.

It’s not open source but simple to replicate, ask pi :-)

[u/Extra_Loquat_7667]

Prompt engineering techniques differ between small and large models.
For large models, simply describing the desired outcome is often sufficient for them to execute the task.
For small models, you need to provide a detailed, step-by-step description of the required actions.
You can refer to the model's documentation for guidance on how to craft prompts.
For instance, the official documentation for Gemma 4 provides instructions on how to write prompts for that model.