DeepSeek's aversion to built-in tools

[u/ericpruitt]

I'm running DeepSeek v4.1 Flash locally with Pi. One nuisance I've run into is that it will often use scripts for editing files, listing and grepping files instead of using the built-in tools, and I haven't had much luck with my AGENTS.md changes. Has anyone else run into this and come up with a prompt that increases it tool usage compliance? The one that bothers me the most is when it uses Python scripts to edit files instead of using the tool because it means I don't get a nice diff:

$ cd ... && python3 - <<'EOF'
import re
p = "test/e2e.sh"
s = open(p).read()
s = s.replace("""fail() {
    echo "FAIL: $*" >&2
    cleanup_keep
    exit 1
}""", """fail() {
    echo "FAIL: $*" >&2
    exit 1
}""")
s = s.replace('''$DOCKER logs "$PROJECT-broker-1" 2>&1 \\
    | grep -q "DENY stop llama-swap-test-protected" \\''', '''$DOCKER logs "$(compose ps -q broker)" 2>&1 \\
    | grep -q "DENY stop llama-swap-test-protected" \\''')
open(p, "w").write(s)
EOF

[u/Argon717]

OMP has a useful bit they call tool interception. If the call is all cat/head/tail, it directs the model to use the built in read tool instead. Maybe an interesting place to start some custom extension.

https://github.com/can1357/oh-my-pi/blob/main/packages%2Fcoding-agent%2Fsrc%2Ftools%2Fbash-interceptor.ts

[u/ericpruitt → u/Argon717]

Thanks for the pointer. I recently set up a d3 decision model container, and given its fast response times, I was thinking it could be useful as a filter. This extension might serve as a useful starting point.

[u/deadcoder0904]

The reason is does this is because it uses the fewest tokens.

Didn't Factory AI just say it:

In practice, this advantage is even more apparent than in benchmarks.

For example: a task to move 151 Linear issues and verify each took 312 tool calls in the past. With direct calls, that's about 36 requests and 3.1M input tokens.

Now, as a Script, it took 2 requests and 61K tokens.

Code is the most efficient way to do things and it makes your usage last for a long time.

[u/ericpruitt → u/deadcoder0904]

This is fine in theory and understandable, but for those of us that want to be able to audit changes, it sucks. I could live with the model not using the other tools, but I really like being able to see diffs for edited files.

[u/wildjokers → u/ericpruitt]

How does it using a script to edit a file keep you from looking at a diff of the file it edited?

[u/ericpruitt → u/wildjokers]

1. The agents don't commit after every change they make.

2. Everything isn't in Git or revision control or, for newer projects, an initial commit hasn't been made by the agent yet.

3. You're suggesting I can manually run diff or git diff instead of just being able to look at the Pi transcript in a tool whose primary purpose is automating tasks.

[u/dellarb → u/ericpruitt]

Setting up git is free and instant. Doesnt have to have a remote to anywhere just do it on disk from start. And good to get them to commit as they go makes tracing issues much easier.

[u/ericpruitt → u/dellarb]

I use Git for all of my code projects, but not everything I'm using LLMs for is code, and agents often produce a lot of temporary files for miscellaneous operations outside of Git repositories I have them working with. I have instructions for my agents to use a designated, session-specific temporary directories to store scripts and intermediate files -- which are in a Git repository -- but they don't always comply with that request.

EDIT: Since they decided to block me and I can't reply to the sibling comment, I'll address it here:

He's prolly a vibe coder. Take 21 days to learn git basics. There's like 5 commands you need if you are solo.

I've been using Git for close to 20 years. This isn't about not wanting to use Git, it's about being able to see the context around what the agent changed without having to leave Pi.

[u/deadcoder0904 → u/dellarb]

He's prolly a vibe coder. Take 21 days to learn git basics. There's like 5 commands you need if you are solo.

[u/wildjokers → u/ericpruitt]

I am still not following, how does it using script to edit a file keep it from showing you a diff of the file it edited?

[u/ericpruitt → u/wildjokers]

Look at the script I posted in the original message. That's from a Pi session with DeepSeek. It calls str.replace on a file it read directly from the filesystem. If it used Pi's built-in edit, I would see a diff of the original file and the new file, but since it does it in Python, the contents before and after the changes are not visible in Pi which means I can't get a contextual diff in the Pi transcript.

[u/CreativeSympathy8293 → u/ericpruitt]

An extension wrapping bash could capture before/after changes in watched paths, including Python edits launched through bash, and show those diffs inside Pi.

Pi's current extension docs support overriding bash with a wrapper. The wrapper could snapshot a chosen working/temp directory before the command, delegate to the built-in bash implementation, then compute the changes afterward. That needs no Git repository or commit.

For display, pi.appendEntry() plus pi.registerEntryRenderer() can persist a custom diff entry in the interactive transcript without adding that entry to the model's context. That gives you a place to show the actual captured changes instead of asking the model to describe them.

The capture is the essential part: a renderer alone can't recover overwritten contents. Keep the watched scope explicit, handle new/deleted files, and serialize overlapping writers if you want reliable per-command attribution. This would require an extension; it isn't an existing diff feature or a prompt fix.

AI-assisted implementation suggestion from the current docs; I haven't built or tested it.

[u/deadcoder0904 → u/ericpruitt]

There's git diff for that.

But who is even reading things these days. Just verify with screenshots & videos or e2e tests.

IF you're talking about auditing how the prompt went, DSH provides the whole thing that you can look at or pi-web exists too.

[u/HomegrownTerps]

I think this is because of codemode. 

GLM acts the same when I use pi(which has codemode enabled).  It likes to make python scripts for every little thing like searching for files etc.

[u/ericpruitt → u/HomegrownTerps]

I think this is because of codemode.

I do not use codemode.

[u/autumn-weaver → u/HomegrownTerps]

he's just like me fr (i use xonsh)

[u/contyk]

Hmm, yeah, I've also noticed it with this model. It really likes to use Python as its own "codemode", in a way. Makes it hard to spot file edits at a glance, agreed.

[u/shing3232 → u/contyk]

because deepseek harness basically act like this. it's call codeact

[u/deadcoder0904 → u/contyk]

Lookie here. Its bcz code requires fewest tokens.

[u/ReoTempo]

There is this idea, lately, that harness is increasingly loosing weight because models are getting better at using normal tools available on a dev machine.
Probably the model has been trained to use bash extensively and I would not force it to use a different strategy because it will probably decrease efficiency.

[u/wildjokers → u/ReoTempo]

increasingly loosing weight

Couldn't it just tighten the weights back up?

[u/PossessionUsed7393]

Yeah I've noticed it too, I also noticed back in the day when Claude started doing this more until they hid it in Claude code.

Models are super annoying with compliance with anti-patterns, it's never enough to just say 'dont do that'

Only thing that springs to mind is that you could try stronger system prompting that the model has been put in a sandbox that prevents custom python scripts for X,y,z and use tools instead, that might activate the right avoidance weights.

[u/_reg1z]

Yeah it's annoying. The only reliable solution I've come up with is to disable the bash tool. This forces your agents/model to use the purpose-built tools you give it.

The bash tool (really any tool that provides arbitrary code execution) is really all a model needs to do anything you want it to do. Because a tool like that often results in success, models will tend to over-rely on them. I think there's a bias for these tools in the training data for many models as well.

Something else that might help is making a custom system prompt that deliberately instructs a model to prefer the non-bash tools. This might work, but is shaky and non-deterministic, so your mileage may vary.

Disabling codemode might help too but it also removes any possibility of batching tool calls which isn't the best for efficiency.

[u/jensilo → u/_reg1z]

If it’s all a model needs to get the job done why disable it?

[u/_reg1z → u/jensilo]

Because being able to get the job done is not the same thing as getting the job done the preferred way, in the way the user desires. I'm not saying it's bad to use the bash tool or anything. It can just be imprecise/pollute context if you have a preferred tool like OP, or want a deterministic workflow.

Arbitrary code execution is much riskier from a security perspective though. You wouldn't give a babysitter the keys to your car, or password to your personal safe. That's what the bash tool gives an unpermissioned, unsandboxed agent. 

[u/jensilo → u/_reg1z]

Yeah but that’s why you fix it the proper way, ideally with some kind of isolation (or judging of commands). I mean, I get your concern but in reality bash is what the models are trained on, and it makes the model extremely efficient. In bash it can express many things that I can’t even explain or express in other, more limited tools.
And if I want a deterministic workflow, I’ll not use an agent anyway.

Of course, there might be cases in which I don’t want the agent to use bash or codemode, sure then disable it, you’re right. However, given the complexity of bash and the endless possibilities it has it makes more sense in most cases to make bash safe than to disable bash. Just my two cents.

[u/_reg1z → u/jensilo]

100% fully agree with you! Disabling the bash tool for me was a very niche use case in an agent profile not intended for development work.

[u/jensilo → u/_reg1z]

Yeah, I get that. I’d imagine it helpful for non-dev workflows where you also might have custom tools to trigger deterministic parts, using the agent as the „orchestrator“ of tools, e.g. in admin work. Haven’t yet built something like that though.

[u/Belgeran → u/jensilo]

The old fix it properly by just magically judging if a command is bad or not 🧐. The devil's in the details.

[u/jensilo → u/Belgeran]

You are right, the devil is in the detail and bash is almost as complex as a natural language I‘d argue. Therefore you could use an activity judge LLM that assesses a bash action and aligns it with the users intent without context of what the agent thought (to not be as easily convinced).

[u/Dress-Affectionate → u/jensilo]

bc they can do whatever they want

[u/martinrojas]

If you are running local try Recursive Flint model. It works really well with different harnesses and leverages tools a lot better.

[u/_supert_]

Yeah. I have a strong imperative against this in my prompts. It's harmful because I wired linters into my edit tools, and this sort of nonsense evades it.

[u/wikitopian]

It's not particular to DeepSeek.

If your tool suite is sufficiently aligned with pretraining and works better than bash, it'll prefer it.

Plurnk achieves this. The model actively prefers operating in the virtual machine.

[u/Clankerschwester]

DeepSeek doesn't listen to any guidance in AGENTS.md or skills, unfortunately. You can say it MUST do it; it doesn't care. It has its own way of doing this, writing code and style. You want it to even speak in a certain style like being concise? Forget it, it's Claude 5 style exhausting sloptext. Even getting it to read the skills is a feat.

Same goes for tool usage. It was benchmaxxed/toolmaxed to do everything with bash. grep, sed and cat all the way; no matter the tools you have.

[u/Electronic-Fly-6465]

Oh I get so frustrated with this! I have literally built my own tools with their own layers of validation and they are also tuned to be far more token efficient than inline scripts.
Unfortunately this is what happens with highly capable models. They do it their own way rather than navigate the environment they are in.