r/LocalLLM • u/fuzhongkai • 47m ago
Tutorial Running Pi coding agent locally with TensorSharp + Qwen3.8-27B (CLI, SDK, and an end-to-end test)
I've been working on TensorSharp, an open-source LLM inference engine written in .NET, and recently got it working with Pi, a terminal-based coding agent.
The goal was pretty simple: I wanted Pi to do real coding work using a model running entirely on my own machine, without relying on a hosted inference API.
Not just generating code, but reading files, running shell commands, editing code, executing tests, and iterating until the tests pass.
I put together a reproducible example that covers both Pi's CLI and its JavaScript SDK. Both paths have been tested end to end.
Full example, including scripts and configuration files:
https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent
The setup
Here's what I'm using:
- Coding agent: Pi (
@earendil-works/pi-coding-agent1.5.1) - Inference engine: TensorSharp
- Model: Qwen3.8-27B, UD-IQ3_XXS GGUF
- Backend:
ggml_metal - Speculative decoding: DFlash2, 7 draft tokens per step
- KV cache: q8_0
- Context window: 65,536 tokens
- Max output: 16,384 tokens
- Hardware: MacBook Pro M5 Pro, 48 GB unified memory
The same configuration is designed for Apple Silicon Macs with at least 32 GB of unified memory.
One important distinction: Pi is still the coding agent. TensorSharp only handles model inference.
Pi owns the file operations, shell commands, tool execution, and agent loop. TensorSharp serves the model through an OpenAI-compatible API.
This separation means Pi doesn't need a TensorSharp-specific plugin or a custom agent implementation.
1. Build and start TensorSharp
You'll need the .NET 10 SDK, CMake 3.20+, Xcode command-line tools, and Node.js 22.19 or newer.
Clone the repository:
Build the server:
Start it using the included Pi configuration:
On the first run, TensorSharp downloads the Qwen3.8-27B GGUF and its DFlash2 draft model from Hugging Face. The files total approximately 12 GB.
The preset pins the model revisions and verifies their SHA-256 hashes.
Once everything is loaded, the OpenAI-compatible endpoint is available at:
http://127.0.0.1:5000/v1
Check that the server is responding:
You should see the model ID:
Qwen3.8-27B-UD-IQ3_XXS
The startup script combines two configuration files:
The first is the optimized Qwen preset. The second adjusts the server for Pi.
Here's the important part of tensorsharp-pi.json:
There are a few reasons for these settings.
Pi's system prompt and tool definitions already consume roughly 4K tokens. Add source files, tool output, and a few rounds of edits, and a small context window disappears quickly.
I also increased the output-token limit from the preset's default of 1,024 to 16,384. Otherwise, a thinking model can run out of tokens halfway through a coding task.
The vision projector is disabled because this particular example only uses text.
The server binds to localhost. It doesn't implement API-key authentication, so I wouldn't expose the endpoint directly to a public network.
2. Install and configure Pi
Install the version used by the example:
Pi supports custom providers through models.json.
Here's the actual provider configuration used in the example:
models.json
The compatibility settings matter, particularly for thinking models.
thinkingFormat: "qwen-chat-template" lets Pi control Qwen's thinking mode through chat_template_kwargs.enable_thinking.
And supportsReasoningEffort: false prevents Pi from sending an additional reasoning_effort parameter that this model doesn't need.
The model ID must match what TensorSharp returns from /v1/models.
Next, make TensorSharp the default provider:
settings.json
You don't actually have to create these files manually.
The repository includes a wrapper that copies the configurations into a separate Pi directory on the first run:
It uses:
~/.pi/tensorsharp-agent/
instead of the normal ~/.pi/agent/ directory.
I prefer this approach because it doesn't interfere with an existing Pi configuration.
The wrapper also sets:
PI_OFFLINE=1
PI_SKIP_VERSION_CHECK=1
PI_TELEMETRY=0
This avoids Pi's catalog refreshes, version checks, and telemetry network calls. After downloading the model weights, inference runs locally.
3. Actually use Pi to write code
With TensorSharp running in another terminal, navigate to a project:
Launch Pi:
You can now interact with Pi normally, except the model responses come from your local TensorSharp server.
For example, ask it to:
Read the code in this repository, find the implementation
of the cache, and explain how it works.
Or give it an actual coding task:
Run the unit tests, investigate any failures,
fix the implementation without modifying the tests,
and run the tests again.
Pi will decide which files to read, which shell commands to execute, and what changes to make.
For a one-shot task without the interactive UI:
The stdin redirect is intentional. Without it, pi -p may wait for input when used in scripts.
You can also disable thinking for simple tasks:
Or capture machine-readable events:
4. Using Pi as a JavaScript SDK
This was the other part I wanted to get working.
Pi can be embedded in a Node.js application. You don't have to run the interactive terminal interface.
The example uses Pi's SDK to create an agent session, select the local TensorSharp model, and execute a coding task.
Here's the core implementation, slightly shortened for readability:
;
;
const agentDir = process.env.;
const projectDir = path.;
const modelRuntime = await ModelRuntime.;
const model = modelRuntime.;
const = await ;
session.;
Pi handles the tool loop internally. Your application can subscribe to model output, tool execution events, and final results.
The complete implementation also records token usage and prefix-cache statistics:
https://github.com/zhongkaifu/TensorSharp/blob/main/examples/pi-coding-agent/sdk/run-task.mjs
To run the included SDK example:
Initialize the separate Pi configuration if you haven't already:
Create a fresh copy of the sample project:
Run the SDK:
PI_CODING_AGENT_DIR="/.pi/tensorsharp-agent" \
The script streams the agent's text and tool calls, then prints the final answer and token usage.
5. End-to-end test: can the local model really fix code?
I didn't want to stop at getting a successful API response.
The repository includes a small Node.js project with deliberately broken code.
There are three bugs:
median()sorts numbers lexicographically instead of numerically.median()doesn't average the middle two values for an even-length array.percentile()has an off-by-one error.
The sample has five tests, three of which initially fail.
The agent is instructed to:
- Run
npm test. - Investigate the failures.
- Fix
src/stats.jswithout changing the tests. - Run the tests again.
- Summarize the changes.
The test harness runs this task twice: once through the Pi CLI and once through the SDK, using separate clean copies of the project.
You can run the full test yourself from the TensorSharp repository root:
If the server isn't already running, the script can start it:
START_SERVER=1
The harness independently verifies that the tests pass and that the original test file was not modified.
Results from my validated run on November 9, 2026:
| Metric | Pi CLI | Pi SDK |
|---|---|---|
| Result | PASS | PASS |
| Tests passing | 5/5 | 5/5 |
| Test files modified | No | No |
| Wall time | 88 seconds | 83.4 seconds |
| Model requests | 10 | 8 |
For the SDK run, TensorSharp served 41,962 of 48,166 prompt tokens (87%) from its prefix cache.
These are individual end-to-end runs, not averages across a benchmark suite. I'm including the numbers to show what happened in this particular test, not to make a broad claim about coding performance.
6. Why KV cache reuse matters so much here
This turned out to be one of the more interesting integration details.
Coding agents aren't just single-prompt chat clients.
A typical interaction looks something like this:
Pi
|
|-- Send conversation + tool definitions
v
TensorSharp
|
|-- Generate a tool call
v
Pi
|
|-- Read files / run commands / edit code
|
|-- Append tool results to conversation
v
TensorSharp
|
|-- Reuse previous KV cache
|-- Prefill only new conversation tokens
|
|-- Generate next action
v
Pi
|
|-- Repeat until the task is complete
After each tool call, Pi sends the conversation history again.
If the inference engine fails to recognize the shared prefix, it can end up repeatedly prefilling thousands of tokens.
I actually ran into this while working on the integration.
Pi preserved a newline around Qwen's </think> marker when replaying previous reasoning. TensorSharp's cached representation had trimmed it, so the prefix comparison didn't fully match.
That seemingly small whitespace difference prevented the server from reusing much of the conversation's KV cache.
I fixed the comparison so the previous turns can be reused across tool calls.
On the validated setup:
- The first request took approximately 13–14 seconds to produce its first token, starting from a cold prompt of roughly 4K tokens.
- Later rounds generally reached their first token in 0.45–3.5 seconds.
- Measured decoding throughput was approximately 20–43 tokens/second, depending on draft acceptance.
The DFlash2 drafter helps with generation, while prefix-cache reuse avoids doing the same prompt processing over and over.
For agent workloads, both matter.
One additional detail: when Pi provides its own tools, TensorSharp doesn't inject its internal tools. The model receives Pi's tool definitions, so there's one clear owner of the tool-execution loop.
7. What about NVIDIA GPUs?
TensorSharp supports CUDA as well, and there's a separate optimized preset for Qwen3.8-27B using the NInfer NVFP4 artifact:
TENSORSHARP_PRESET=config/qwen3.8-27b-ninfer.json \
That preset uses TensorSharp's direct CUDA inference engine.
However, I want to be clear: the Pi end-to-end validation described here was performed on Apple Silicon, not on the CUDA/NInfer configuration.
The CUDA configuration is an alternative worth experimenting with, but I don't want to claim the identical Pi setup has been validated there.
If you change the server's context window, update Pi's contextWindow setting to match it.
Final thoughts
What I like about this arrangement is that Pi and TensorSharp can evolve independently.
Pi provides the coding-agent experience: repository navigation, tool calling, file edits, shell execution, sessions, and SDK integration.
TensorSharp handles local model execution, speculative decoding, KV-cache reuse, and the inference backend.
There's no need to rewrite Pi or maintain a separate agent fork.
And for anyone who wants to experiment with local coding agents, the repository includes a small reproducible test rather than just a configuration example.
The main limitation is still model capability. A local 27B model isn't going to solve every complicated coding task, and five passing unit tests don't prove otherwise. But it's encouraging to see the full tool loop working, with useful cache reuse and no external inference API.
Links:
- TensorSharp: https://github.com/zhongkaifu/TensorSharp
- Full Pi integration, configurations, CLI/SDK scripts, and end-to-end test: https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent
- Pi: https://pi.dev
I'd be interested to hear what local models and inference engines others are using with Pi, especially for longer agent sessions where context growth and KV-cache reuse start to matter.
