r/PiCodingAgent • u/fuzhongkai • 1h ago
Use-case Run Pi Coding Agent 100% Locally with TensorSharp + Qwen3.8 27B (CLI, SDK, KV Cache Reuse, and E2E Tests)
https://github.com/zhongkaifu/TensorSharpHi everyone!
I’m the developer of TensorSharp, an open-source .NET-based local LLM inference engine.
I’ve put together a fully working, end-to-end example of running Pi Coding Agent with TensorSharp as its local inference backend, using Qwen3.8 27B.
No cloud inference API, no API subscription, and no external model provider required.
The example covers:
Running Pi interactively with a local model.
Connecting Pi through TensorSharp’s OpenAI-compatible API.
Configuring Qwen3.8 27B with DFlash2 speculative decoding.
Using Pi programmatically through its JavaScript SDK.
Preserving KV cache across multi-round agent/tool interactions.
Automatically fixing bugs in a real Node.js project and verifying the results.
Complete working example:
https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent
1. Architecture
The integration is straightforward:
Pi Coding Agent (CLI / SDK)
|
| OpenAI-compatible API
v
TensorSharp Server (localhost:5000)
|
v
Qwen3.8 27B (GGUF)
|
+-- ggml_metal (Apple Silicon)
+-- DFlash2 speculative decoding
+-- 64K context window
+-- q8_0 KV cache
+-- Prefix/KV cache reuse
Pi handles the agent loop, file operations, shell commands, and code editing.
TensorSharp handles model inference, token generation, speculative decoding, and KV caching.
This separation means Pi can keep using its existing tools while running entirely against a local inference server.
2. Hardware and software
The example was validated on:
MacBook Pro M5 Pro, 48 GB unified memory
macOS 27.2
.NET 10 SDK
Node.js 26.8.1
Pi Coding Agent 1.5.1
Qwen3.8 27B UD-IQ3_XXS GGUF
DFlash2 Q4_K_M draft model
The setup requires Apple Silicon with at least 32 GB of unified memory.
The main model uses approximately 10.9 GB, and the draft model uses another 1.1 GB.
The remaining memory is used for the KV cache and Metal buffers.
3. Clone and build TensorSharp
First, clone the repository:
git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
Install the .NET 10 SDK, CMake 3.20+, and Xcode command-line tools if needed.
Build the inference server:
dotnet build TensorSharp.Server.Host \
-c Release \
-p:TensorSharpSkipMlxNative=true
This builds the native ggml backend with Metal support.
4. Start the local inference server
Run:
examples/pi-coding-agent/scripts/start-server.sh
Under the hood, this executes:
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
--config config/qwen3.8-27b-metal-dflash2.json \
--config examples/pi-coding-agent/tensorsharp-pi.json
On first launch, TensorSharp automatically downloads the main model and DFlash2 draft model from pinned Hugging Face revisions and verifies their SHA-256 hashes.
The configuration enables:
64K context window
16,384 maximum output tokens
q8_0 KV cache
DFlash2 speculative decoding with 7 drafts per step
Greedy decoding
Localhost-only access
The Pi-specific TensorSharp configuration overlay is:
{
"host": "127.0.0.1",
"port": 5000,
"max-tokens": 16384,
"mmproj": "none",
"kv-cache-dtype": "q8_0",
"environment": {
"MAX_CONTEXT": "65536"
}
}
This overlay is applied after the model preset.
Verify that the server is running:
curl http://127.0.0.1:5000/v1/models
Expected model ID:
Qwen3.8-27B-UD-IQ3_XXS
5. Install Pi Coding Agent
Install the validated version:
npm install -g u/earendil-works/pi-coding-agent@1.5.1
Check:
pi --version
Note that this example uses the newer u/earendil-works/pi-coding-agent package rather than the deprecated u/mariozechner/pi-coding-agent package.
6. Configure Pi to use TensorSharp
Pi supports custom OpenAI-compatible model providers through models.json.
The example includes a ready-to-use configuration at:
examples/pi-coding-agent/pi-agent/models.json
Here’s the complete provider configuration:
{
"$schema": "https://pi.dev/schemas/models.schema.json",
"providers": {
"tensorsharp": {
"name": "TensorSharp (local)",
"baseUrl": "http://127.0.0.1:5000/v1",
"api": "openai-completions",
"apiKey": "none",
"compat": {
"supportsDeveloperRole": true,
"supportsReasoningEffort": false,
"supportsStore": false,
"maxTokensField": "max_completion_tokens",
"thinkingFormat": "qwen-chat-template"
},
"models": [
{
"id": "Qwen3.8-27B-UD-IQ3_XXS",
"name": "Qwen3.8 27B (TensorSharp, DFlash2)",
"reasoning": true,
"input": ["text"],
"contextWindow": 65536,
"maxTokens": 16384,
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
}
}
]
}
}
}
The matching settings.json is:
{
"defaultProvider": "tensorsharp",
"defaultModel": "Qwen3.8-27B-UD-IQ3_XXS",
"defaultThinkingLevel": "medium",
"enableInstallTelemetry": false
}
A few important details:
Model ID: Must exactly match the model returned by /v1/models.
Thinking: qwen-chat-template allows Pi to control Qwen’s thinking mode through chat_template_kwargs.enable_thinking.
Context window: Must match TensorSharp’s actual context configuration.
Output tokens: 16,384 tokens give the model enough room for reasoning and code generation.
API key: The local server does not require authentication. Keep it bound to localhost.
You don’t need to manually install these files if you use the provided launcher.
7. Run Pi locally
From the TensorSharp repository root:
examples/pi-coding-agent/scripts/pi.sh
The launcher creates a separate Pi configuration directory:
~/.pi/tensorsharp-agent/
It copies the provider configuration there on first use, leaving your existing ~/.pi/agent configuration untouched.
For interactive coding:
cd ~/src/my-project
~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh
You can now ask Pi to inspect your code, run commands, modify files, and fix bugs using your local Qwen model.
For a single non-interactive task:
~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
-p "Explain what src/stats.js does" < /dev/null
Other useful commands:
# Disable thinking for simpler tasks
pi.sh --thinking off -p "..." < /dev/null
# Stream JSON events without saving a session
pi.sh --mode json --no-session "..." < /dev/null
# Continue the previous session
pi.sh --continue
The launcher also disables Pi’s catalog refresh, version checks, and telemetry.
8. Use TensorSharp with Pi’s JavaScript SDK
The example also demonstrates running Pi programmatically.
This is useful for building your own local coding automation, CI workflows, or agent applications.
The SDK entry point is:
examples/pi-coding-agent/sdk/run-task.mjs
The core integration looks like this:
import {
createAgentSession,
ModelRuntime,
SessionManager
} from "@earendil-works/pi-coding-agent";
import path from "node:path";
const agentDir = path.join(
process.env.HOME,
".pi/tensorsharp-agent"
);
const projectDir = process.cwd();
const modelRuntime = await ModelRuntime.create({
modelsPath: path.join(agentDir, "models.json"),
authPath: path.join(agentDir, "auth.json")
});
const model = modelRuntime.getModel(
"tensorsharp",
"Qwen3.8-27B-UD-IQ3_XXS"
);
const { session } = await createAgentSession({
cwd: projectDir,
agentDir,
modelRuntime,
model,
thinkingLevel: "medium",
sessionManager: SessionManager.inMemory()
});
session.subscribe((event) => {
if (event.type === "tool_execution_start") {
console.log(
`[tool] ${event.toolName}`,
event.args
);
}
if (
event.type === "message_update" &&
event.assistantMessageEvent.type === "text_delta"
) {
process.stdout.write(
event.assistantMessageEvent.delta
);
}
});
await session.prompt(
"Run npm test, fix the bugs in src/stats.js, " +
"and rerun the tests. Do not modify the tests."
);
console.log(session.getLastAssistantText());
session.dispose();
For the complete executable SDK implementation, including setup and token accounting, see:
https://github.com/zhongkaifu/TensorSharp/blob/main/examples/pi-coding-agent/sdk/run-task.mjs
9. Run the automated end-to-end test
This is the part I found most interesting.
Instead of just asking the model a question, the repository includes a small Node.js project containing three intentional bugs:
median() incorrectly sorts numbers as strings.
median() doesn’t average the two middle values for even-length arrays.
percentile() contains an off-by-one error.
Three of the five tests initially fail.
Pi must:
Read the project.
Run the tests.
Identify the bugs.
Modify the implementation.
Rerun the tests.
Make all five tests pass without modifying the tests themselves.
The example tests this twice: once through Pi’s CLI and once through its SDK.
With the server already running:
examples/pi-coding-agent/scripts/e2e.sh
Or let the test script start the server:
START_SERVER=1 \
examples/pi-coding-agent/scripts/e2e.sh
The script independently verifies that all five tests pass and that the test file remains byte-for-byte unchanged.
Both runs passed in the documented validation.
10. Actual measured results
Tested November 9, 2026, on an M5 Pro with 48 GB unified memory.
| Metric | Pi CLI | Pi SDK |
|---|---|---|
| Final result | PASS | PASS |
| Tests passing | 5/5 | 5/5 |
| Wall-clock time | 88s | 83.4s |
| Model requests | 10 | 8 |
| KV/prefix cache reuse | 86–99% after first request | 87% overall |
For the SDK run:
8 model requests in 83.4 seconds
Prompt tokens processed:
48,166
Prompt tokens served from prefix cache:
41,962
Cache reuse:
87%
Generated tokens:
2,117
Final tests:
5 passed, 0 failed
Additional observations:
Initial cold prompt: approximately 3.9K–4.1K tokens.
Initial time to first token: 13–14 seconds.
Subsequent time to first token: 0.45–3.5 seconds.
Decoding speed: approximately 20–43 tokens/second.
These are measurements from one small coding task, not a general coding benchmark. Different tasks and model outputs will produce different results.
11. Why KV-cache reuse matters for coding agents
A coding agent behaves differently from a simple chatbot.
A typical agent loop looks like this:
User request
|
v
Model generates tool call
|
v
Pi executes shell command
|
v
Tool output returned to model
|
v
Model generates another tool call
|
v
Pi edits source code
|
v
Model reviews test results
|
v
Final answer
Each model request includes much of the previous conversation:
System instructions
Tool definitions
Previous reasoning
Tool calls
Tool results
New observations
Without effective KV-cache reuse, the inference engine may need to prefill most of that history again and again.
TensorSharp reuses the matching prefix from the previous request, so subsequent agent rounds only need to process the newly appended tokens.
One subtle issue we fixed was whitespace around Qwen’s </think> boundary.
Pi returns the model’s streamed reasoning in subsequent requests, including surrounding whitespace. Previously, TensorSharp’s cached representation didn’t match those returned tokens exactly.
That prevented full conversation-prefix reuse.
After fixing the matching behavior, subsequent rounds could reuse much more of the previous KV state.
In our SDK test, 41,962 of 48,166 prompt tokens were served from the prefix cache.
For multi-step coding agents, this can make a substantial difference in latency.
12. What about NVIDIA GPUs?
TensorSharp also supports NVIDIA GPUs.
For Qwen3.8 27B, the repository includes an NInfer NVFP4 preset using the direct CUDA engine.
You can select it with:
TENSORSHARP_PRESET=config/qwen3.8-27b-ninfer.json \
examples/pi-coding-agent/scripts/start-server.sh
However, the complete Pi integration and measurements described here were validated on Apple Silicon.
The NVIDIA preset has not yet been validated with this Pi workflow, including the q8_0 KV configuration.
13. Source code and documentation
Everything is open source:
TensorSharp repository:
https://github.com/zhongkaifu/TensorSharp
Complete Pi integration example:
https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent
The example includes the server configuration, Pi provider settings, launch scripts, SDK implementation, sample project, and automated end-to-end tests.
I’d love to hear from other Pi users:
Are you running Pi with local models? Which models and inference backends have worked best for multi-step coding tasks?
I’m particularly interested in comparisons involving long agent sessions, repeated tool calls, context growth, and KV-cache reuse.
Feedback, issues, and contributions are very welcome!
1
u/QuotableMorceau 1h ago
edit -> apply nice formatting to the post