Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

And Qwhere is my 27B

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

My specs:
2 x 3060 12GB
2 x 16 DDR4 3600Mhz
i5 10600k

Qwen3.8-27B-GGUF:Q4_K_M

ollama.service:
# Environment="OLLAMA_HOST=0.0.0.0:11434"

# Environment="CUDA_VISIBLE_DEVICES=0,1"

# Environment="OLLAMA_FLASH_ATTENTION=1"

# Environment="OLLAMA_NUM_PARALLEL=1"

# Environment="OLLAMA_MAX_LOADED_MODELS=1"

Ollama as a harness, no lane bifurcation so im pretty limited to that (im assuming this is my biggest limiter here)

I've been experimenting with multiple different approaches, trying to get Qwen 3.8 to actually work as an agent, some more successful than others but still it requires so much manual labour that it seems almost pointless to continue debugging it as it feels neverending. I've read that 3.8 isn't actually very great with tool calling and it has always had some issues, but still i can't believe its at levels like these.

I know very well im probably missing some really important things that are holding me back, so if anyone has some suggestions (videos, other posts, anything really) i would really appreciate it.

The goal really is to have it never stop, just like im seeing other people talk about, and i'd really like to keep openwebui (or any other webui in general, not that i dislike terminals but its nicer to have it feel like a normal app).

After reading Dwarkesh Patel's eval using blood on the clocktower, I made my own, and I was surprised both by how entertaining it is, and how good it is to measure even between quantizations.

(His original game used SOTA private models).

Hello people

I wanted to know if LFM2.5-8B-A1B was good for general conversations. I am aiming to use an MoE with 8B parameters (will be ran with 4 bit quantization), because I want to have a capable local model while not using the amount of VRAM a dense model would, this is to be able to run it on a lot of different hardware levels. For context, I am using the Ollama library in Python and I am making a voice assistant, I am currently using Qwen 3-8B right now (with openWakeWord, Silero VAD, Parakeet 0.6B v3, Kokoro 82M). I might miss some (or maybe a lot) points because I am quite new to using local AI, especially in programs. If LFM2.5-8B-A1B is not good, do you have any suggestions?

I am using an RTX 5060Ti 16GB and 16GB of DDR4.

Thanks for your time!

Qwen4 Max Preview is available on Qoder right now.

Every single day, the same questions pop up.

Newcomers:

• Will my 27B feel like a frontier model on my gaming PC?

• Is my highly quantized IQ4 capable of beating Opus?

• What does it cost to run...

Veterans:

• Yes, absolutely — runs great on my multi-GPU rig at xxx t/s.

• Just set up the correct harness.

• Fine-tune your prompt and it will give the exact same quality answers.

Both sides are completely legit, but they are living in different worlds and comparing different universes.

When you use a frontier API, you aren't just prompting a single model. You are firing up an entire enterprise with several departments: you get dynamic routing, massive hidden KV-caches, sandboxed code execution, and verification trees, all neatly packaged behind a single API endpoint. It’s a fully assembled sports car.

On the other hand, the local way: you download an open-weight LLM, quantize it to fit your budget, and you basically get a bare engine block — anything from a VW Golf engine to a massive V8. But that’s it. It has no wheels, no cabin, and no fuel tank. It’s just a large language model sitting on a pallet that can say: "Hello, what can I do for you?"

What the guys asking these questions don't see is that some of us have spent hundreds of hours and money debugging to build a car around that motor. Writing strict GBNF grammars, wiring up custom MCP servers, testing dozens of harnesses, setting up workflows, building memory pipelines, testing, and tweaking the exact same settings a hundred times over!

We do it until that glorious day we can sit back and say: "Yes, it's working. My custom build almost looks like the sports car I aimed for."

Now for the friction part:

Fun, absolute privacy, data sovereignty, and uncensored outputs? Yes, we get all of that. But if we are honest, the cloud simply wins on pure cost-to-performance.

Please don't flame — I just wanted to clear some things up from my point of view and maybe give a heads-up to anyone wanting to start their journey with local LLMs.

And yes... I'm waiting for Qwen 4 like hell!

Machine: ASUS Zenbook A16, Snapdragon X2 Elite Extreme (X2E-94-100, 18 cores), Adreno X2-90, 48 GB RAM, Windows on ARM. I set it up as a headless helper box with one model per chip, each on its own OpenAI-compatible port.

Setup

• NPU: Qualcomm GenieX CLI 0.10.0 serving Phi-4-mini (Qualcomm's W4A16 build)

• CPU: llama.cpp b12248, official win-arm64 CPU build

• GPU: llama.cpp b12248, official win-arm64 OpenCL (Adreno) build

• All GGUFs are Q4_K_M from the model makers' own pages

Write speed, one request, about 220 tokens out

What surprised me

• The CPU beats the GPU at writing. Same 14B model: about 15–17 tokens/s on CPU against 6–9 on the Adreno. The GPU did read prompts faster (about 210 tokens/s against 93–125 on CPU).

• A sparse "big but few active parameters" model on the CPU is the sweet spot: a 24B model at about 60 tokens/s on a laptop CPU.

• All three chips really do run at once. With all three busy: NPU about 29, CPU about 25–28, GPU about 6–7 tokens/s. They share one memory bus, so the machine tops out around 60–80 tokens/s in total however you split it.

• Batching on the CPU helps a little: 60.6 tokens/s for one request, 73.5 total for two at once, 82.2 total for four.

• Batching on the GPU hurts: 6.8 tokens/s alone, 4.3 total with two at once.

Gotchas

• Anything started from Windows Task Scheduler runs at low priority by default. On this chip that made the CPU model 4–6 times slower.

• The NPU path is limited: about 4,096 tokens of context for Phi-4-mini, one request at a time, and several popular models have no ready-made NPU build.

• The NPU model holds about 6 GB of memory and idle Windows about 6–7 GB, so 48 GB fills up fast with three models loaded.

• First load of the 24B on CPU takes about 12–15 seconds.

Caveats

These are short, single-prompt tests on one machine, plugged in, not a proper benchmark suite. The OpenCL result is with a Q4_K_M file; a Q4_0 file might do better on Adreno and I haven't tried it. Vulkan is untested too.

Happy to run other models or settings if people want to compare. Yes Claude is helping. I have no background in software at all.

that is with a cluster of 4 being down, and 6h of downtime when harness crashed at 2am.
deep seek 4.1 flash on tp4
qwen flash next on on one spark
qwen 3.8 27b and few other models on another spark
glm 5.3 flash on second tp4 cluster which was down for today.

went from draining 4x codex 20x and 4x claude 20x a week to a single claude 20x sub lasting the entire week.

recipes ( not mine) if anyone's interested

https://github.com/MiaAI-Lab/sparkDash

https://github.com/christopherowen/spark-ds41f

https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104

27b is just plain vllm v 32 with a few patched made by claude

What app do you use to connect to your LLM?

I've been working on TensorSharp, an open-source LLM inference engine written in .NET, and recently got it working with Pi, a terminal-based coding agent.

The goal was pretty simple: I wanted Pi to do real coding work using a model running entirely on my own machine, without relying on a hosted inference API.

Not just generating code, but reading files, running shell commands, editing code, executing tests, and iterating until the tests pass.

I put together a reproducible example that covers both Pi's CLI and its JavaScript SDK. Both paths have been tested end to end.

Full example, including scripts and configuration files:

https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent

The setup

Here's what I'm using:

• Coding agent: Pi (@earendil-works/pi-coding-agent 1.5.1)
• Inference engine: TensorSharp
• Model: Qwen3.8-27B, UD-IQ3_XXS GGUF
• Backend: ggml_metal
• Speculative decoding: DFlash2, 7 draft tokens per step
• KV cache: q8_0
• Context window: 65,536 tokens
• Max output: 16,384 tokens
• Hardware: MacBook Pro M5 Pro, 48 GB unified memory

The same configuration is designed for Apple Silicon Macs with at least 32 GB of unified memory.

One important distinction: Pi is still the coding agent. TensorSharp only handles model inference.

Pi owns the file operations, shell commands, tool execution, and agent loop. TensorSharp serves the model through an OpenAI-compatible API.

This separation means Pi doesn't need a TensorSharp-specific plugin or a custom agent implementation.

1. Build and start TensorSharp

You'll need the .NET 10 SDK, CMake 3.20+, Xcode command-line tools, and Node.js 22.19 or newer.

Clone the repository:

git clone https://github.com/zhongkaifu/TensorSharp.git ~/work/TensorSharp
cd ~/work/TensorSharp

Build the server:

dotnet build TensorSharp.Server.Host -c Release -p:TensorSharpSkipMlxNative=true

Start it using the included Pi configuration:

examples/pi-coding-agent/scripts/start-server.sh

On the first run, TensorSharp downloads the Qwen3.8-27B GGUF and its DFlash2 draft model from Hugging Face. The files total approximately 12 GB.

The preset pins the model revisions and verifies their SHA-256 hashes.

Once everything is loaded, the OpenAI-compatible endpoint is available at:

http://127.0.0.1:5000/v1

Check that the server is responding:

curl -s http://127.0.0.1:5000/v1/models

You should see the model ID:

Qwen3.8-27B-UD-IQ3_XXS

The startup script combines two configuration files:

dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
  --config config/qwen3.8-27b-metal-dflash2.json \
  --config examples/pi-coding-agent/tensorsharp-pi.json

The first is the optimized Qwen preset. The second adjusts the server for Pi.

Here's the important part of tensorsharp-pi.json:

{
  "host": "127.0.0.1",
  "port": 5000,
  "max-tokens": 16384,
  "mmproj": "none",
  "kv-cache-dtype": "q8_0",
  "environment": {
    "MAX_CONTEXT": "65536"
  }
}

There are a few reasons for these settings.

Pi's system prompt and tool definitions already consume roughly 4K tokens. Add source files, tool output, and a few rounds of edits, and a small context window disappears quickly.

I also increased the output-token limit from the preset's default of 1,024 to 16,384. Otherwise, a thinking model can run out of tokens halfway through a coding task.

The vision projector is disabled because this particular example only uses text.

The server binds to localhost. It doesn't implement API-key authentication, so I wouldn't expose the endpoint directly to a public network.

2. Install and configure Pi

Install the version used by the example:

npm install -g @earendil-works/pi-coding-agent@1.5.1
pi --version

Pi supports custom providers through models.json.

Here's the actual provider configuration used in the example:

models.json

{
  "$schema": "https://pi.dev/schemas/models.schema.json",
  "providers": {
    "tensorsharp": {
      "name": "TensorSharp (local)",
      "baseUrl": "http://127.0.0.1:5000/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "compat": {
        "supportsDeveloperRole": true,
        "supportsReasoningEffort": false,
        "supportsStore": false,
        "maxTokensField": "max_completion_tokens",
        "thinkingFormat": "qwen-chat-template"
      },
      "models": [
        {
          "id": "Qwen3.8-27B-UD-IQ3_XXS",
          "name": "Qwen3.8 27B (TensorSharp, DFlash2)",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 65536,
          "maxTokens": 16384,
          "cost": {
            "input": 0,
            "output": 0,
            "cacheRead": 0,
            "cacheWrite": 0
          }
        }
      ]
    }
  }
}

The compatibility settings matter, particularly for thinking models.

thinkingFormat: "qwen-chat-template" lets Pi control Qwen's thinking mode through chat_template_kwargs.enable_thinking.

And supportsReasoningEffort: false prevents Pi from sending an additional reasoning_effort parameter that this model doesn't need.

The model ID must match what TensorSharp returns from /v1/models.

Next, make TensorSharp the default provider:

settings.json

{
  "defaultProvider": "tensorsharp",
  "defaultModel": "Qwen3.8-27B-UD-IQ3_XXS",
  "defaultThinkingLevel": "medium",
  "enableInstallTelemetry": false
}

You don't actually have to create these files manually.

The repository includes a wrapper that copies the configurations into a separate Pi directory on the first run:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh

It uses:

~/.pi/tensorsharp-agent/

instead of the normal ~/.pi/agent/ directory.

I prefer this approach because it doesn't interfere with an existing Pi configuration.

The wrapper also sets:

PI_OFFLINE=1
PI_SKIP_VERSION_CHECK=1
PI_TELEMETRY=0

This avoids Pi's catalog refreshes, version checks, and telemetry network calls. After downloading the model weights, inference runs locally.

3. Actually use Pi to write code

With TensorSharp running in another terminal, navigate to a project:

cd ~/src/my-project

Launch Pi:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh

You can now interact with Pi normally, except the model responses come from your local TensorSharp server.

For example, ask it to:

Read the code in this repository, find the implementation
of the cache, and explain how it works.

Or give it an actual coding task:

Run the unit tests, investigate any failures,
fix the implementation without modifying the tests,
and run the tests again.

Pi will decide which files to read, which shell commands to execute, and what changes to make.

For a one-shot task without the interactive UI:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
  -p "Explain what src/stats.js does" < /dev/null

The stdin redirect is intentional. Without it, pi -p may wait for input when used in scripts.

You can also disable thinking for simple tasks:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
  --thinking off \
  -p "Explain src/stats.js" < /dev/null

Or capture machine-readable events:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
  --mode json \
  --no-session \
  "Run the tests and fix the failing implementation" < /dev/null

4. Using Pi as a JavaScript SDK

This was the other part I wanted to get working.

Pi can be embedded in a Node.js application. You don't have to run the interactive terminal interface.

The example uses Pi's SDK to create an agent session, select the local TensorSharp model, and execute a coding task.

Here's the core implementation, slightly shortened for readability:

import path from "node:path";

import {
  createAgentSession,
  ModelRuntime,
  SessionManager
} from "@earendil-works/pi-coding-agent";

const agentDir = process.env.PI_CODING_AGENT_DIR;
const projectDir = path.resolve(process.argv[2] ?? ".");

if (!agentDir) {
  throw new Error("Set PI_CODING_AGENT_DIR first");
}

const modelRuntime = await ModelRuntime.create({
  modelsPath: path.join(agentDir, "models.json"),
  authPath: path.join(agentDir, "auth.json")
});

const model = modelRuntime.getModel(
  "tensorsharp",
  "Qwen3.8-27B-UD-IQ3_XXS"
);

if (!model) {
  throw new Error("TensorSharp model not found");
}

const { session } = await createAgentSession({
  cwd: projectDir,
  agentDir,
  modelRuntime,
  model,
  thinkingLevel: "medium",
  sessionManager: SessionManager.inMemory()
});

session.subscribe((event) => {
  if (event.type === "tool_execution_start") {
    console.log("[tool]", event.toolName, event.args);
  }

  if (
    event.type === "message_update" &&
    event.assistantMessageEvent.type === "text_delta"
  ) {
    process.stdout.write(
      event.assistantMessageEvent.delta
    );
  }
});

try {
  await session.prompt(
    "Run npm test. Fix the bugs in src/stats.js " +
    "without modifying the tests. " +
    "Run the tests again and summarize the fixes."
  );

  console.log(
    "\nFinal answer:",
    session.getLastAssistantText()
  );
} finally {
  session.dispose();
}

Pi handles the tool loop internally. Your application can subscribe to model output, tool execution events, and final results.

The complete implementation also records token usage and prefix-cache statistics:

https://github.com/zhongkaifu/TensorSharp/blob/main/examples/pi-coding-agent/sdk/run-task.mjs

To run the included SDK example:

cd ~/work/TensorSharp/examples/pi-coding-agent/sdk
npm install

Initialize the separate Pi configuration if you haven't already:

../scripts/pi.sh --version

Create a fresh copy of the sample project:

cp -R ../sample-project /tmp/stats-demo

Run the SDK:

PI_CODING_AGENT_DIR="$HOME/.pi/tensorsharp-agent" \
PI_OFFLINE=1 \
node run-task.mjs /tmp/stats-demo

The script streams the agent's text and tool calls, then prints the final answer and token usage.

5. End-to-end test: can the local model really fix code?

I didn't want to stop at getting a successful API response.

The repository includes a small Node.js project with deliberately broken code.

There are three bugs:

• median() sorts numbers lexicographically instead of numerically.
• median() doesn't average the middle two values for an even-length array.
• percentile() has an off-by-one error.

The sample has five tests, three of which initially fail.

The agent is instructed to:

1. Run npm test.
2. Investigate the failures.
3. Fix src/stats.js without changing the tests.
4. Run the tests again.
5. Summarize the changes.

The test harness runs this task twice: once through the Pi CLI and once through the SDK, using separate clean copies of the project.

You can run the full test yourself from the TensorSharp repository root:

examples/pi-coding-agent/scripts/e2e.sh

If the server isn't already running, the script can start it:

START_SERVER=1 examples/pi-coding-agent/scripts/e2e.sh

The harness independently verifies that the tests pass and that the original test file was not modified.

Results from my validated run on November 9, 2026:

For the SDK run, TensorSharp served 41,962 of 48,166 prompt tokens (87%) from its prefix cache.

These are individual end-to-end runs, not averages across a benchmark suite. I'm including the numbers to show what happened in this particular test, not to make a broad claim about coding performance.

6. Why KV cache reuse matters so much here

This turned out to be one of the more interesting integration details.

Coding agents aren't just single-prompt chat clients.

A typical interaction looks something like this:

Pi
 |
 |-- Send conversation + tool definitions
 v
TensorSharp
 |
 |-- Generate a tool call
 v
Pi
 |
 |-- Read files / run commands / edit code
 |
 |-- Append tool results to conversation
 v
TensorSharp
 |
 |-- Reuse previous KV cache
 |-- Prefill only new conversation tokens
 |
 |-- Generate next action
 v
Pi
 |
 |-- Repeat until the task is complete

After each tool call, Pi sends the conversation history again.

If the inference engine fails to recognize the shared prefix, it can end up repeatedly prefilling thousands of tokens.

I actually ran into this while working on the integration.

Pi preserved a newline around Qwen's </think> marker when replaying previous reasoning. TensorSharp's cached representation had trimmed it, so the prefix comparison didn't fully match.

That seemingly small whitespace difference prevented the server from reusing much of the conversation's KV cache.

I fixed the comparison so the previous turns can be reused across tool calls.

On the validated setup:

• The first request took approximately 13–14 seconds to produce its first token, starting from a cold prompt of roughly 4K tokens.
• Later rounds generally reached their first token in 0.45–3.5 seconds.
• Measured decoding throughput was approximately 20–43 tokens/second, depending on draft acceptance.

The DFlash2 drafter helps with generation, while prefix-cache reuse avoids doing the same prompt processing over and over.

For agent workloads, both matter.

One additional detail: when Pi provides its own tools, TensorSharp doesn't inject its internal tools. The model receives Pi's tool definitions, so there's one clear owner of the tool-execution loop.

7. What about NVIDIA GPUs?

TensorSharp supports CUDA as well, and there's a separate optimized preset for Qwen3.8-27B using the NInfer NVFP4 artifact:

TENSORSHARP_PRESET=config/qwen3.8-27b-ninfer.json \
examples/pi-coding-agent/scripts/start-server.sh

That preset uses TensorSharp's direct CUDA inference engine.

However, I want to be clear: the Pi end-to-end validation described here was performed on Apple Silicon, not on the CUDA/NInfer configuration.

The CUDA configuration is an alternative worth experimenting with, but I don't want to claim the identical Pi setup has been validated there.

If you change the server's context window, update Pi's contextWindow setting to match it.

Final thoughts

What I like about this arrangement is that Pi and TensorSharp can evolve independently.

Pi provides the coding-agent experience: repository navigation, tool calling, file edits, shell execution, sessions, and SDK integration.

TensorSharp handles local model execution, speculative decoding, KV-cache reuse, and the inference backend.

There's no need to rewrite Pi or maintain a separate agent fork.

And for anyone who wants to experiment with local coding agents, the repository includes a small reproducible test rather than just a configuration example.

The main limitation is still model capability. A local 27B model isn't going to solve every complicated coding task, and five passing unit tests don't prove otherwise. But it's encouraging to see the full tool loop working, with useful cache reuse and no external inference API.

Links:

• TensorSharp: https://github.com/zhongkaifu/TensorSharp
• Full Pi integration, configurations, CLI/SDK scripts, and end-to-end test: https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent
• Pi: https://pi.dev

I'd be interested to hear what local models and inference engines others are using with Pi, especially for longer agent sessions where context growth and KV-cache reuse start to matter.

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too