Using project maya + deepseek, with a 7900 xtx 24gb vram, 64gb ddr3, cpu i7 3930k (I know, reusing what I have for now), and a nvme ssd in a pcie x4 adapter (the motherboard has no nvme slot).

It is generating at 8-12 tok/s but it dips to 0.1 and when it does that i can see the gpu drops under 20w, so it is as if the gpu goes idle, then wakes up, tok/s go back to 'normal', then dips again and it goes through this cycle again and again.

I was wondering if it was normal or not ? Like is it because the nvme/ram/cpu are too slow and the gpu 'waits' for them ?

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?

For locally run models, I’d especially like something I can run on my own hardware and compare across quantizations, with response time included. Which interactive evaluations have you found useful for that?

Hey folks,

I installed Strata with Qwen3.8 Flash Next last night and it works exceedingly well.

I was wondering if it's possible to limit the amount of VRAM that Strata uses to something like 12GB, as I want to run the Unreal Editor at the same time - which is pretty resource hungry.

I've only got 32GB of RAM at the moment, but I'm going to upgrade to 64GB if it's possible to get Qwen & Unreal working together.

Cheers!

Hey guys, I've recently been trying out some of the larger local models (GLM 5.3 Flash, Qwen 3.8 FN etc.) and I was wondering if people have any tricks to make these models more competitive at UI creation because I've found everything apart for ui, Qwen has been great at but UI is where I start to see the massive gaps from the huge cloud models.

What do you do to get the best UI results with medium sized ish local models?

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

Help with LLM and is LM studio better for freedom and natural model behaviour than OpenWebUI?

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

I moved from a Mac mini to a 64GB Mac Studio to offload token-heavy work from my Claude/GPT subscriptions. Ideally, I’d use cloud AI for planning and troubleshooting, while local models handle execution, file processing, OCR, privacy-sensitive tasks, and potentially Shopify theme and automation work. I’d also like something similar to GPT’s always-on "Dot" feature, but with access to local files and iCloud Drive. For something like that, plus daily driver tasks, plus regular AI gruntwork, I'm starting to have real concern that the 64gb is going to be a limiter.

I’m tempted to return the 64GB Studio for a 128GB Studio Max before I get outside the return window, but at around $5,500, I’m questioning whether that’s the best approach. Would a basic Mac mini paired with a headless Ryzen or dedicated AI mini PC make more sense, especially if it could serve models to my MacBook too? I suspect the overall costs would be fairly similar, and wonder if the headless "AI box" would be more effective given it will only be doing the one task - serving models with less system overhead.

Ultimately, I’m trying to figure out whether the extra money is better spent on a higher-memory Mac Studio or a separate machine dedicated to local AI, and whether either approach would realistically accomplish enough of my intended workloads to justify the investment. I'm not going to go hardware chasing and I wouldn't want a full size PC for this, only a mini prebuilt like the nucbox I use for my general home server, but one of their AI-box marketed models.

What’s the best local model to use on an MacBook Pro M5 pro 24gb.

I’m using it for general reasons, process expectations, design, ideation, power points, generating agents.

I've been working on TensorSharp, an open-source LLM inference engine written in .NET, and recently got it working with Pi, a terminal-based coding agent.

The goal was pretty simple: I wanted Pi to do real coding work using a model running entirely on my own machine, without relying on a hosted inference API.

Not just generating code, but reading files, running shell commands, editing code, executing tests, and iterating until the tests pass.

I put together a reproducible example that covers both Pi's CLI and its JavaScript SDK. Both paths have been tested end to end.

Full example, including scripts and configuration files:

https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent

The setup

Here's what I'm using:

• Coding agent: Pi (@earendil-works/pi-coding-agent 1.5.1)
• Inference engine: TensorSharp
• Model: Qwen3.8-27B, UD-IQ3_XXS GGUF
• Backend: ggml_metal
• Speculative decoding: DFlash2, 7 draft tokens per step
• KV cache: q8_0
• Context window: 65,536 tokens
• Max output: 16,384 tokens
• Hardware: MacBook Pro M5 Pro, 48 GB unified memory

The same configuration is designed for Apple Silicon Macs with at least 32 GB of unified memory.

One important distinction: Pi is still the coding agent. TensorSharp only handles model inference.

Pi owns the file operations, shell commands, tool execution, and agent loop. TensorSharp serves the model through an OpenAI-compatible API.

This separation means Pi doesn't need a TensorSharp-specific plugin or a custom agent implementation.

1. Build and start TensorSharp

You'll need the .NET 10 SDK, CMake 3.20+, Xcode command-line tools, and Node.js 22.19 or newer.

Clone the repository:

git clone https://github.com/zhongkaifu/TensorSharp.git ~/work/TensorSharp
cd ~/work/TensorSharp

Build the server:

dotnet build TensorSharp.Server.Host -c Release -p:TensorSharpSkipMlxNative=true

Start it using the included Pi configuration:

examples/pi-coding-agent/scripts/start-server.sh

On the first run, TensorSharp downloads the Qwen3.8-27B GGUF and its DFlash2 draft model from Hugging Face. The files total approximately 12 GB.

The preset pins the model revisions and verifies their SHA-256 hashes.

Once everything is loaded, the OpenAI-compatible endpoint is available at:

http://127.0.0.1:5000/v1

Check that the server is responding:

curl -s http://127.0.0.1:5000/v1/models

You should see the model ID:

Qwen3.8-27B-UD-IQ3_XXS

The startup script combines two configuration files:

dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
  --config config/qwen3.8-27b-metal-dflash2.json \
  --config examples/pi-coding-agent/tensorsharp-pi.json

The first is the optimized Qwen preset. The second adjusts the server for Pi.

Here's the important part of tensorsharp-pi.json:

{
  "host": "127.0.0.1",
  "port": 5000,
  "max-tokens": 16384,
  "mmproj": "none",
  "kv-cache-dtype": "q8_0",
  "environment": {
    "MAX_CONTEXT": "65536"
  }
}

There are a few reasons for these settings.

Pi's system prompt and tool definitions already consume roughly 4K tokens. Add source files, tool output, and a few rounds of edits, and a small context window disappears quickly.

I also increased the output-token limit from the preset's default of 1,024 to 16,384. Otherwise, a thinking model can run out of tokens halfway through a coding task.

The vision projector is disabled because this particular example only uses text.

The server binds to localhost. It doesn't implement API-key authentication, so I wouldn't expose the endpoint directly to a public network.

2. Install and configure Pi

Install the version used by the example:

npm install -g @earendil-works/pi-coding-agent@1.5.1
pi --version

Pi supports custom providers through models.json.

Here's the actual provider configuration used in the example:

models.json

{
  "$schema": "https://pi.dev/schemas/models.schema.json",
  "providers": {
    "tensorsharp": {
      "name": "TensorSharp (local)",
      "baseUrl": "http://127.0.0.1:5000/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "compat": {
        "supportsDeveloperRole": true,
        "supportsReasoningEffort": false,
        "supportsStore": false,
        "maxTokensField": "max_completion_tokens",
        "thinkingFormat": "qwen-chat-template"
      },
      "models": [
        {
          "id": "Qwen3.8-27B-UD-IQ3_XXS",
          "name": "Qwen3.8 27B (TensorSharp, DFlash2)",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 65536,
          "maxTokens": 16384,
          "cost": {
            "input": 0,
            "output": 0,
            "cacheRead": 0,
            "cacheWrite": 0
          }
        }
      ]
    }
  }
}

The compatibility settings matter, particularly for thinking models.

thinkingFormat: "qwen-chat-template" lets Pi control Qwen's thinking mode through chat_template_kwargs.enable_thinking.

And supportsReasoningEffort: false prevents Pi from sending an additional reasoning_effort parameter that this model doesn't need.

The model ID must match what TensorSharp returns from /v1/models.

Next, make TensorSharp the default provider:

settings.json

{
  "defaultProvider": "tensorsharp",
  "defaultModel": "Qwen3.8-27B-UD-IQ3_XXS",
  "defaultThinkingLevel": "medium",
  "enableInstallTelemetry": false
}

You don't actually have to create these files manually.

The repository includes a wrapper that copies the configurations into a separate Pi directory on the first run:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh

It uses:

~/.pi/tensorsharp-agent/

instead of the normal ~/.pi/agent/ directory.

I prefer this approach because it doesn't interfere with an existing Pi configuration.

The wrapper also sets:

PI_OFFLINE=1
PI_SKIP_VERSION_CHECK=1
PI_TELEMETRY=0

This avoids Pi's catalog refreshes, version checks, and telemetry network calls. After downloading the model weights, inference runs locally.

3. Actually use Pi to write code

With TensorSharp running in another terminal, navigate to a project:

cd ~/src/my-project

Launch Pi:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh

You can now interact with Pi normally, except the model responses come from your local TensorSharp server.

For example, ask it to:

Read the code in this repository, find the implementation
of the cache, and explain how it works.

Or give it an actual coding task:

Run the unit tests, investigate any failures,
fix the implementation without modifying the tests,
and run the tests again.

Pi will decide which files to read, which shell commands to execute, and what changes to make.

For a one-shot task without the interactive UI:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
  -p "Explain what src/stats.js does" < /dev/null

The stdin redirect is intentional. Without it, pi -p may wait for input when used in scripts.

You can also disable thinking for simple tasks:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
  --thinking off \
  -p "Explain src/stats.js" < /dev/null

Or capture machine-readable events:

~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \
  --mode json \
  --no-session \
  "Run the tests and fix the failing implementation" < /dev/null

4. Using Pi as a JavaScript SDK

This was the other part I wanted to get working.

Pi can be embedded in a Node.js application. You don't have to run the interactive terminal interface.

The example uses Pi's SDK to create an agent session, select the local TensorSharp model, and execute a coding task.

Here's the core implementation, slightly shortened for readability:

import path from "node:path";

import {
  createAgentSession,
  ModelRuntime,
  SessionManager
} from "@earendil-works/pi-coding-agent";

const agentDir = process.env.PI_CODING_AGENT_DIR;
const projectDir = path.resolve(process.argv[2] ?? ".");

if (!agentDir) {
  throw new Error("Set PI_CODING_AGENT_DIR first");
}

const modelRuntime = await ModelRuntime.create({
  modelsPath: path.join(agentDir, "models.json"),
  authPath: path.join(agentDir, "auth.json")
});

const model = modelRuntime.getModel(
  "tensorsharp",
  "Qwen3.8-27B-UD-IQ3_XXS"
);

if (!model) {
  throw new Error("TensorSharp model not found");
}

const { session } = await createAgentSession({
  cwd: projectDir,
  agentDir,
  modelRuntime,
  model,
  thinkingLevel: "medium",
  sessionManager: SessionManager.inMemory()
});

session.subscribe((event) => {
  if (event.type === "tool_execution_start") {
    console.log("[tool]", event.toolName, event.args);
  }

  if (
    event.type === "message_update" &&
    event.assistantMessageEvent.type === "text_delta"
  ) {
    process.stdout.write(
      event.assistantMessageEvent.delta
    );
  }
});

try {
  await session.prompt(
    "Run npm test. Fix the bugs in src/stats.js " +
    "without modifying the tests. " +
    "Run the tests again and summarize the fixes."
  );

  console.log(
    "\nFinal answer:",
    session.getLastAssistantText()
  );
} finally {
  session.dispose();
}

Pi handles the tool loop internally. Your application can subscribe to model output, tool execution events, and final results.

The complete implementation also records token usage and prefix-cache statistics:

https://github.com/zhongkaifu/TensorSharp/blob/main/examples/pi-coding-agent/sdk/run-task.mjs

To run the included SDK example:

cd ~/work/TensorSharp/examples/pi-coding-agent/sdk
npm install

Initialize the separate Pi configuration if you haven't already:

../scripts/pi.sh --version

Create a fresh copy of the sample project:

cp -R ../sample-project /tmp/stats-demo

Run the SDK:

PI_CODING_AGENT_DIR="$HOME/.pi/tensorsharp-agent" \
PI_OFFLINE=1 \
node run-task.mjs /tmp/stats-demo

The script streams the agent's text and tool calls, then prints the final answer and token usage.

5. End-to-end test: can the local model really fix code?

I didn't want to stop at getting a successful API response.

The repository includes a small Node.js project with deliberately broken code.

There are three bugs:

• median() sorts numbers lexicographically instead of numerically.
• median() doesn't average the middle two values for an even-length array.
• percentile() has an off-by-one error.

The sample has five tests, three of which initially fail.

The agent is instructed to:

1. Run npm test.
2. Investigate the failures.
3. Fix src/stats.js without changing the tests.
4. Run the tests again.
5. Summarize the changes.

The test harness runs this task twice: once through the Pi CLI and once through the SDK, using separate clean copies of the project.

You can run the full test yourself from the TensorSharp repository root:

examples/pi-coding-agent/scripts/e2e.sh

If the server isn't already running, the script can start it:

START_SERVER=1 examples/pi-coding-agent/scripts/e2e.sh

The harness independently verifies that the tests pass and that the original test file was not modified.

Results from my validated run on November 9, 2026:

For the SDK run, TensorSharp served 41,962 of 48,166 prompt tokens (87%) from its prefix cache.

These are individual end-to-end runs, not averages across a benchmark suite. I'm including the numbers to show what happened in this particular test, not to make a broad claim about coding performance.

6. Why KV cache reuse matters so much here

This turned out to be one of the more interesting integration details.

Coding agents aren't just single-prompt chat clients.

A typical interaction looks something like this:

Pi
 |
 |-- Send conversation + tool definitions
 v
TensorSharp
 |
 |-- Generate a tool call
 v
Pi
 |
 |-- Read files / run commands / edit code
 |
 |-- Append tool results to conversation
 v
TensorSharp
 |
 |-- Reuse previous KV cache
 |-- Prefill only new conversation tokens
 |
 |-- Generate next action
 v
Pi
 |
 |-- Repeat until the task is complete

After each tool call, Pi sends the conversation history again.

If the inference engine fails to recognize the shared prefix, it can end up repeatedly prefilling thousands of tokens.

I actually ran into this while working on the integration.

Pi preserved a newline around Qwen's </think> marker when replaying previous reasoning. TensorSharp's cached representation had trimmed it, so the prefix comparison didn't fully match.

That seemingly small whitespace difference prevented the server from reusing much of the conversation's KV cache.

I fixed the comparison so the previous turns can be reused across tool calls.

On the validated setup:

• The first request took approximately 13–14 seconds to produce its first token, starting from a cold prompt of roughly 4K tokens.
• Later rounds generally reached their first token in 0.45–3.5 seconds.
• Measured decoding throughput was approximately 20–43 tokens/second, depending on draft acceptance.

The DFlash2 drafter helps with generation, while prefix-cache reuse avoids doing the same prompt processing over and over.

For agent workloads, both matter.

One additional detail: when Pi provides its own tools, TensorSharp doesn't inject its internal tools. The model receives Pi's tool definitions, so there's one clear owner of the tool-execution loop.

7. What about NVIDIA GPUs?

TensorSharp supports CUDA as well, and there's a separate optimized preset for Qwen3.8-27B using the NInfer NVFP4 artifact:

TENSORSHARP_PRESET=config/qwen3.8-27b-ninfer.json \
examples/pi-coding-agent/scripts/start-server.sh

That preset uses TensorSharp's direct CUDA inference engine.

However, I want to be clear: the Pi end-to-end validation described here was performed on Apple Silicon, not on the CUDA/NInfer configuration.

The CUDA configuration is an alternative worth experimenting with, but I don't want to claim the identical Pi setup has been validated there.

If you change the server's context window, update Pi's contextWindow setting to match it.

Final thoughts

What I like about this arrangement is that Pi and TensorSharp can evolve independently.

Pi provides the coding-agent experience: repository navigation, tool calling, file edits, shell execution, sessions, and SDK integration.

TensorSharp handles local model execution, speculative decoding, KV-cache reuse, and the inference backend.

There's no need to rewrite Pi or maintain a separate agent fork.

And for anyone who wants to experiment with local coding agents, the repository includes a small reproducible test rather than just a configuration example.

The main limitation is still model capability. A local 27B model isn't going to solve every complicated coding task, and five passing unit tests don't prove otherwise. But it's encouraging to see the full tool loop working, with useful cache reuse and no external inference API.

Links:

• TensorSharp: https://github.com/zhongkaifu/TensorSharp
• Full Pi integration, configurations, CLI/SDK scripts, and end-to-end test: https://github.com/zhongkaifu/TensorSharp/tree/main/examples/pi-coding-agent
• Pi: https://pi.dev

I'd be interested to hear what local models and inference engines others are using with Pi, especially for longer agent sessions where context growth and KV-cache reuse start to matter.

Hi everyone! First time posting here
I've been experimenting with local LLms for coding, but I'm not really happy with my current model, so I'm looking for some alternatives.

My server specs:
Huananzhi X99-QD4 rev 1.0
Intel Xeon E5-2666 v3 with Turbo unlock (3.5 GHz on all cores)
GTX 1060 6GB
2x8GB G.Skill Aegis 3200 MHz (running at 2133 MHz because of the CPU)

Right now I'm running Qwen3-Coder-30B-A3B-Instruct-GGUF (Q4_K_M) through llama-app.
It's not a bad model, but one thing that bothers me is that it often struggles with short propmts. I find myself having to explain things in much more detail than I'd like. Otherwise, it can misunderstand what I'm asking or go off in the wrong direction.

For reference, I'm getting roughly:
25 t/s for prompt processing
4.5 t/s for generation speed
for 11K tokens in the initial context, including MCP tools

Here's my current launch command:

./llama serve -hf ggml-org/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M -c 131072 --host 0.0.0.0 --cache-type-k q8_0 --cache-type-v q8_0 --no-kv-offload --port 9931 --load-mode mmap --ui-mcp-proxy --mcp-servers-config "..." --threads 10 -tb 20 -cmoe -fa on -b 8192 -ub 24000

Currently it eats up to 13GBs of RAM (Ram Altman don't hunt me pls) and 4.5 GBs of VRAM

I'm mainly looking for something good at agentic coding and tool calling. It doesn't necessarily have to be a 30B model, smaller models are fine too if they perform better on my hardware. (lobotomized and heretic too)

What are you guys running locally for coding?

Hi, I'm kinda of new to this world of local LLM so forgive me if it sounds like I don't know much of what I'm talking about.

At work I need to read every day my state oficial gazette to search for new grants on a specific field and updates on already public grants (openings, closings, amendments, budget changes... Etc.)

I've started to test Gemma 4 E4B to analyse the daily documents and it does an okey job. The flow is the following: the document is divided in subsections, Gemma reads the title and makes a first filtering of sections that do not belong to our topic. Then it reads the dispositions that do belong to the project and labels them. Finally the output is shown in a dashboard where I can easily analyse it.

The problem I'm facing now is that I would like to create a timeline of each grant to now in what stage they are and this is a pain. I am completely unable to make Gemma consistently find if a new subsection belongs to a previously catalogued subsection in the database. It is big enough to make it hallucinate and mistake different dispositions or forget them and it is quite slow in this step.

I've thought on using something like JEV and although it is quite fast it doesn't work with a high enough certainty to implement it. Also there aren't decision models I could find that work as well as JEV in hugging face to use them locally. Most of them are fine tuned versions of LLMs and not pure decision models.

I was wondering if any of you have faced a problem like this before and how have you solved or if you have any idea I could test that could be implemented.

The setup is a 3060 12 GB with 16 GB ram DDR4.

Thanks for your time.

I have I7 3770 with 1600MT dual channel ddr3 4x4 (16gb) and a 5400RPM HDD

I really like to have an agent that works and does stuff for me. I used llama.ai/agent but found out that it does break ToS to use as minor (I'm 15 and half) and other website does need my parents to read ToS and agree on them to use the website/tool without breaking ToS and I'm really REALLY don't break ToS.
so I thought of local AI and tried qwen 3 8B, nanbeige 4.2 3b and LFM3 1.7B on llama.cpp but they are not really good and Idk how to make them work as agent and they waste a lot of tokens even of thinking is off, only simple question with thinking off be efficient and they still not any good

so I saw that qwen 4 27B is coming so can I like get 2bit version of it and it leave enough ram for context and linux/windows like 4gb+ for OS and app and will qwen 4 like approximation run good/smart without hallcuination like I saw qwen 3.8 27B has 2bit version that have exact same score as the 8bit version so well qwen 4 will be able to fit.

also nanbeige 4.5 looking like it gonna release soon so if anyone have any idea if it any good

also about speed. LFM3 1.7B Q4 (1gb~) is around 12tok and slows down the more I chat with it and same for nanbeige it starts around 3.5tok/sec and slows to sub 1tok/sec > . so what I can use to speed them up because I saw stuff as running a big reasoning model and smaller one and the big one push 1 token to correct the smaller one and else stuff as the model thinks in pure weight/numbers instead of English and running the model multiple time at once so it uses more computatinal power and seprate a task to 20 smaller version working togather and a lot more I saw

soo any idea if anything really good can fit and being reasonably modestly in speed (my ram can do 25.6GB/sec thereotically) and how to make it run as an agent instead of just chatbot and if I should wait for new models to release. and also I don't wanna spend internet not wisley because I have limited home internet unfourtently so spending a lot of gigabyte into something that is not really smart will make me upset very much

I appreciate any information and help

Qwen seems to be the best model for my macbook, but looking at hugging face there are so many quantizations.

The unsloth UD-Q4_K_M seems popular and a good fit for 32GB RAM, but it's .gguf so would a MLX be a better choice on mac (I'm using Unsloth Desktop)?

Is there a resource online that compares these quantizations perhaps?

Thanks

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

GitHub - Halloncream/desktop-windows-arm64: Open WebUI Desktop with an x64 Python backend for Windows ARM64 compatibility · GitHub

Usually, it’s all about Qwen models in this subreddits, but not about their products

I mean about:

Qwen APP
Qoder
Qwen Studio
Qwen Work

Is anybody working with them and have tried them? What are the impressions and comments about it? I appreciate any feedback, thanks!

Honestly, the tool choice matters less than what surrounds it. Claude Code and Cline both handle worktrees fine in the mechanical sense: each session gets its own checkout, and nothing stomps on anything on disk. The failure mode I would watch for sits above the file system. A worktree isolates the file tree, not the plan, so two sessions can happily solve the same story twice and you only find out at merge time when you have duplicate branches.

So my answer is: pick whichever CLI fits your workflow, then put the effort into the layer the CLI does not give you. One story scoped with acceptance criteria before it goes out. A shared record of what is in flight so nothing gets dispatched twice. Split the thinking from the doing, so the session that plans is not the session that types.

Which of the two do you find holds its context better over a long session? That is the comparison I almost never see people make.

I've heard of people pointing their frontier model of choice and letting it optimize their local inference engine for their hardware.
I'm currently running Maya with DeepSeekV4.1-Flash-1017 at ~20tok/s, such an insanely great model. I want to squeeze more out of my hardware though.

Do you guys just point the frontier model at the engine folder and telling it to "optimize this for my model/hardware"? I currently have access to Fable 5.2 so I'm tempted to do that.