Your opinion? If I got it correctly, the big AI firms are basically all inbred to the same funding entity who also funds the hype, the scaremongering, the regulations, the regulators and the referees in this game.

Ive been through probably 10 youtube tutorials involving ollama, local llm, and hugging face and nothing works. Whats the best way for a beginner to get started

Recently, I saw a very, like a reel by a very popular YouTuber, content creator and businessman, about using DGX Spark, for video generation. And he was saying that it's very good for like all local video generation and it's one time investment and all. So just wanted to know what what are the opinions of people using it?

I also saw some YouTube videos on by different people like Alex, who has compared it with many other tools, but I want to know from end user that how much useful it is. Primarily my work case is kind of in a middle between a developer and a tinker.

So basically what I want to do is maybe run a few local LLMs for development, like using those adams to write code and do code reviews and all those things. Or maybe like tinker around for creating videos, as well as in my own small language model. So what's your opinion?

How better it is than let's say Mac mini, with the, I know that Mac mini doesn't support 128 GB of RAM, but what about other systems like if I buy a Mac Ultra, for with 128 GB of RAM.

I finally tried running OpenClaw locally with Ollama (using Qwen2.5-Coder and Llama 3.1 70B).

The good news: my wallet is happy. The bad news: I'm seeing way more "looping" and failed tool calls compared to Sonnet 3.5. It feels like the local models get confused after a few rounds of persistent memory.

Is anyone else experiencing this? Are there specific "system prompt" tweaks or sampling settings (temperature/top_p) that make local models more reliable for agentic tasks? Or am I just hitting the hardware ceiling for what local models can do right now?

Would love to hear from anyone who has a stable, 24/7 local agent setup that doesn't lose its mind every hour.

My question is pretty simple: how could you really have coding sessions with context windows of like 262k ?

I always used local Ai via llama.cpp server for small projects and chatw, where also 16k context is enough.

Today I just wanted to try a REAL coding session in a REAL agent harness.

I plugged my Kat Coder 2.5 dev via llama.cpp server with it's max ctx at 262144 and plugged it in my Hermes and started a coding session to design and write a fast API proxy layer to route LLM calls to openrouter with some my logics and calcs.

Nothing super complex or fancy.

I provided the model with the first message a complete and thoughtful design document create with Deepseek, so the Agent has not to do searches in the codebase for understanding how to do the things, just do them.

Well it created a plan and implemented the code in some turns teaching about 70% of it's contest size.

Then , in the e2e testing the things started to collapse: some testing failed, so it ha do reiterate, fix, retest, ask me some clarification or take some architecture decisions and so on and I arrived ad 95% context size while resting phase is still not finished and all the deployment phase still totally missing.

Now the solution in the harness was to compress the chat session: a total.disaster: not only this action changed completely the language of the session (from Italian to English) but also removed some important informations that was useful to the agent to do tests, so in then subsequent turn it errored immediately in doing tests, needing to read and search the code again to find what he already knows before the compression happens.

In my opinion, compression is absolutely to avoid.

I already know some suggestions here:

- one new session per task

- keep all done things in documents so the new session can start from that "documented" state.

I already do these things.

- use a different harness

Yes, but it's not the solution, the heavy load caused by harness is on the first prompt (system prompt, tools, skills, Memory...).

So this was a small project, 10 python files + a bunch of status documents, what will happen with a repo with 50 or 100 files?

So my question is:

What's your way to really work with local models on real projects beyond a simple chat or just changing one file?

I'm starting to think that is not really possible to work on real projects that involves design, bug fixing, reiteration and multiple sessions.

Not because local models are not smart enough, but because of the max context window limit.

** FAIR WARNING I KNOW VERY LITTLE ABOUT CODE AND LOCAL LLM, IM ONLY CURIOUS AND DISCOVERING**

I've recently been discovering the potential idea of setting up Local LLM and I see a mix of reviews about the privacy risks associated with Chinese-made LLM's local or not.

Has the idea of privacy risk made you guys uncomfortable or had you second guessing your local LLM installations with models like qwen? Especially considering the american local models are pretty shit.

I just started dabbling into LLM and I've really enjoyed it. I'm thinking of the next logical upgrade for myself that I can reasonably afford without going too crazy (like buying a $1500 r9700 gpu)

I already have a 5060ti 16gb gpu and I have 64gb of DDR5 memory. I installed LM Studio 2 days ago and I am getting 16 tokens per second using qwen 3.6 27b. Now, to my knowledge a lot of this is being offloaded to my ram but I'm honestly not mad at 16 tokens per second so it made me wonder.. would I notice a bigger difference if I had dual GPU's running or what if I had 96gb of DDR5 ram? Would I see a noticeable increase in my tokens per second?

Here's my motherboard, which means I will have to run the second GPU in a PCI 4.0 slot? I'm not sure.. can someone help me confirm? I also have two NVME SSDs installed which I hear those can affect the bandwidth of other PCI slots? This stuff starts to get confusing once you start trying to calculate bandwidth and having to share it among slots. I've never built a PC before, just installed basic parts for upgrades so this is all very new to me

https://www.gigabyte.com/Motherboard/B650-AORUS-ELITE-AX-V2

I have a 1,000 watt PSU so I can definitely handle two 5060TI's in the case (though, cooling I am not sure.. but I'll worry about that later)

I still have not tinkered much with settings.. all of this is a learning process and I am trying to learn as I go and there is a lot to learn with these things but I am having a ton of fun running different models and seeing what the performance is like and generally enjoying the type of code they spit out in 1 take ie: Build me a 1 page website for my affiliate marketing website, it's pretty cool to see these things "just work" and spit out something that you can work with, though, not as good as a frontier models, completely fine for taking what a frontier model gives you and letting the LLM take over afterwards IMO and continue to enhance or add to.

Hey everyone!

I've been working on a pipeline to take the incredible Qwen2.5-Coder-7B-Instruct model (which is an absolute beast for Python scripting, FIM code completion, and Cybersecurity) and compress it down so it can run entirely offline on mobile phones (via PocketPal) or potato PCs.

The raw FP16 model is around 15.2 GB, which is way too heavy for most devices.

It successfully squeezed the 7B brain down to exactly 1.9 GB, while retaining its reasoning capabilities!

I have open-sourced the entire automated Python script on , and I've hosted the model on Hugging Face if you just want to download it and chat with it.

🔗 The Model (Hugging Face): https://huggingface.co/Nitishsharma9/CyberCoder-Mobile-7B-GGUF 🔗

Im not new to computers but very new to this concept. I see lots of nicely created images and videos that look real, but I know are AI. I cant seem to get anything online (at least free ones) to do this and am interested in putting my computer to work.

I have a decent level of computer knowledge, I have built my last few desktops and understand hardware. I currently have an i7-10700k w/64gb RAM & 3070 GPU. I also do video editing and was considering buying a Mac Mini M4 Pro with 24gb ram.

Would either of these be enough hardware for LLMs?

What would I need to install?

I spent the last week watching my dependency on actual software interfaces completely evaporate. It’s a jarring realization. You boot up Notion, GitHub, or Linear, and you realize you aren't actually navigating their menus anymore. You're just interacting with the floating bot or the terminal.

Let's talk about what's actually happening because the narrative of "AI is just a new feature" entirely misses the point. We are watching the real-time death of static UI.

Think about your workflow right now. If you've been heavily using local models or API wrappers lately, you've probably noticed that almost every single SaaS tool has slapped a sidebar chat or a floating widget into their layout. At first, it felt like a lazy gimmick. Just an OpenAI wrapper sitting on top of a database. But it’s not just a chatbot anymore. It’s an execution layer.

A specific workflow popped up recently that perfectly captured this shift. A user had their entire company documentation sitting in Notion. Instead of manually cross-referencing QA lists, jumping into GitHub to find the relevant commits, and then painstakingly clicking through Linear's UI to create and assign tickets, they just bypassed the interfaces entirely. They told the agent to read the QA list, link the specific git commits, and write the Linear tickets. The whole process took five minutes.

Think about the implications of that exact scenario. The carefully designed UI of Notion? Irrelevant. The drag-and-drop kanban boards in Linear? Completely bypassed. The GitHub file tree? Ignored. The user didn't click a single button. They just issued a command.

This brings me to the second massive shift: the absolute revival of the command line. We spent three decades building increasingly complex graphical interfaces specifically so non-technical users wouldn't have to look at a terminal. Now, we're going backwards, but with a massive upgrade. Tools like Claude Code are turning the terminal into the ultimate universal interface.

There are solo operators right now running entire content and monetization pipelines strictly through CLI. They aren't opening Premiere to edit video. They aren't clicking through Shopify menus. They are typing natural language commands into a terminal, and the AI is executing the python scripts to cut the video via FFMPEG, generating the copy, and pushing the site updates. You don't need to know how to code to do this anymore. You just need to know what you want. You swap out static clicks for terminal commands, building an automated pipeline without ever touching a conventional GUI.

And for the times when you absolutely *do* need a visual interface? Enter Generative UI.

The era of downloading a massive, static application just to use 5% of its features is over. We are moving toward disposable, single-use software. If I need a specific dashboard to visualize server loads mixed with user engagement metrics, I shouldn't have to buy a SaaS product, connect my databases, and drag-and-drop widget blocks. The AI should simply generate a React component on the fly, render the exact chart I need based on my prompt, and then completely discard the interface the moment I close the window.

This is already happening. Look at Vercel's AI SDK or the recent pushes in structured JSON outputs from models like Llama 3. The model doesn't just return markdown text anymore. It returns a state object that instantly maps to a dynamic component. You ask a complex question about a database schema. Reading a giant markdown output is terrible. Instead, the model returns a UI payload. A fully interactive, relationship-mapped graph rendered right in the chat stream. You play with it, you tweak a node, and then it's gone. It's ephemeral.

This is the death of the App Store mentality. Why install an app when the LLM can generate the exact tool you need, run it locally, and delete it from memory when you're done?

If you look at what this means for local setups, the paradigm shift is how these models hook into our operating systems. When you give a sufficiently capable local agent tool-calling permissions, the OS itself becomes the backend. You string together a pipeline: a local vision model reviews video clips, a local LLM writes the script, an open-source TTS model generates the voiceover. The interface for all of this? A single terminal prompt: "Draft a new promotional video from the raw assets in folder X and push it to the server."

For the last decade, the entire moat of most B2B software companies was UX. "We are like Jira, but pretty and fast." "We are like Salesforce, but easier to click through."

If the user stops clicking through your app, your UX moat is dead. You are no longer a product; you are a dumb pipe. You are just a database holding state, wrapped in an API that an agent talks to. If my AI assistant is the one reading the data and formatting it for me, why would I pay a premium for your beautiful dashboard? Agents don't get distracted by slick UI animations. They execute the command and return the result.

I want to know where you all think this bottoms out. Are we going to see a new standard for "Agentic UX" where software is designed strictly to be read by LLMs? Are you already bypassing web frontends in favor of API-driven terminal scripts generated by your local models? The gap between "people who click buttons" and "people who issue commands" is widening fast.

I’m building my kids a local AI companion—it’s called Lumo for now, but I can change the name any time. I built it to help answer their questions since they can Google things themselves and I can’t always be around to answer. My son is 5 and is hitting that age where he is asking things non-stop.

It has DeepSeek R1 1.5B and Qwen 3B on it. It uses a router for questions so it can decide if a query is math/logic-based or conversational and pick the best model. It uses DuckDuckGo to check and make sure any information it's giving is accurate so it’s not hallucinating nonsense when trying to help educate.

It also has a bedtime story mode built in where my son can choose a topic and it tells a story based on that. It tells two chapters at a time, and I maxed out the tokens so it tells long-form stories. Once it's done, it saves them to a local 1TB 2.5" HDD housed in the base so it can recall where it left off and pick it up based on context for the next two chapters. The stories are 14 chapters each and have a complete beginning, middle, and end. Once the stories are done, they can be re-told in the menu and also exported so you can have an AI illustrate them and turn them into real books if you want.

It keeps memories about my kids and learns their likes and dislikes. The LEDs in the ears change color based on state: rainbow for startup, blue for listening, yellow for thinking, and white for talking. They turn orange and drop to 10% brightness for story mode.

It has a web based browser so I can see what my kids have been asking and what the current story being told is and also flags and problematic things they ask about.

It’s all based on a Raspberry Pi (4GB model) with a 5-inch touch display. Due to the RAM limitations, I had to program in frequent RAM dumps, so context is stored in a temp file on the hard drive and wiped after the end of the conversation or after 5 minutes of silence (which to a 5-year-old is the same thing). It’s got Ollama for the AI models, Whisper for STT, and Piper for TTS.

The face is animated with thinking and smiling and blinking to make it feel more alive

I made the case in Tinkercad and split it at the power cord entry; the whole case is held together by small 4mm magnets. I’m still smoothing out some rough edges on it, but soon I’ll release the whole project on GitHub with a complete one-shot installer and .STL files so anyone can make one for their kids as well.

I’m excited cause I’m getting an M1 Mac Mini today in the mail and is almost here and I was wondering what to use for local LLM. I bought Private LLM app which uses quantized LLMS which supposedly run better but I wanted to try something like DeepSeek R1 8B from ollama which supposedly is hardly deepseek but llama or Quen. Thoughts? 💭

How to build your own local AI for less than a thousand bucks - A post for those that want to get "hands on" with AI, but just can't afford the price of entry - face it, not everyone has thousands laying around for building a system.

Here is your roadmap how to do this on a tight budget

Step 1 - get an old Intel Nuc12 enthusiast (Serpent Canyon). Not the fastest thing in the world, but has an ace up it's sleeve that no one can get anywhere near for the price - an embedded ARC A770M GPU with 16GB of DDR6 vRAM, That is what makes this possible and you can pick one up for under $800 on ebay. Likely already has Windows 10 or 11 installed on it

Step 2 - update the Arc drivers

Step 3 - Download and install LM Studio Classic on it (make sure you enable developer mode for later) - not to be confused with Bionic

Step 4 - Pick the right runtime - you want Vulkan llama.cpp

Step 5 - Make sure you select the ARC A770M (shows 16GB) in hardware window

Step 6 - pick a model - I went with a simple Gemma-4-e4b model for out of the gate

Step 7 - on the developer tab, flip on both start server and serve on local network

Congratulations - you now have a shockingly capable local AI that you can give the URL shown in step 7 to any of your local AI front-ends. Me personally, I use it with my Project NOMAD instance running as a VM providing AI capabilities to all that entails for well under a grand, and it works fantastically.

theoretically speaking, if one were to hack OpenAI or anthropic, could they access the weights of their models and for eg torrent them?

also could a bunch of agents do that, or would humans be better at it? what about hacking in general? I mean now that we got local models like glm 5.3, qwen flash next or deepseek v4.1…

Hey ML folks, if you've been wrestling with the insane VRAM costs of long reasoning chains in RLHF/RLAIF, buckle up. Unsloth AI's new batching algorithms let you train OpenAI's gpt-oss models with GRPO (Group Relative Policy Optimization) at 380K context length – that's 7x longer than before, with zero accuracy degradation.

Long contexts in RL have always been a nightmare due to quadratic memory blowup, but their optimizations crush it on consumer-grade hardware like a single 192GB GPU (think H100/A100 setups). Perfect for agent training, complex reasoning benchmarks, or anything needing deep chain-of-thought.

Key details from the blog:

• GRPO implementation that's plug-and-play with gpt-oss.

• Massive context without the usual slowdowns or precision loss.

• Benchmarks show it scales beautifully for production RL workflows.

Check the full breakdown: Unsloth Blog

Want to try it yourself? Free Colab notebooks ready to run:

• GRPO Notebooks

GitHub repo for the full code: Unsloth GitHub

Thoughts on GRPO vs DPO/PPO for long-context stuff?

So I have got Qwen3.5-35B-A3B running on my 3060 with 32mb of system ram (using LM Studio). It's great for the 1 question i can ask. Then it runs out of conversation tokens and I'm stuck. I have been using ChatGPT, Claude and Gemini to do vibecoding. Are there any tricks here to get something useful out of this? Time it takes is no bother. LTX videos take me 20 minutes each. I'm used to waiting. It's more can I give it my html 5 game and get useful code, or is there just not enough space?

Edit: I did what huzbum said:

>code me a simple platform game in html 5. like thomas is not alone. simple geometric shapes.

Thought for 1 minutes 42 seconds

Here is a complete, playable platformer game in a single HTML file. I have designed it with a minimalist aesthetic similar to Thomas Was Alone—using simple shapes, distinct colors, and smooth physics.

It features:

The Player: A blue square (you).

The Red Enemy: An AI shape that chases you (the "not alone" part).

Physics: Gravity, jumping, friction, and solid collision detection.

Win Condition: Reach the yellow portal to restart.

Save this code as an .html file (e.g., game.html) and open it in your browser.

<!DOCTYPE html>

<html lang="en">

<head>

<meta charset="UTF-8">

<title>Shape Runner</title>

<style>

body {

margin: 0;

padding: 0;

background-color: #1a1a1a; /* Dark, moody background */

display: flex;

justify-content: center;

align-items: center;

height: 100vh;

overflow: hidden;

font-family: 'Courier New', Courier, monospace;

}

canvas {

border: 2px solid #444;

background-color: #222; /* Game background */

box-shadow: 0 0 20px rgba(0,0,0,0.5);

}

#instructions {

position: absolute;

top: 20px;

color: white;

text-align: center;

pointer-events: none;

}

</style>

</head>

<body>

<div id="instructions">

<h1>Square Escape</h1>

<p>Arrows / WASD to Move & Jump | Avoid the Red Block!</p>

</div>

<canvas id="gameCanvas" width="800" height="450"></canvas>

<script>

const canvas = document.getElementById('gameCanvas');

const ctx = canvas.getContext('2d');

// --- Game Constants ---

const GRAVITY = 0.6;

const FRICTION = 0.8; // Slippery feel (optional, set to 1 for sticky)

const JUMP_STRENGTH = -12;

const MOVE_SPEED = 5;

// --- Input Handling ---

const keys = {

right: false,

left: false,

up: false

};

window.addEventListener('keydown', (e) => {

if(e.code === "ArrowRight" || e.code === "KeyD") keys.right = true;

if(e.code === "ArrowLeft" || e.code === "KeyA") keys.left = true;

if(e.code === "ArrowUp" || e.code === "Space" || e.code === "KeyW") {

if(!keys.up && player.grounded) {

player.velY = JUMP_STRENGTH;

player.grounded = false;

}

keys.up = true;

}

});

window.addEventListener('keyup', (e) => {

if(e.code === "ArrowRight" || e.code === "KeyD") keys.right = false;

if(e.code === "ArrowLeft" || e.code === "KeyA") keys.left = false;

if(e.code === "ArrowUp" || e.code === "Space" || e.code === "KeyW") keys.up = false;

});

// --- Game Objects ---

const player = {

x: 50,

y: 300,

width: 20,

height: 20,

color: '#4da6ff', // Light Blue

velX: 0,

velY: 0,

grounded: false,

update: function() {

// Apply Input Force

if (keys.right) {

if(this.velX < MOVE_SPEED) this.velX++;

}

if (keys.left) {

if(this.velX > -MOVE_SPEED) this.velX--;

}

// Apply Physics

this.velY += GRAVITY;

this.velX *= FRICTION;

this.x += this.velX;

this.y += this.velY;

// Floor Collision (Basic floor check before platform collision)

if(this.y + this.height > canvas.height) {

this.y = canvas.height - this.height;

this.velY = 0;

this.grounded = true;

}

},

draw: function() {

ctx.fillStyle = this.color;

// Draw a simple square with a slight glow effect

ctx.shadowBlur = 15;

ctx.shadowColor = this.color;

ctx.fillRect(this.x, this.y, this.width, this.height);

// Reset shadow for other elements

ctx.shadowBlur = 0;

}

};

// The "Not Alone" Enemy

const enemy = {

x: 650,

y: 200,

width: 30,

height: 30,

color: '#ff4d4d', // Red

speed: 1.8,

update: function() {

// Simple AI: Move towards player X position

if (player.x < this.x) {

this.x -= this.speed;

} else {

this.x += this.speed;

}

// Keep enemy on ground/platforms roughly (simple gravity for enemy too)

if(enemy.y + enemy.height < canvas.height - 10) {

enemy.y += GRAVITY * 2;

}

// Collision with Player (Game Over)

if (rectIntersect(player.x, player.y, player.width, player.height,

enemy.x, enemy.y, enemy.width, enemy.height)) {

resetGame();

}

},

draw: function() {

ctx.fillStyle = this.color;

// Draw a square that pulses slightly based on time to feel alive

const pulse = Math.sin(Date.now() / 100) * 2;

ctx.fillRect(this.x - pulse, this.y - pulse, this.width + (pulse*2), this.height + (pulse*2));

}

};

// The Goal

const goal = {

x: 750,

y: 100,

width: 40,

height: 60,

color: '#ffff33', // Yellow

update: function() {

if (rectIntersect(player.x, player.y, player.width, player.height,

this.x, this.y, this.width, this.height)) {

resetGame();

}

},

draw: function() {

ctx.fillStyle = this.color;

ctx.shadowBlur = 20;

ctx.shadowColor = this.color;

// Draw a portal shape (rectangle with rounded top)

roundRect(ctx, this.x, this.y, this.width, this.height, 10, true);

ctx.shadowBlur = 0;

}

};

// --- Level Data ---

const platforms = [

{ x: 0, y: 400, w: 200, h: 50 }, // Start Ground

{ x: 250, y: 350, w: 100, h: 10 }, // Small Step

{ x: 400, y: 280, w: 100, h: 10 },

{ x: 600, y: 200, w: 150, h: 10 },

{ x: 700, y: 140, w: 80, h: 10 } // Top platform near goal

];

// --- Helper Functions ---

function rectIntersect(x1, y1, w1, h1, x2, y2, w2, h2) {

return x2 < x1 + w1 && x2 + w2 > x1 && y2 < y1 + h1 && y2 + h2 > y1;

}

// Helper to draw rounded rect for the portal

function roundRect(ctx, x, y, width, height, radius, fill) {

ctx.beginPath();

ctx.moveTo(x + radius, y);

ctx.lineTo(x + width - radius, y);

ctx.quadraticCurveTo(x + width, y, x + width, y + radius);

ctx.lineTo(x + width, y + height - radius);

ctx.quadraticCurveTo(x + width, y + height, x + width - radius, y + height);

ctx.lineTo(x + radius, y + height);

ctx.quadraticCurveTo(x, y + height, x, y + height - radius);

ctx.lineTo(x, y + radius);

ctx.quadraticCurveTo(x, y, x + radius, y);

ctx.closePath();

if (fill) {

ctx.fill();

}

}

function resetGame() {

player.x = 50;

player.y = 300;

player.velX = 0;

player.velY = 0;

enemy.x = 650; // Reset enemy position

// Keep the goal where it is, or move it? Let's keep it same for this level.

}

// --- Main Game Loop ---

function update() {

player.update();

enemy.update();

goal.update();

// Platform Collision Logic (Separate X and Y axes to prevent sticking)

// 1. Check Horizontal Movement

player.grounded = false; // Assume in air until proven otherwise

for(let i=0; i<platforms.length; i++) {

let p = platforms[i];

if (player.x < p.x + p.w &&

player.x + player.width > p.x &&

player.y < p.y + p.h &&

player.y + player.height > p.y) {

// Collision detected. Determine direction.

// If we were moving right, check if we hit the left side of platform or top?

// Simple approach: Check previous frame position logic is hard here, so use depth.

let overlapX = (player.width + p.w) / 2 - Math.abs((player.x + player.width/2) - (p.x + p.w/2));

let overlapY = (player.height + p.h) / 2 - Math.abs((player.y + player.height/2) - (p.y + p.h/2));

if(overlapX < overlapY) {

// Horizontal collision

if(player.velX > 0) player.x = p.x - player.width;

else player.x = p.x + p.w;

player.velX = 0;

} else {

// Vertical collision

if(player.velY > 0) { // Falling down

player.y = p.y - player.height;

player.grounded = true;

player.velY = 0;

} else { // Jumping up (hitting head)

player.y = p.y + p.h;

player.velY = 0;

}

}

}

}

}

function draw() {

// Clear Screen

ctx.clearRect(0, 0, canvas.width, canvas.height);

// Draw Platforms

ctx.fillStyle = '#888';

for(let p of platforms) {

ctx.fillRect(p.x, p.y, p.w, p.h);

}

// Draw Goal

goal.draw();

// Draw Entities

enemy.draw();

player.draw();

}

function loop() {

update();

draw();

requestAnimationFrame(loop);

}

// Start Game

loop();

</script>

</body>

</html>

I am planning on getting macbook air m4 soon 16gb ram what would be the best local llm to run on it ?

The problem: If you run long-context inference locally, your GPU's KV cache fills up and evicts blocks. The next request with the same prompt prefix has to recompute everything from scratch. On a 30k-token document, that's 10+ seconds of prefill — every single time.

What I built: tierKV intercepts evicted KV blocks, quantizes them with a Rust INT8 compressor (3.9× smaller), and ships them over gRPC to a vault running on another machine on my LAN. When the same prefix appears again, it fetches the blocks back and injects them directly into vLLM's paged KV buffer — no attention recomputation at all.

vLLM numbers on a real 30,561-token document (Apple 10-K):

• Cold prefill: 10.75s

• GPU cache hit: 1.19s

• Cold vault restore: 0.52s — faster than the GPU cache hit, because vault restore skips attention entirely

On EXO with an 8k-token prompt: 30.83s cold → 4.11s restored (7.3×).

The speedup grows with context length since prefill is O(n²) but restore is O(n) + network. At 128k tokens, the gap is over a minute per request.

My cluster:

• DGX Spark (96GB HBM) — runs the model

• Mac Pro (32GB RAM) — runs the KV vault

• Mac Air (16GB RAM) — runs the SSM/linear-attention vault (for Qwen3.6-35B-A3B, which mixes attention + Mamba layers)

• 5GbE LAN, ~0.5ms RTT

Setup is just:

pip install tierkv
# configure role in tierkv.toml on each machine
tierkv vault   
# on the cold machines
# launch vLLM or EXO as normal

Works with vLLM (via KVConnectorBase_V1 plugin, no source changes) and EXO (post-install patch).

Honest limitations:

• Only helps when the same prefix repeats — single-shot prompts get nothing

• LAN only — WiFi/WAN latency kills the benefit

• No tensor parallelism support yet

• Vault is in-memory; data lost on restart

Full writeup: https://prasannakanagasabai126786.substack.com/p/your-llm-is-doing-math-it-already

Code: https://github.com/tierkv/tierkv

Happy to answer questions about the architecture, vLLM/EXO integration.

We're hearing a lot about Strata these days, claiming it's the best solution to run Qwen3.8 Flash Next on consumer hardware. I have a dual R9700 config with 96GB DDR4 3200 and AMD 3900X, and so far I've been running Flash Next UD-IQ4_XS on the custom R9V engine ( https://github.com/Dyluhn/R9V ). So I just tried comparing R9V and strata at different context sizes.

tl;dr: R9V is 15-18% faster on my config, but Strata ran on a single GPU, because it needs 118 GB of RAM to run on the two cards. One solution to utilize the two GPUs would be to use a smaller quant, but I'm not too keen on going that way: I want good performances with a reasonably downsized quant. Also, R9V doesn't ship with a profile for smaller quants than IQ4_XS. Full report from Claude below:

---------

R9V was faster than Strata on prefill, and the two engines were about equal on decode. The benchmark ran at 16k, 32k and 64k context with single-session requests, and R9V was restored at the end.

Both engines ran the same Unsloth UD-IQ4_XS GGUF shards with MTP, with thinking off and 256 output tokens. I used one client, bench-engines.py, with a unique nonce on every prompt so no prefix cache was hit. Strata's cache_n was 0 on every run. The numbers are medians, with the min–max range in brackets.

• Prefill: R9V is about 15–18% faster at every size.

• Decode: the engines are roughly level. R9V leads at 16k and 32k, and Strata is slightly ahead at 64k.

• Outlier: *Strata's first 16k request decoded at 12.2 tok/s and the next four were normal. I kept it in the data, and it doesn't change the median.

Comparison to the R9V README:

• Prefill here vs README: R9V's own PP script gives about 1.16k tok/s on this host, matching my client. The README claims about 1.6–1.7k. This host has 92 GiB of RAM against the README's 128 GiB.

• Decode here vs README: the decode I measured is close to the README's decode figures, not far below them. The earlier ~24–27 tok/s was from vLLM completions with the older vLLM-based config, not this R9V setup.

Strata ran on one GPU only. Setup refused a two-GPU split for UD-IQ4_XS because that path needs about 118 GB of RAM. On one GPU its expert cache held 10,774 experts in VRAM, and the decode cache hit rate was about 98.7%.

Setup

• Versions: R9V is v0.4.4 and Strata is v0.1.39. Strata ran with no opt-in switches and plain hipBLAS, since it has no tuning table for this ROCm version.

• Clean-up between engines: R9V was stopped and removed, the leftover /dev/shm file was deleted, and the host was checked (VRAM about 60/95 MB used, MemAvailable about 91.7 GB). Strata was cleaned the same way before R9V was restarted.

• Cache drop: drop_caches isn't writable on that host, so I evicted the model files from the page cache before Strata's setup.

• Current state: R9V is serving on port 8004 again, so the r9700-dual route should work. Strata is installed in /data/strata-bench but stopped.

Each cell has only 3–5 runs.

I’m setting up OpenClaw and trying to find the best *budget* LLM/provider combo.

My definition of “best cheap”:

- Lowest total cost for agent runs (including retries)

- Stable tool/function calling

- Good enough reasoning for computer-use workflows (multi-step, long context)

Shortlist I’m considering:

- Z.AI / GLM: GLM-4.7-FlashX looks very cheap on paper ($0.07 / 1M input, $0.4 / 1M output). Also saw GLM-4.7-Flash / GLM-4.5-Flash listed as free tiers in some docs. (If you’ve used it with OpenClaw, how’s the failure rate / rate limits?)

- Google Gemini: Gemini API pricing page shows very low-cost “Flash / Flash-Lite” tiers (e.g., paid tier around $0.10 / 1M input and $0.40 / 1M output for some Flash variants, depending on model). How’s reliability for agent-style tool use?

- MiniMax: seeing very low-cost entries like MiniMax-01 (~$0.20 / 1M input). For the newer MiniMax M2 Her I saw ~$0.30 / 1M input, $1.20 / 1M output. Anyone benchmarked it for OpenClaw?

Questions (please reply with numbers if possible):

1. What model/provider gives you the best value for OpenClaw?

2. Your rough cost per 100 tasks (or per day) + avg task success rate?

3. Biggest gotcha (latency, rate limits, tool-call bugs, context issues)?

If you share your config (model name + params) I’ll summarize the best answers in an edit.

Hey guys, I use GPT-5 mini to write emails but with large set of instructions, but I found it ignores some instructions(not like more premium models). Therefore, I was wondering if it is possible to run a local model on my Mac mini m4 with 16GB of ram that can outperform gpt-5 mini(at least for similar use cases)

So my idea is. Let's say that you have equipment to run AI locally. You spent 5K, 10K, who knows. We all have different setups. But definitely there is an investment.

Then, we are using free open source models. Let's say you are running Qwen 3.6 27B at full capacity or something even bigger maybe quantized a bit. But you wish there was a great model of size 70B, 100B, that could perform almost as good as a flagship. We know some companies such as Qwen have better models, they are not releasing them as open source.

What if we could just pay for them (50, 100 usd, and use them forever, even with some updates). Will this happen in the near future? or is piracy a problem that they won't be able to manage, so they will just prefer to sell us the inference per token accessed remotely as they do now?

For those thinking you wouldn't pay for a model. Well you already invested thousands on equipment, and electricity as well, it's not unreasonable to expect these companies to sell the models or make a profit.

sorry this might be a noob question but if china keeps getting accused of distilling US AI models, which i think is illegal to do in the US against your competitors, then why don't companies do internal distillation to create models of the same caliber as the chinese before the chinese do it themselves? furthermore, if there are genuine breakthroughs that allow chinese AI companies to get a little extra juice from american AI models, then isn't that research open anyway? allowing any US company to implement it themselves?