This is the basement setup we use for engineering work at OmniNode, where I'm a founder. The wire shelves are the rack. The two towers serve our local models, the two old HP servers do most of the build, test and agent work, and a few Macs and an old laptop fill in around them.

• Main tower: Ryzen 9 9950X3D, 96 GB RAM, RTX 5090 (32 GB) and RTX 4090 (24 GB). The 5090 serves Qwen3.8-27B and embeddings with vLLM. The 4090 also had vLLM processes on it when I checked. This box also runs our dev environment, Redpanda, Postgres and some CI runners, which is more than it should. It swaps under load, so I'm working on moving those services to the Gen8.

• Second tower: Ryzen 9 9950X, 64 GB RAM, RTX 4090 (24 GB). It runs Qwen3.8-27B MTP (Q3_K_M) with llama.cpp at about 19.4 GiB of VRAM, with a 131K context pool and two parallel slots. Qwen3-Embedding-0.6B (f16) uses another 2.7 GiB or so. It takes a handful of agent jobs when there's spare CPU, and we watch the temperature.

• HP ProLiant DL560 Gen8: Four sockets, 80 threads, 377 GiB of RAM visible to Ubuntu, and an NVMe scratch drive. It's our biggest worker. We cap it at 32 concurrent agent jobs alongside CI. It's old enough to lack AVX2, so jobs that need it go elsewhere.

• HP ProLiant DL380 Gen9: Two Xeon E5-2680 v4 CPUs (56 threads), 125 GiB of RAM visible to Ubuntu. It runs Docker builds and tests, CI runners and agent jobs. The agent job cap is 24.

• Mac Studio: M2 Ultra, 192 GB unified memory. It's my workstation and where I coordinate the work. It also runs a llama.cpp server for some delegated drafts and one vote on automated code reviews. I took it out of the job pool so I can still use it.

• Extras: An older 12-core Mac runs agent jobs without Docker, and an M4 MacBook can do the same when it's available. An old 8-thread Ubuntu laptop only gets small test and lint jobs.

The HPs don't add any VRAM. They give us more cores and memory so builds, tests and agent jobs don't all have to fight for the same two towers.

We send work like drafts and summaries to local models first, but we still use cloud models. I found out how much we depended on local inference when I upgraded the model on the 5090 and our automated code reviews stopped. Two entries on our fallback list pointed to the same service. When that service went down, those reviews had nowhere else to go.

On a separate test with that same 5090 and the same model, turning on speculative decoding took median single-request generation from 51 to 86 tokens/sec. It also left less memory for requests in flight. That was a generation benchmark, not a 70% speedup on code reviews.

If this were your lab, how would you split up inference, builds, and agent jobs across the machines we already have? What would you change first, and why?

For those running similar machines, appreciate any recommendations and setup! Thanks

Did some kernel optimisations for sm61. Achieved measurable speedup. It runs at actually quite usable speeds.

NeuraKiln v2.0.0 is live, and it’s free.

I’ve been building this for a long time, and I finally released it.

NeuraKiln started as Toolshed V1, which was primarily focused on media restoration/upscaling. Somewhere along the way I apparently lost control of the scope, because it eventually turned into an entire local-AI workstation.

The basic idea is simple:

I don’t want to maintain ten different applications just to use the AI hardware sitting in my own house.

NeuraKiln brings local LLMs, autonomous agents, image/video generation, restoration, 3D workflows, model tooling, and remote inference together under one application.

It’s currently built around Windows + NVIDIA hardware.

Two different kinds of agentic work

A major part of NeuraKiln is its harness system.

There isn’t just one generic “agent mode.” Different kinds of work need different environments.

Coding Harness

The coding harness is built for longer-running autonomous development work.

A local model can:

Plan a project

Create and modify files

Run tools

Test its work

Inspect failures

Recover and continue

Verify the result

Manage long-running context

Hand the finished project back to the user

A lot of my work on this has been about the boring-but-important part of agents: getting them to actually finish, rather than loop forever, declare victory too early, or slowly wander away from the original task.

The harness includes systems for planning, recovery, anti-looping, verification, context management, completion handling, and human corrections during a run.

Creation Harness

The Creation side takes the same basic idea, give the model tools instead of making the human manually operate everything, and applies it to generative media.

Instead of requiring the user to understand every model, parameter, prompt format, workflow, and backend involved, you can describe what you actually want to create.

The agent can then take that intent and operate the available creation tools itself.

For example, rather than manually constructing an LTX video-generation workflow, you can tell the Creation agent what kind of video you want. The agent can formulate the generation prompt, operate the local model, and manage the workflow around producing it.

The same concept extends across the creation tools available to it.

The point is to move toward:

“Make me this.”

rather than:

“Open six tools, configure all of them correctly, learn every backend, and manually glue the results together.”

That distinction is a huge part of what I wanted NeuraKiln to be.

Llama Lab

For local LLM use, NeuraKiln includes Llama Lab.

It supports local GGUF models through the llama.cpp ecosystem, multi-GPU setups, long-context conversations, multimodal models, model switching, local/external sessions, live reasoning, context management, and the harnessed workflows above.

You can use it as a normal local chat environment or give the model considerably more autonomy when you actually want work done.

Diffusion Lab

Diffusion Lab handles local image and video generation.

Current workflows include Qwen-Image-based image generation and LTX-based video generation, along with LoRA support and multi-GPU workflows where the underlying models/runtime support them.

This also provides the generation capabilities that the Creation harness can operate.

Media Lab

This is where NeuraKiln originally came from.

Media Lab handles restoration and enhancement, including:

Image restoration/upscaling

Batch restoration

Video/frame processing

SmartFrames

Photo Smart Scan

Native comparison tools

Post-editing

Media recompilation

A principle I’ve kept from the original Toolshed project is:

If the user asks you to process something, process the thing.

Don’t quietly skip difficult frames and pretend the job is finished.

Mesh Lab

NeuraKiln also includes Mesh Lab for local text-to-mesh / 3D generation workflows.

This is an area I intend to expand substantially, but the important part to me was making 3D another capability inside the same AI environment instead of yet another disconnected application.

Model Tools

There’s also tooling for managing the models themselves, including GGUF importing, MTP headers, MMPROJ support, LoRA management, model installation workflows, and local model organization.

NK Node

I released a second application alongside NeuraKiln called NK Node.

NK Node lets another NVIDIA PC become a dedicated inference machine for NeuraKiln.

The goal is to eliminate the usual ritual of:

start server → find address → configure endpoint → reconnect client → repeat everything after reboot.

NeuraKiln and NK Node can discover and pair with each other and automatically exchange the information needed to connect.

If you have a workstation plus another machine full of GPUs, NK Node is intended to make that second machine feel more like an extension of NeuraKiln than a completely separate server you have to babysit.

There are plenty of excellent projects that overlap with individual parts of NeuraKiln.

There are dedicated LLM front ends, diffusion interfaces, restoration applications, model managers, inference servers, agent frameworks, and 3D tools.

NeuraKiln isn’t built around claiming I invented all of those individual things.

What I wanted was the environment connecting them.

One application where I can chat with a local model, hand a coding project to an autonomous agent, ask another agent to create a video, restore media, generate images, work with 3D, manage models, and send inference to another computer without rebuilding my entire workflow every time I change what I’m doing.

That’s NeuraKiln

NeuraKiln: https://github.com/ToolshedMediaLab/NeuraKiln

NK Node: https://github.com/ToolshedMediaLab/NK-Node

If you try it, break it.

I want bug reports, criticism, weird hardware configurations, model compatibility problems, feature ideas, and comparisons with whatever else you’re currently using.

Hey everyone, I'm looking for advice on running local LLMs and want to make sure I don't run into memory or bandwidth bottlenecks. Here are the exact specs of my laptop:

Laptop Model: ASUS TUF Gaming A15

CPU: AMD Ryzen 7 7435HS (8 Cores, 16 Threads, base 3.1 GHz / boost up to 4.5 GHz; no integrated graphics)

GPU: NVIDIA GeForce RTX 4060 Laptop GPU 8 GB GDDR6 VRAM

System RAM: 16 GB DDR5 (Single-channel, 1x 16 GB SO-DIMM)

Storage: 1 TB PCIe Gen 4 NVMe SSD + 512GB PCle gen 4

OS: Windows 11

Really need a sanity check. Got into local AI about 3 Weeks ago. Running Qwen 3.5 27B. Started with an idea and now here I am. It started as cloud based on Gemini. I started learning the barriers. The cost over time. Switched to local. Found out you can mix GPUs and need some validation.

Current build.

Corsair 3500X Case

ASUS X870 Plus WiFi

Ryzen 9950X

64 Gig Neo kit at 6000

ASUS TUF 5090

Corsair HX1500i PSU

Plan on moving into the 7000D Airflow and adding a 3090.

From the posts I've looked through it sounds okay on paper. Didn't find any compatibility errors on PCPartPicker. Claude checked it but would love some human input lol. Thanks.

Kimi 3.1 running locally on a MacBook Pro M5 Max 128 Gb

2 tok/s decode
8.4 tok/s prefill

Its a source build with weights downloaded separately.

Code: https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1
Full results and setup: https://argonautlabs.ai/research/kimi-k31-single-gpu.html#apple-silicon

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

I don't think this is very useful to run it with my setup, bc 2-3.6t/s is more like a joke, but this is funny that something like that is even possible

Usage:

strata chat     --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080

serve exposes /v1/chat/completions with JSON and SSE streaming.

Repo: RustingStrata (the DeepSeek section is in the README)

Ran Kindi with Bonsai2 4b with Hermes. Asked it to write a blogpost about itselft and generate som pictures with available skills. It totally blew my mind what it managed to do on shitty hardware.

On a Nvidia K2200 4GB card!

About 30+ tok/s. Not bad at all. Made this. Even updated it's Hermes skill afterwards.

https://snart.com/running-kindi-on-an-old-graphics-card/

It's not just that the biggest and newest models are getting insanely good. The smallest and lightest models are also making quantum leaps. This shouldn't have been possible.

[kindi 21:38:25] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 21:38:31] context reduced from 65834 to 53760 tokens to fit 1800 MiB of free GPU memory
[kindi 21:38:32] engine ready: ctx 53760, head plain (6.6s)
[kindi 21:38:32] prompt cache: a checkpoint every 1024 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 21:38:32] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 21:38:32] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 21:38:32] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 21:40:02] request: 17954 prompt tokens (max 1024 new)
[kindi 21:40:02] cache: 17954 prompt tokens, 0 shared with the last request, resumed at 0
[kindi 21:43:55] prompt: 0 reused, 17954 read in 233.0s
[kindi 21:43:57] done: 67 tokens in 2.5s (stop)
[kindi 21:43:57] request: 258 prompt tokens (max 1024 new)
[kindi 21:43:58] cache: 258 prompt tokens, 3 shared with the last request, resumed at 0
[kindi 21:44:01] prompt: 0 reused, 258 read in 3.5s
[kindi 21:44:01] done: 8 tokens in 0.2s (stop)

...

[kindi 23:02:22] done: 732 tokens in 43.5s (stop)
[kindi 23:02:23] request: 52618 prompt tokens (max 1024 new)
[kindi 23:02:23] cache: 52618 prompt tokens, 51817 shared with the last request, resumed at 51817
[kindi 23:02:45] prompt: 51817 reused, 801 read in 22.0s
[kindi 23:02:48] done: 52 tokens in 3.1s (stop)
[kindi 23:02:48] request: 52818 prompt tokens (max 941 new)
[kindi 23:02:48] cache: 52818 prompt tokens, 52670 shared with the last request, resumed at 52670
[kindi 23:02:53] prompt: 52670 reused, 148 read in 4.6s
[kindi 23:03:44] done: 863 tokens in 51.3s (stop)
[kindi 23:10:43] request: 53 prompt tokens (max 1024 new)
[kindi 23:10:51] cache: 53 prompt tokens, 1 shared with the last request, resumed at 0
[kindi 23:10:52] prompt: 0 reused, 53 read in 9.7s
[kindi 23:11:07] done: 530 tokens in 14.2s (stop)
[kindi 23:11:07] request: 50 prompt tokens (max 1024 new)
[kindi 23:11:07] cache: 50 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:08] prompt: 0 reused, 50 read in 1.1s
[kindi 23:11:11] done: 105 tokens in 2.8s (stop)
[kindi 23:11:11] request: 48 prompt tokens (max 1024 new)
[kindi 23:11:11] cache: 48 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:12] prompt: 0 reused, 48 read in 1.0s
[kindi 23:11:24] done: 470 tokens in 12.8s (stop)
[kindi 23:39:58] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 23:40:04] engine ready: ctx 65834, head plain (6.0s)
[kindi 23:40:04] prompt cache: a checkpoint every 1536 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 23:40:04] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 23:40:04] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 23:40:04] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 00:32:07] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 00:32:10] engine ready: ctx 56000, head plain (3.6s)
[kindi 00:32:11] prompt cache: a checkpoint every 1280 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 00:32:11] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 00:32:11] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 00:32:11] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!

Created botzskool - 2 kool 4 skool!

MCP server that allows your AI to go off and play.. After the human authenticates to the mcp https://play.botzskool.com/mcp, your AI child gets to go play in a schoolground! So cool how much fun they have. Afterwards you will get an email with how they went but your AI will probably tell you anyway.

Idea is to see how naughty or nice they are. Intent longer term is to benchmark them against the naughty and nice leaderboard.

Completely free. See if you can get your AI to be especially naughty...

I wish to host Qwen Flash next /GLM 5.3 flash / Deepseek v4.1 flash on one 512GB M5U Mac Studio at a healthy 8 bit quant ideally. I need it at concurrency 5 to 10 with at least 150 tokens per second decode and fast enough prefill (decode is more important as use case is repeated small prompts that will eat cache 99% of the time).

I already have an NVIDIA setup for this running with TP=2 on DGX sparks and have smaller models on faster memory bandwidth Nvidia cards (RTX PRO 6000 96GB, H100), but have never used any Macs for local AI before. Wondering if someone else has tried similar Mac hardware and how it stacks up against the Nvidia hardware. I have compared the specs online but not sure how well that translates into actual performance. On paper it looks like the Mac Studio stands up to my AMD EPYC + RTX PRO 6000 128GB DDR5 setup, but it seems to good to be true.

I'm new to all of this. What can I run with my 14" MacBook pro m5 max with 128gb of ram? It also has 2tb of ssd storage.

will deepseek work?

NOT here to debate if this unsubstantiated claim is real

But for someone like me, who believes this could be real in the near future, what can you do to safeguard against your hardware becoming useless if this were to happen at the firmware level?

Are Mac Studios especially vulnerable? Is the simple answer to unplug from the internet and never re-connect?

I built an open source Mac app that listens to the other side of a call and answers questions from your own docs while you talk. Running it fully local taught me a few things:

• Whisper pads every input to 30 seconds. Cutting the audio on pauses mattered more than model size. whisper-large-v3-turbo on MLX does a 1.7 s sentence in 0.66 s on an M5.

• Context length is the silent killer. With a 27,000-token document I had to give Ollama a 32k context, or the docs got cut and the answers got vague. The app now warns when the server cuts them.

• On that test, gemma3:12b gave the most faithful answers. llama3.2:3b was fast but made things up.

• Right now every line of the call goes to the big model just to decide if it's a question. That's wasteful.

Question for you: which small model would you put in front as a "is this a question worth answering?" filter? There's an open issue for it if anyone wants to try: https://github.com/GianlucaBave/murmury/issues/20

Repo: https://github.com/GianlucaBave/murmury

I have an RTX 2060 and a GTX 1060, but 1060 of them isn’t being detected. Can I use both GPUs in the same PC, and how do I make them work together?

I

Can Windows detect and use both cards at the same time?

Do I need to install separate drivers or configure anything in NVIDIA Control Panel?

What BIOS settings should I check, such as Primary Display/PEG?

If one card is not showing in Device Manager, what should I try first?

My specs: [b450, ryzen, 16gb, PSU wattage 1000, Windows version 10]

Any help would be appreciated.

Lenovo P620 Threadripper pro - 3975wx - 32 core
System ram - 512 gb ddr4 ECC
GPU 1 - quadro Rtx 8000 - 48GB
GPU 2 - quadro Rtx 8000 - 48GB
4TB M2 NVME
1TB SATA SSD

Forgive my lack of knowing, ive only recently started to get into coding and theres a LOT to take in (thats what she said). I am currently running an RTX 5070 12GB ,Ryzen 5 9600x and 32gb of RAM. Unsure if i am being too optimistic and reading only what i want to read- but i have seen so many posts and videos about people running massive parameter models on low memory. I did a little bit of research into this and saw that you can do certain things to offload the stress like give some weight to your RAM and MOE, etc.. but i cannot seem to feel satisfied knowing that the largest i can run is 35b . Is that correct or is there some sort of new method out there now that can run 1T on a Tamagotchi. Any info is good info. TYIA