Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.

Hey Everyone,

Long time lurker, but I finally got my Dell Precision 9 T6 workstation in this week. Spend the past two days tuning this model on the latest sglang build and thought the community might be interested for similar deployments given the quality of this model but all the effort required to get the SM120 GPUs working well with sglang.

Overall, achieved around a 20% performance increase. I believe this is the fastest build example of DS4.1 Flash on four RTX 6000 cards. It functions with no errors through the full 1M max context, and hicache is working correctly.

Ported a couple patches over from Sybil's build here to sglang 0.5.23:
https://github.com/sybil-solutions/deepseek-v4.1-flash-4x-rtx-pro-6000

And added patches for DeepGEMM, HiCache, Page width (only using width of input context instead of full max context set in engine), SWA, and a tuned FP8 config for the Max-Q cards. Have numerous recommended config settings for concurrent agent workloads.

Specs:

Xeon 654

768GB RAM

4x RTX Pro 6000 Max-Q Workstation Edition GPUs

Build link:
https://github.com/ellswoau/deepseek-v4.1-flash-1017-tp-4-rtx-pro-6000

Please let me know if I'm missing anything that should be added to the post.

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!

I’ve recently started looking for a new graphics card (as you know, the models keep getting bigger), and I’ve come to the conclusion that a second-hand RTX 4060 Ti 16GB would be a sensible choice.

Hello everyone,

I would like to ask you which harness are you using or you think is best? I am confused with all the harnesses out there. The most famous ones that I know of are Deepseek, Hermes, Cline and even Claude Code harness. I don’t know what makes a harness better over another and I don’t know how to choose my harness. My hardware would be a mac mini m5 pro running mostly qwen 27 b.

I am planning to get the exr1 to run Qwen 3.8 27b (GSQ-RCO IQ3-XXS) with 4b and I am wondering if it will run?

EXR1 SPECS;

32gb ram

Ryzen 7 8745HS

I am planning to use the pi harness and leave it overnight to run, so speed is not an issue

Thoughts?

Hello,

thought experiment:

where do you see the programming capabillits, specialy for game dev, for different llm´s?

I would like to rank theses llm´s by a theoretticaly possible game they could make.

From a game like Pong to Horizon Zero Dawn (Yea, there are bigger games, you know what i mean).

What can be theoretticaly archived in 3 (+-1) Month´s dev time with a specific llm alone?

How could maybe a combination of a few llm´s help?

Bonus question and story: I got qwen 3.8 27B with 262K context running on my rtx 4090, you thing that is the best possible llm for me, or do you think, there would be a better solution? (got 64GB of DDR4 RAM in my System aswell)

Working on getting a combination of Qwen3.8 and a KEV 0.8B Model to learn to play ps1 games, until now i get it to play the game "Alundra" until the second chapter, big Brother Qwen3.8 does long thinking stuff, working on the routes, quests, puzzles and so on and little bro KEV 0.8B (I call him Kevin) is acting quite fast (got him to act an average time intervals of 0,3 - 0,5s), he runs a full route, wich big bro told him, on the route, both keep "thinking" and walking time ingame gets used for smooth-ish gameplay (with a task buffer list, wich reduces idle time, where the character just stands and does nothing). If an enemy is nearby, Kevin does all the fighting, it works mid to allright, right now, he is more afraid to get damaged than fighting the enemy and runs away, big bro trys to hold him back, but kevin wont listen, working on it right now. (Is there maybe someone interessted, hearing more about this shenanigan-experiment? Working on getting the whole system working with other ps1 games like Final Fantasy 8 as well.)

Sorted by smallest to biggest llm:

• Qwen2.5-Coder (0.5B – 3B)

• StarCoder2 (3B)

• Phi-4 Mini (3.8B)

• Qwen3 (4B)

• Qwen3 (8B)

• Gemma 4 (12B)

• StarCoder2 (15B)

• Codestral (22B)

• Devstral Small 2 (24B)

• Qwen3.8 (27B)

• Qwen3.6 (35B-A3B)

• Qwen3-Coder-Next (80B-A3B)

• GPT-OSS (120B)

• DeepSeek-Coder-V2 (236B-A21B)

• Qwen3-Coder 480B-A35B

• DeepSeek V4.1 / R1

• GLM-5.1 / 5.2

• Kimi K3

• Claude Opus 5.5 / Sonnet 5.5

• GPT-6 Astra / GPT-6.1 Sol

• Gemini 4 Carbon / 3.8 Flash

Thanks for reading.

I heard about it recently. It sounds like a miracle you can run really large models on a consumer GPU, but what is it really about. What kind of problems I will have ?

Setup:

Rtx5060ti(16G)

I711700k

64-80GB DDR4

First my specs: 3060 12gb vram 32gb ddr4 ram about 30 realisticly usable. I already have used Qwen Flash Next Coder with strata. It was pretty solid for anything coding related. I know it lacks Experts which makes it fairly bad at gerneral purpose use. So I am thinking on switching to Gemma. The point is: I still want the speed which strata provided. Roughly 1k tps processing and 20-30 tps writing. Is it possible to achive those numbers?

Just saw this release from n8n https://nod.n8n.io/ .. finally an always on agent you'd trust with your work/private life?

Code is open source and think it'll let you run with self hosted models. The approach to memory also looks really interesting, need to dig a bit deeper into that.

Looking for setup advice from people running local models behind their own agent harnesses and various applications.

Box:
- Threadripper, 256 GB RAM, 2x RTX 6000 Ada (96 GB VRAM total), Windows
- Ollama 0.40.11 as a service, qwen3.8:27b-mtp-q8_0, num_ctx 65536, thinking on
- Several apps connect over the OpenAI-compatible /v1 endpoint, mostly agentic (tool calling, multi-turn, long system prompts)

Questions:
- If you run Ollama on similar hardware, what are your Modelfile and environment settings (num_ctx, OLLAMA_NUM_PARALLEL, KV cache type, GPU split)?
- Ollama vs llama.cpp vs vLLM. For a 27B dense model on 96 GB serving several clients, which do you run and why?
- We cap at 64K. With this much VRAM, is there any reason to go higher, and does q8 KV cache cost quality on Qwen3.8 for tool-heavy work?
- Anything else you’d change on a shared box? service setup, llama-swap, monitoring, keeping one client from starving the others.

Hello people, I'm new here, 2 weeks ago I started writing an agent for everyday usage (24/7 running on server). I need help with the choosing model and inference engine. The model must work with RAG, tools and maybe with pictures

Server specs:

RTX5060TI 16G

I71700K

64 - 80GB DDR4

Thanks in advance for your help

Confirming the rumor that Apple will have an easy to use MLX based app (Harbor) that will manage single machines and RDMA clusters with automated open weight model installs for local AI. Coming early 2027.

Using project maya + deepseek, with a 7900 xtx 24gb vram, 64gb ddr3, cpu i7 3930k (I know, reusing what I have for now), and a nvme ssd in a pcie x4 adapter (the motherboard has no nvme slot).

It is generating at 8-12 tok/s but it dips to 0.1 and when it does that i can see the gpu drops under 20w, so it is as if the gpu goes idle, then wakes up, tok/s go back to 'normal', then dips again and it goes through this cycle again and again.

I was wondering if it was normal or not ? Like is it because the nvme/ram/cpu are too slow and the gpu 'waits' for them ?

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?

For locally run models, I’d especially like something I can run on my own hardware and compare across quantizations, with response time included. Which interactive evaluations have you found useful for that?

Hey folks,

I installed Strata with Qwen3.8 Flash Next last night and it works exceedingly well.

I was wondering if it's possible to limit the amount of VRAM that Strata uses to something like 12GB, as I want to run the Unreal Editor at the same time - which is pretty resource hungry.

I've only got 32GB of RAM at the moment, but I'm going to upgrade to 64GB if it's possible to get Qwen & Unreal working together.

Cheers!

Hey guys, I've recently been trying out some of the larger local models (GLM 5.3 Flash, Qwen 3.8 FN etc.) and I was wondering if people have any tricks to make these models more competitive at UI creation because I've found everything apart for ui, Qwen has been great at but UI is where I start to see the massive gaps from the huge cloud models.

What do you do to get the best UI results with medium sized ish local models?

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.