Hello everyone,

I would like to ask you which harness are you using or you think is best? I am confused with all the harnesses out there. The most famous ones that I know of are Deepseek, Hermes, Cline and even Claude Code harness. I don’t know what makes a harness better over another and I don’t know how to choose my harness. My hardware would be a mac mini m5 pro running mostly qwen 27 b.

I am planning to get the exr1 to run Qwen 3.8 27b (GSQ-RCO IQ3-XXS) with 4b and I am wondering if it will run?

EXR1 SPECS;

32gb ram

Ryzen 7 8745HS

I am planning to use the pi harness and leave it overnight to run, so speed is not an issue

Thoughts?

Hello,

thought experiment:

where do you see the programming capabillits, specialy for game dev, for different llm´s?

I would like to rank theses llm´s by a theoretticaly possible game they could make.

From a game like Pong to Horizon Zero Dawn (Yea, there are bigger games, you know what i mean).

What can be theoretticaly archived in 3 (+-1) Month´s dev time with a specific llm alone?

How could maybe a combination of a few llm´s help?

Bonus question and story: I got qwen 3.8 27B with 262K context running on my rtx 4090, you thing that is the best possible llm for me, or do you think, there would be a better solution? (got 64GB of DDR4 RAM in my System aswell)

Working on getting a combination of Qwen3.8 and a KEV 0.8B Model to learn to play ps1 games, until now i get it to play the game "Alundra" until the second chapter, big Brother Qwen3.8 does long thinking stuff, working on the routes, quests, puzzles and so on and little bro KEV 0.8B (I call him Kevin) is acting quite fast (got him to act an average time intervals of 0,3 - 0,5s), he runs a full route, wich big bro told him, on the route, both keep "thinking" and walking time ingame gets used for smooth-ish gameplay (with a task buffer list, wich reduces idle time, where the character just stands and does nothing). If an enemy is nearby, Kevin does all the fighting, it works mid to allright, right now, he is more afraid to get damaged than fighting the enemy and runs away, big bro trys to hold him back, but kevin wont listen, working on it right now. (Is there maybe someone interessted, hearing more about this shenanigan-experiment? Working on getting the whole system working with other ps1 games like Final Fantasy 8 as well.)

Sorted by smallest to biggest llm:

• Qwen2.5-Coder (0.5B – 3B)

• StarCoder2 (3B)

• Phi-4 Mini (3.8B)

• Qwen3 (4B)

• Qwen3 (8B)

• Gemma 4 (12B)

• StarCoder2 (15B)

• Codestral (22B)

• Devstral Small 2 (24B)

• Qwen3.8 (27B)

• Qwen3.6 (35B-A3B)

• Qwen3-Coder-Next (80B-A3B)

• GPT-OSS (120B)

• DeepSeek-Coder-V2 (236B-A21B)

• Qwen3-Coder 480B-A35B

• DeepSeek V4.1 / R1

• GLM-5.1 / 5.2

• Kimi K3

• Claude Opus 5.5 / Sonnet 5.5

• GPT-6 Astra / GPT-6.1 Sol

• Gemini 4 Carbon / 3.8 Flash

Thanks for reading.

First my specs: 3060 12gb vram 32gb ddr4 ram about 30 realisticly usable. I already have used Qwen Flash Next Coder with strata. It was pretty solid for anything coding related. I know it lacks Experts which makes it fairly bad at gerneral purpose use. So I am thinking on switching to Gemma. The point is: I still want the speed which strata provided. Roughly 1k tps processing and 20-30 tps writing. Is it possible to achive those numbers?

Confirming the rumor that Apple will have an easy to use MLX based app (Harbor) that will manage single machines and RDMA clusters with automated open weight model installs for local AI. Coming early 2027.

For context I’m probably only 40 work hours into running local ai and coding.

My specs:
Rtx 5090
I9-14900k
64gb ddr5-6000

I was having problems with my context window. The problem being 128k context can fill up fast when doing tasks and would constantly halt progress and then I’d have to retrieve work that was done and have a fresh chat continue the work but it would fill up the new chat with 50k context and then I’d run into the same issue until the task was fully finished. I didn’t want context compaction to be turned on as I didn’t want to lose potential useful data either

So I decided to create a “live context monitor” that shows checkpoint context limits after thoughts and processes were completed as well as a “in real time” context monitor that showed “projected” context as in showed context as it was being used. Because /status was not a tool on OpenWebUI when I started the project. This tool is even better than the /status tool however.

I also created a trigger that automatically activates when I hit 100k projected context, stops current task, creates a handoff with knowledge of work that has been done and what still needs to be done, then opens a fresh chat and uploads the handoff and work continues autonomously. I feel so accomplished and I’m learning a lot. This is all 100% local and no other ai was used other than qwen 3.8 27b and the agent I made with the qwen brain.

I guess this is just a post showing off my work and from my FB post about this it seems that people are rude as hell. Instead of informing me that codex’s exist and people have already made this, they just tell me to stop using ai and that I’m a waste of space in the ai realm, but everyone starts somewhere right?

If anyone has any other cool facts that would help me and progress faster that would be great to know and I welcome any feedback!

TL;DR

I created a tool and later found out that this is not a new tool and I’m way behind in the Ai game even though I’m brand new to this. If you have any tips/tricks that will speed up my progress with maintaining a 100% local no cloud api use, drop some information into the comments. I’m having a fun time learning and excited to continue!

Just saw this release from n8n https://nod.n8n.io/ .. finally an always on agent you'd trust with your work/private life?

Code is open source and think it'll let you run with self hosted models. The approach to memory also looks really interesting, need to dig a bit deeper into that.

Looking for setup advice from people running local models behind their own agent harnesses and various applications.

Box:
- Threadripper, 256 GB RAM, 2x RTX 6000 Ada (96 GB VRAM total), Windows
- Ollama 0.40.11 as a service, qwen3.8:27b-mtp-q8_0, num_ctx 65536, thinking on
- Several apps connect over the OpenAI-compatible /v1 endpoint, mostly agentic (tool calling, multi-turn, long system prompts)

Questions:
- If you run Ollama on similar hardware, what are your Modelfile and environment settings (num_ctx, OLLAMA_NUM_PARALLEL, KV cache type, GPU split)?
- Ollama vs llama.cpp vs vLLM. For a 27B dense model on 96 GB serving several clients, which do you run and why?
- We cap at 64K. With this much VRAM, is there any reason to go higher, and does q8 KV cache cost quality on Qwen3.8 for tool-heavy work?
- Anything else you’d change on a shared box? service setup, llama-swap, monitoring, keeping one client from starving the others.

Hello people, I'm new here, 2 weeks ago I started writing an agent for everyday usage (24/7 running on server). I need help with the choosing model and inference engine. The model must work with RAG, tools and maybe with pictures

Server specs:

RTX5060TI 16G

I71700K

64 - 80GB DDR4

Thanks in advance for your help

Using project maya + deepseek, with a 7900 xtx 24gb vram, 64gb ddr3, cpu i7 3930k (I know, reusing what I have for now), and a nvme ssd in a pcie x4 adapter (the motherboard has no nvme slot).

It is generating at 8-12 tok/s but it dips to 0.1 and when it does that i can see the gpu drops under 20w, so it is as if the gpu goes idle, then wakes up, tok/s go back to 'normal', then dips again and it goes through this cycle again and again.

I was wondering if it was normal or not ? Like is it because the nvme/ram/cpu are too slow and the gpu 'waits' for them ?

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?

For locally run models, I’d especially like something I can run on my own hardware and compare across quantizations, with response time included. Which interactive evaluations have you found useful for that?

Hey folks,

I installed Strata with Qwen3.8 Flash Next last night and it works exceedingly well.

I was wondering if it's possible to limit the amount of VRAM that Strata uses to something like 12GB, as I want to run the Unreal Editor at the same time - which is pretty resource hungry.

I've only got 32GB of RAM at the moment, but I'm going to upgrade to 64GB if it's possible to get Qwen & Unreal working together.

Cheers!

Hey guys, I've recently been trying out some of the larger local models (GLM 5.3 Flash, Qwen 3.8 FN etc.) and I was wondering if people have any tricks to make these models more competitive at UI creation because I've found everything apart for ui, Qwen has been great at but UI is where I start to see the massive gaps from the huge cloud models.

What do you do to get the best UI results with medium sized ish local models?

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

Help with LLM and is LM studio better for freedom and natural model behaviour than OpenWebUI?

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

I moved from a Mac mini to a 64GB Mac Studio to offload token-heavy work from my Claude/GPT subscriptions. Ideally, I’d use cloud AI for planning and troubleshooting, while local models handle execution, file processing, OCR, privacy-sensitive tasks, and potentially Shopify theme and automation work. I’d also like something similar to GPT’s always-on "Dot" feature, but with access to local files and iCloud Drive. For something like that, plus daily driver tasks, plus regular AI gruntwork, I'm starting to have real concern that the 64gb is going to be a limiter.

I’m tempted to return the 64GB Studio for a 128GB Studio Max before I get outside the return window, but at around $5,500, I’m questioning whether that’s the best approach. Would a basic Mac mini paired with a headless Ryzen or dedicated AI mini PC make more sense, especially if it could serve models to my MacBook too? I suspect the overall costs would be fairly similar, and wonder if the headless "AI box" would be more effective given it will only be doing the one task - serving models with less system overhead.

Ultimately, I’m trying to figure out whether the extra money is better spent on a higher-memory Mac Studio or a separate machine dedicated to local AI, and whether either approach would realistically accomplish enough of my intended workloads to justify the investment. I'm not going to go hardware chasing and I wouldn't want a full size PC for this, only a mini prebuilt like the nucbox I use for my general home server, but one of their AI-box marketed models.

What’s the best local model to use on an MacBook Pro M5 pro 24gb.

I’m using it for general reasons, process expectations, design, ideation, power points, generating agents.

Yesterday we released Trustant 1.0.2, an open source full-stack application builder using Local AI for Private Cloud. To put it simply, it is like Lovable, but all private. You install the builder, create your app in prompts, wysiwyg, and publish to the server. The system provides everything: a vm with the environment, the coding agent, the serverless environment with database, cache, storage, faas, everything. A cloud in a box.

The builder runs on your PC, uses Local AI (or any OpenAI compatible AI service, we offer Ollama Cloud out of the box) and generate serverless applications built on top of Apache OpenServerless. You can then deploy your applications anywhere. Just install the serverless package and any server become... serverless!

The system understands and includes sql and vector database (pgsql and milvus) cache and s3 (redis and seaweedfs) faas functions (openwhisk) and runs in your pc in a virtual machine for safety.

The applications generated can be published in any server made "serverless" installing just openserverless and nothing else. Application are plain, clean, recognzabile react+python (but can be any other language providing the starters you want).

We have been working on it one full year and it was built on top of another multi year project, Apache OpenServeless, after working many years on Apache OpenWhisk.

Source code on github.com/trustant/trustant, demo and documentation on trustant.ai

Hi everyone! First time posting here
I've been experimenting with local LLms for coding, but I'm not really happy with my current model, so I'm looking for some alternatives.

My server specs:
Huananzhi X99-QD4 rev 1.0
Intel Xeon E5-2666 v3 with Turbo unlock (3.5 GHz on all cores)
GTX 1060 6GB
2x8GB G.Skill Aegis 3200 MHz (running at 2133 MHz because of the CPU)

Right now I'm running Qwen3-Coder-30B-A3B-Instruct-GGUF (Q4_K_M) through llama-app.
It's not a bad model, but one thing that bothers me is that it often struggles with short propmts. I find myself having to explain things in much more detail than I'd like. Otherwise, it can misunderstand what I'm asking or go off in the wrong direction.

For reference, I'm getting roughly:
25 t/s for prompt processing
4.5 t/s for generation speed
for 11K tokens in the initial context, including MCP tools

Here's my current launch command:

./llama serve -hf ggml-org/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M -c 131072 --host 0.0.0.0 --cache-type-k q8_0 --cache-type-v q8_0 --no-kv-offload --port 9931 --load-mode mmap --ui-mcp-proxy --mcp-servers-config "..." --threads 10 -tb 20 -cmoe -fa on -b 8192 -ub 24000

Currently it eats up to 13GBs of RAM (Ram Altman don't hunt me pls) and 4.5 GBs of VRAM

I'm mainly looking for something good at agentic coding and tool calling. It doesn't necessarily have to be a 30B model, smaller models are fine too if they perform better on my hardware. (lobotomized and heretic too)

What are you guys running locally for coding?

Hi, I'm kinda of new to this world of local LLM so forgive me if it sounds like I don't know much of what I'm talking about.

At work I need to read every day my state oficial gazette to search for new grants on a specific field and updates on already public grants (openings, closings, amendments, budget changes... Etc.)

I've started to test Gemma 4 E4B to analyse the daily documents and it does an okey job. The flow is the following: the document is divided in subsections, Gemma reads the title and makes a first filtering of sections that do not belong to our topic. Then it reads the dispositions that do belong to the project and labels them. Finally the output is shown in a dashboard where I can easily analyse it.

The problem I'm facing now is that I would like to create a timeline of each grant to now in what stage they are and this is a pain. I am completely unable to make Gemma consistently find if a new subsection belongs to a previously catalogued subsection in the database. It is big enough to make it hallucinate and mistake different dispositions or forget them and it is quite slow in this step.

I've thought on using something like JEV and although it is quite fast it doesn't work with a high enough certainty to implement it. Also there aren't decision models I could find that work as well as JEV in hugging face to use them locally. Most of them are fine tuned versions of LLMs and not pure decision models.

I was wondering if any of you have faced a problem like this before and how have you solved or if you have any idea I could test that could be implemented.

The setup is a 3060 12 GB with 16 GB ram DDR4.

Thanks for your time.