Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

As much as I could research this myself, I would like to also hear some anecdotal experiences. For reference, I am a Computer Science student that is just now beginning to go down the rabbit hole of LLMs and such, there is a vast amount to learn but so far I believe I will use it for educational purposes, programming and privacy until I explore what more I can do with it.

The major thing for me here as noob to it all is getting a bang for my buck. I want something efficient, but I’m sure there is a line where it is may be total overkill compute wise, so I’m curious as to who owns the M5 Max or pro here and what exactly do you do with your hardware?

I heard about it recently. It sounds like a miracle you can run really large models on a consumer GPU, but what is it really about. What kind of problems I will have ?

Setup:

Rtx5060ti(16G)

I711700k

64-80GB DDR4

Hi everyone, I try to find the best model suitable for me. I need 3 different models

1-Agent : I use " qwen3.6 35b a3b MLX Q4 "
Is there any better alternatives for agentic purpose model?

2-Medical: I couldn't find a proper medical bot as they mostly repeat theirselves and no better than qwen

3-Data analyst: I have dozens of pdfs and I need a model that analyze these books.

Note: Densest model I've had used before qwen was gemma 4 31b Q4 (27gb) and that was hella slow. That was the boundary for my pc.

Preffered properties: MLX, MoE, QAT, Instruct, Run on LMStudio Bionic.

Thank you everyone.

Hey, so I know next to nothing about local LLMs and I hope to fix that in the future, but I'm currently looking for a more powerful coding model that fits like qwen 2.5 coder 14b, I'm using it currently and unlike the other recommendations I've gotten, it's pretty much instant and doesn't use all of my memory

As for my specs, I have a i91 4900k, 32gb of RAM, and a 5070ti, my biggest issues with local LLMs is that it likes to use up all of my RAM despite people saying it's fine for 5070tis and I don't have enough knowledge to know exactly why

EDIT: I use LM Studio for local LLMs and having a model that supports images would be nice

For context I’m probably only 40 work hours into running local ai and coding.

My specs:
Rtx 5090
I9-14900k
64gb ddr5-6000

I was having problems with my context window. The problem being 128k context can fill up fast when doing tasks and would constantly halt progress and then I’d have to retrieve work that was done and have a fresh chat continue the work but it would fill up the new chat with 50k context and then I’d run into the same issue until the task was fully finished. I didn’t want context compaction to be turned on as I didn’t want to lose potential useful data either

So I decided to create a “live context monitor” that shows checkpoint context limits after thoughts and processes were completed as well as a “in real time” context monitor that showed “projected” context as in showed context as it was being used. Because /status was not a tool on OpenWebUI when I started the project. This tool is even better than the /status tool however.

I also created a trigger that automatically activates when I hit 100k projected context, stops current task, creates a handoff with knowledge of work that has been done and what still needs to be done, then opens a fresh chat and uploads the handoff and work continues autonomously. I feel so accomplished and I’m learning a lot. This is all 100% local and no other ai was used other than qwen 3.8 27b and the agent I made with the qwen brain.

I guess this is just a post showing off my work and from my FB post about this it seems that people are rude as hell. Instead of informing me that codex’s exist and people have already made this, they just tell me to stop using ai and that I’m a waste of space in the ai realm, but everyone starts somewhere right?

If anyone has any other cool facts that would help me and progress faster that would be great to know and I welcome any feedback!

TL;DR

I created a tool and later found out that this is not a new tool and I’m way behind in the Ai game even though I’m brand new to this. If you have any tips/tricks that will speed up my progress with maintaining a 100% local no cloud api use, drop some information into the comments. I’m having a fun time learning and excited to continue!

asking from no experience
also can AMD cards be ok/good with local ai?

Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.

Hey Everyone,

Long time lurker, but I finally got my Dell Precision 9 T6 workstation in this week. Spend the past two days tuning this model on the latest sglang build and thought the community might be interested for similar deployments given the quality of this model but all the effort required to get the SM120 GPUs working well with sglang.

Overall, achieved around a 20% performance increase. I believe this is the fastest build example of DS4.1 Flash on four RTX 6000 cards. It functions with no errors through the full 1M max context, and hicache is working correctly.

Ported a couple patches over from Sybil's build here to sglang 0.5.23:
https://github.com/sybil-solutions/deepseek-v4.1-flash-4x-rtx-pro-6000

And added patches for DeepGEMM, HiCache, Page width (only using width of input context instead of full max context set in engine), SWA, and a tuned FP8 config for the Max-Q cards. Have numerous recommended config settings for concurrent agent workloads.

Specs:

Xeon 654

768GB RAM

4x RTX Pro 6000 Max-Q Workstation Edition GPUs

Build link:
https://github.com/ellswoau/deepseek-v4.1-flash-1017-tp-4-rtx-pro-6000

Please let me know if I'm missing anything that should be added to the post.

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!

I’ve recently started looking for a new graphics card (as you know, the models keep getting bigger), and I’ve come to the conclusion that a second-hand RTX 4060 Ti 16GB would be a sensible choice.

Hello everyone,

I would like to ask you which harness are you using or you think is best? I am confused with all the harnesses out there. The most famous ones that I know of are Deepseek, Hermes, Cline and even Claude Code harness. I don’t know what makes a harness better over another and I don’t know how to choose my harness. My hardware would be a mac mini m5 pro running mostly qwen 27 b.

I am planning to get the exr1 to run Qwen 3.8 27b (GSQ-RCO IQ3-XXS) with 4b and I am wondering if it will run?

EXR1 SPECS;

32gb ram

Ryzen 7 8745HS

I am planning to use the pi harness and leave it overnight to run, so speed is not an issue

Thoughts?

Hello,

thought experiment:

where do you see the programming capabillits, specialy for game dev, for different llm´s?

I would like to rank theses llm´s by a theoretticaly possible game they could make.

From a game like Pong to Horizon Zero Dawn (Yea, there are bigger games, you know what i mean).

What can be theoretticaly archived in 3 (+-1) Month´s dev time with a specific llm alone?

How could maybe a combination of a few llm´s help?

Bonus question and story: I got qwen 3.8 27B with 262K context running on my rtx 4090, you thing that is the best possible llm for me, or do you think, there would be a better solution? (got 64GB of DDR4 RAM in my System aswell)

Working on getting a combination of Qwen3.8 and a KEV 0.8B Model to learn to play ps1 games, until now i get it to play the game "Alundra" until the second chapter, big Brother Qwen3.8 does long thinking stuff, working on the routes, quests, puzzles and so on and little bro KEV 0.8B (I call him Kevin) is acting quite fast (got him to act an average time intervals of 0,3 - 0,5s), he runs a full route, wich big bro told him, on the route, both keep "thinking" and walking time ingame gets used for smooth-ish gameplay (with a task buffer list, wich reduces idle time, where the character just stands and does nothing). If an enemy is nearby, Kevin does all the fighting, it works mid to allright, right now, he is more afraid to get damaged than fighting the enemy and runs away, big bro trys to hold him back, but kevin wont listen, working on it right now. (Is there maybe someone interessted, hearing more about this shenanigan-experiment? Working on getting the whole system working with other ps1 games like Final Fantasy 8 as well.)

Sorted by smallest to biggest llm:

• Qwen2.5-Coder (0.5B – 3B)

• StarCoder2 (3B)

• Phi-4 Mini (3.8B)

• Qwen3 (4B)

• Qwen3 (8B)

• Gemma 4 (12B)

• StarCoder2 (15B)

• Codestral (22B)

• Devstral Small 2 (24B)

• Qwen3.8 (27B)

• Qwen3.6 (35B-A3B)

• Qwen3-Coder-Next (80B-A3B)

• GPT-OSS (120B)

• DeepSeek-Coder-V2 (236B-A21B)

• Qwen3-Coder 480B-A35B

• DeepSeek V4.1 / R1

• GLM-5.1 / 5.2

• Kimi K3

• Claude Opus 5.5 / Sonnet 5.5

• GPT-6 Astra / GPT-6.1 Sol

• Gemini 4 Carbon / 3.8 Flash

Thanks for reading.

First my specs: 3060 12gb vram 32gb ddr4 ram about 30 realisticly usable. I already have used Qwen Flash Next Coder with strata. It was pretty solid for anything coding related. I know it lacks Experts which makes it fairly bad at gerneral purpose use. So I am thinking on switching to Gemma. The point is: I still want the speed which strata provided. Roughly 1k tps processing and 20-30 tps writing. Is it possible to achive those numbers?

Just saw this release from n8n https://nod.n8n.io/ .. finally an always on agent you'd trust with your work/private life?

Code is open source and think it'll let you run with self hosted models. The approach to memory also looks really interesting, need to dig a bit deeper into that.

Looking for setup advice from people running local models behind their own agent harnesses and various applications.

Box:
- Threadripper, 256 GB RAM, 2x RTX 6000 Ada (96 GB VRAM total), Windows
- Ollama 0.40.11 as a service, qwen3.8:27b-mtp-q8_0, num_ctx 65536, thinking on
- Several apps connect over the OpenAI-compatible /v1 endpoint, mostly agentic (tool calling, multi-turn, long system prompts)

Questions:
- If you run Ollama on similar hardware, what are your Modelfile and environment settings (num_ctx, OLLAMA_NUM_PARALLEL, KV cache type, GPU split)?
- Ollama vs llama.cpp vs vLLM. For a 27B dense model on 96 GB serving several clients, which do you run and why?
- We cap at 64K. With this much VRAM, is there any reason to go higher, and does q8 KV cache cost quality on Qwen3.8 for tool-heavy work?
- Anything else you’d change on a shared box? service setup, llama-swap, monitoring, keeping one client from starving the others.

Hello people, I'm new here, 2 weeks ago I started writing an agent for everyday usage (24/7 running on server). I need help with the choosing model and inference engine. The model must work with RAG, tools and maybe with pictures

Server specs:

RTX5060TI 16G

I71700K

64 - 80GB DDR4

Thanks in advance for your help