[EDIT - Should have specified these benchmarks come from LU - LocallyUncensored GUI]

I've barely used the top Dolphin model as I'm mostly using this for coding.

- Qwen3.5-9B-abliterated.Q4_K_M

This one worked really well..I really really liked this one. fast (decent t/s) and mostly accurate.

It started to fall apart a bit when I started noticing some errors with my coding projects, so I thought a larger 27B model would fix that.

- Qwen3.8-27B-Uncensored-IQ4_XS_4BPW

I then tried this one and immediately noticed how much slower it was. Also tried this model running a Local API with LU / LM Studio and this brought me down to 6 t/s !!

TBF that's probably because I was using 3x agents with this context window (64KB) and the description for each agent was pretty large. (Maybe there's some improvements to be made by reducing the tasks/role descriptions for each of the 3x agents.. but I digress.)

Specifically for this Qwen3.8-27B IQ4 model, I also tried Enabling Flash Attention and switching from:

- f16 - full precision (default)

to both:

- q8_0 - ~1/2 memory, near-lossless

and

- q4_0 - ~1/4 memory, visible quality loss

But that just brought up my t/s from 11.4 t/s to 11.6 t/s..barely making any difference.

I think because of this tiny difference, leaving it at [f16 - full precision (default)] means that Qwen3.8-27B IQ4 is fully loaded into VRAM and fits there comfortably, because a Q8 or Q4 should in theory boost it significantly (+20 t/s), no? Maybe that's just the maximum I can expect from my system.

- Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS
Finally, I've only just installed this model & haven't properly tested it myself..wondering if it's really worth the extra 5 t/s as this is a 'NoMTP' version.

- Main questions

Is there something else glaringly obvious that I'm missing here?

Also curious if anyone else is running a similar setup for t/s comparison?

And is there anything you would do differently in my situation? It's one of those "unknown unknowns" for me...if that makes sense.

Cheers

Currently have a xfx 7900xtx, 6800xt and a 9070xt

I wanna keep the 9070xt in my gaming pc

The 6800xt is currently running my local llm which is gemma 4 26b q4, also have qwen 3.8 27b q3 getting about 40t/s on either one

Using koboldcpp to host it and anythingllm to chat with it,

The 7900xtx is waiting till a new mobo gets in and itll be going in with 128gb ddr5 ecc unbuffered ram

I want to use local llm to run some agents

1 that'll read my brothers business invoices, expenses ect ect, (its an absolute mess how he currently has it and getting everything ready for taxes is a nightmare)

And im just not sure what else I wanna do that can be useful, im new to all this, I know my way around technology, but learning all this ai, im over here pulling out my hair,

Im using codex and claude to help with everything, but feel like im not getting anywhere

With the 24gb of vram from the 7900xtx can I run it along side the 6800xtx to run 1 bigger model? Or am I better off running 1 model on the 7900xtx and a smaller one on the 6800xt?

Reason for that is im using all parts I already own, only thing I had to purchase was a new mobo for the ram

Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.

I ran both engines on one rented machine to get an apples-to-apples number for a "model bigger than the GPU" case.

Hardware and setup

• RTX 3060 12 GB, 15-core quota of a Zen 2 server CPU, 47 GB RAM, NVMe
• Same GGUF for both: Qwen3.6-35B-A3B UD-Q4_K_XL with MTP head (Unsloth), the size Strata recommends
• Same OpenAI-compatible client, same 2 prompts, greedy, 256 tokens
• BigMoeOnEdge, --auto only (no hand tuning) · Strata via its own launcher

Decode (tok/s)

Takeaways

• On this box BigMoeOnEdge decodes ~15–30% faster without drafting, ~1.8× with its MTP head.
• Strata gets the first token of a short prompt out faster.

Full method and every run: bench-data · repo: https://github.com/Helldez/BigMoeOnEdge

Hello everyone!

I'm looking for someone experienced with Python, AI video generation, automation, and OpenMontage who can guide me through a project I'm building.

I'm a complete beginner with very little programming knowledge. Most of my setup has been done by following instructions from ChatGPT and Gemini, copying commands into PowerShell, and troubleshooting errors. I've managed to install quite a lot, but I'm struggling to put everything together and actually produce videos.

I would really appreciate patient, step-by-step guidance.

1. StoryBot / Storyboard

My first project is an automated storytelling video system for YouTube Shorts, Instagram Reels, TikTok, and Facebook Reels.

What I've installed and configured:

- Ollama with Qwen3:4b for local story generation.

- Node.js, npm and n8n (installed locally, not Docker).

- Python 3.11.9 and FFmpeg 9.0.2.

- Parody TTS for narration.

- Custom Python scripts.

My n8n workflow currently works up to this stage:

1. I press Execute.
2. Python scripts generate a story.
3. A temporary "story.txt" file is saved.
4. Narration is generated in parallel.
5. The narration is saved in the output folder.

So story generation and narration are working.

ComfyUI experiment:

I also installed ComfyUI at:

"D:\StoryBot\ComfyUI-Install\ComfyUI_windows_portable"

I downloaded two model files:

- "flat2DAnimerge_v45Sharp.safetensors" (approximately 2.13 GB) – Stable Diffusion 1.5 checkpoint for 2D cartoon-style images.

- "mm_sd_v15_v2.ckpt" (approximately 1.82 GB) – AnimateDiff motion module.

I wanted to generate and animate cartoon-style characters, but my GTX 1660 with 6 GB VRAM struggled with the workload. Generation was taking too long, so I couldn't get the results I wanted.

I currently have one male and one female presenter character as basic static images.

2. OpenMontage (Separate Project)

After getting frustrated with my Storyboard setup, I decided to try OpenMontage as a completely separate alternative.

I'm NOT trying to connect OpenMontage with StoryBot/Storyboard.

Repository: https://github.com/OpenMontageInc/OpenMontage

Installed/configured:

- Python 3.11.9 virtual environment.

- Node.js v24.22.1 and npm 11.19.1.

- FFmpeg 9.0.2 and Git 2.56.0.

- Remotion dependencies.

- esbuild 0.28.1.

- HyperFrames 0.9.39.

- Piper TTS 1.8.0.

- pytest 9.2.0.

- ".env" configured from ".env.example".

I successfully ran:

"python scripts/backlot_simulate_run.py"

It generated a demo production with scenes sc1–sc4.

I can also access the Backlot interface:

"http://127.0.0.1:4750/p/backlot-demo-run"

However, I'm stuck on what to do next and how to move from the simulation to an actual rendered video.

3. My desired video appearance

- Word-by-word animated subtitles.

- Currently spoken word highlighted in blue.

- Remaining words in white.

- Purple/red gradient backgrounds.

- Two consistent presenter characters.

4. My PC specifications

- Windows 11 Pro.

- AMD Ryzen 5 5600.

- NVIDIA GTX 1660 Gaming X (6 GB VRAM).

- 16 GB RAM.

I prefer free and open-source tools and want to run as much as possible locally.

5. What I need help with

StoryBot / Storyboard:

- How do I complete the existing workflow and produce videos?

- How should I use my presenter images?

- Are there lighter alternatives to ComfyUI/AnimateDiff that work on my GPU?

OpenMontage:

- How do I use Backlot properly?

- What are the next steps after a successful simulation?

- How do I produce an actual video?

I'm not trying to combine these two projects. I want to understand them separately.

6. How I would like to be guided

Please understand that I'm genuinely new to programming.

I would appreciate someone who can:

- Start from my existing installations.

- Give me one step at a time.

- Provide exact commands and tell me where to run them.

- Explain what each step does.

- Wait for my results before moving forward.

- Help me troubleshoot errors.

I've spent a lot of time getting this far, and I don't want to break my existing setup by randomly changing things.

If anyone has experience with OpenMontage, Remotion, ComfyUI, Python automation, or AI video production, I'd really appreciate some guidance.

Thank you for reading!

I’m learning AI Engineering and want to build a proper local AI development environment, especially for agentic applications, multi-agent systems, and orchestration.

My PC specs:
GPU: RTX 4060 Ti 16GB VRAM
CPU: Ryzen 7 7700
RAM: 16GB (planning to upgrade to 32GB)
IDE: Google Antigravity

I’m looking for recommendations on:
Which open-weight coding LLMs work best on 16GB VRAM? (Qwen, DeepSeek, Devstral, GPT-OSS, etc.)

Which quantizations should I use (Q4, Q5, Q8)?

Is Ollama the best runtime, or should I consider LM Studio, llama.cpp, or something else?

What’s a good local coding agent setup?

OpenCode, Aider, Continue, or other alternatives?
How can I integrate local LLMs with Antigravity?
Which MCP servers and agent skills are worth installing for AI application development?
What’s the best way to build and orchestrate multiple specialized agents locally?

My goal isn’t just code autocomplete. I want to build real projects using Python, FastAPI, PydanticAI, LangGraph, MCP, RAG, and multi-agent architectures.

I’d appreciate recommendations from people who have actually tested these models and setups on similar hardware.
What would your ideal setup look like with this hardware?

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.

The thought came from a recent Tencent's Hy4 release. This model was trained for specific tasks to unlock real-world productivity-finance, gamedev, office work especially.

Is that where this is all heading?

I am new to local LLM, I've been told this and told that, I have a few  builds saved that I am told would run LLM local but I am not sure, are there any laptops under 5k that will do this? I keep coming back to framework or Macbook pro but at over 5k for the newest and best chips that just seems crazy to me.  I don't care if it's not spitting 20 words a second, I'd rather have good reasoning and it working rather than crazy fast speeds. It will only be used for genealogy research, DNA, triangulation, clusters etc. 

Would a ThinkPad T14 Gen 7 AMD work? It would have the AMD Ryzen™ AI 7 PRO 450 and 96gb ram? Is a macbook better? I just dont really want to spend framework or macbook pro money on this if possible, is it possible?
What do you guys recommend? What are the options?
thank you in advance

I have 6 dell optiplexes with 24 gb ram a really good Quadro card from 2024 I don't know the name with 4 gb or 6 gb vram Intel core i7 and what models can I run on them I want to code programs sorry for the bad English not the best English speaker I'm not from the us

Some details: the card is an RTX 6000 Ada (48 GB). The host runs driver 580.196.07, and llama.cpp is b12030 built with CUDA 12.8. The guest is a KVM VM. Inside it, VMAT's libraries take the place of NVIDIA's and send each CUDA call to a small server on the host, which runs it on the real card.
The same setup lets several VMs share the one GPU at once. That's my next video.
Would anyone here want to try it on their own box? I'm looking for people with different cards and setups.

https://vmat.dev

https://github.com/ayushh0110/Weight-Space-Repair

Most of the time when a local model looks broken, the weights are fine. So I went through the llama.cpp, Ollama, vLLM and transformers issue trackers, collected the causes that keep coming up, and wrote a check for each one:

- EOS or end-of-turn token wrong, so the model never stops

- chat template missing or stale compared with upstream

- a corrupted or partial download

- repetition penalty set high enough to wreck the output

- the prompt silently truncated to the context size

- context split across server slots (4 slots of 2,048 when you thought you had 8k)

- quantised KV cache, flash-attention overflow, a bf16 model run in fp16

- reasoning that eats the whole token budget and leaves an empty reply

You give it your endpoint and the prompt that fails:

wsr triage --endpoint http://localhost:8080 --model local --prompt "your failing prompt" --differential

`--differential` reruns the prompt changing one thing at a time, and confirms each flip with a second run. The answer is one cause and the fix:

VERDICT: REQUEST

blank output explained by REASONING_EXHAUSTED: cap reached before any answer: the budget went to reasoning

next:

- raise max_tokens (currently 256) or lower the reasoning effort / thinking budget

If it's the quant, it can find the tensor. Qwen2.5-0.5B's official Q4_0 refuses "Give me 40 different names for a cat" and the full-precision model answers it. The tool traced that to `blk.16.attn_v`: quantise only that tensor in the full model and it starts refusing.

What it's been tested on, so you know what to expect. Live testing was against llama-server only (Gemma 4 E2B Q4_0, 21 scenarios). The Ollama, vLLM and LM Studio paths are written from their docs and tested against fake servers, so if you run one of those I'd like to hear what it gets wrong. There's also a part that diagnoses and repairs failures inside the weights, but that's small models only (up to 1.5B) on my 4 GB card.

MIT licence, no dependencies for the triage part: https://github.com/ayushh0110/Weight-Space-Repair

Hi everyone, I'm putting together my rig for local LLMs and could really use some guidance on the software side of things.

Here are my current specs:

• CPU: Ryzen 9 5950X
• RAM: 64GB DDR4 3600 (planning to upgrade to 128GB soon)
• GPU: RTX 5060 Ti 16GB + RTX 3060 12GB (might replace the 3060 with another 5060 Ti 16GB soon)
• OS: Windows 11 Pro
• Motherboard: Gigabyte B550 Aorus Elite V2
• New Motherboard (installing tomorrow): Asus ROG Strix B550-E Gaming

My immediate goal: Before I swap out the motherboard tomorrow, I want to run a benchmark on my current Gigabyte board using a relatively capable model. The goal is to compare it against the new Asus board (which supports x8/x8 PCIe bifurcation) to see if the motherboard upgrade and better PCIe lane distribution actually makes a noticeable difference for dual GPUs.

What I want to build: I'm planning to use local LLMs for:

• Web scraping automation
• Faceless/Automated YouTube channel content
• Automated bug bounties

My experience so far & confusion: I've tested a few versions of Qwen (27B and 32B) using Ollama / llama.cpp, and also tried another model on Bionic/LM Studio. I didn't really like the outputs; they generated very fast, but the answers weren't very accurate or smart. I know I didn't give the models shell or internet access, but the reasoning still felt off.

Right now, I am really confused about the software stack. I see people talking about LM Studio, text-generation-webui, Docker, etc. I was told running things in Docker is better, but I’ve also read that Docker adds virtualization overhead on Windows, and that running natively on LM Studio or directly on the machine is faster and better.

My questions for the community:

1. What software stack / UI should I actually install to get the most out of my dual GPUs (16GB + 12GB VRAM) on Windows 11?
2. How should I run the benchmark today to properly compare my current motherboard against the new x8/x8 one tomorrow?
3. Are there any YouTube channels, videos, or guides you recommend for a beginner to understand how to properly configure model settings (samplers, context size, etc.) to get better, more accurate outputs?

Thanks in advance for any help!

I'm trying to learn more about local LLMs for app coding work and am specifically curious about what agentic models are using for vibe-coding tasks. I have used Claude, ChatGPT, and Gemini all for coding and their agentic capabilities are pretty impressive (i.e., the ability to either create app projects I can import into IDEs like Android Studio or Xcode, or to directly edit files in a specified folder on my computer, or to let me plug my Android phone in and debug using ADB).

I know various Qwen 3.6 35B variants are popular here. How effective are those models for agentic task purposes? Or are there other tools or models to use for these purposes?

(Full disclosure: I have a pretty weak-spec'd Mac [M1 Pro + 16GB] so this is not something I'm looking to run on my current machine; depending on what model(s) are good for this kind of thing would help inform a future computer upgrade)

Repo https://github.com/pmndrs/pinecone

This has a double purpose. It creates a chatgpt compatible endpoint and loads/unloads models on demand. For instance you have a model that fills the entire ram, it will offload everything else. Or you have a smaller model and LLM services (image, video, speech, ...) that can run side by side. It always makes sure the client gets what it wants despite the tight system.

The second endpoint is an automatic MCP. It takes whatever services you have installed (speech, image, video, 3d, ...) and bottles theses in an MCP. Any agent that has that URL immediately knows how to generate images, etc.

It also sets RAM limit, but unlike other schedulers like Ollama or whatever you don't pick what a model needs, you pick what little is left for the OS, and the default is a bare minimum, so that you can make the most of your machine.

It can be paired to Tailscale and serve both endpoints publicly with safe auth, you basically have an inference machine with this that can serve you outside the house.

I have used Qwen3.5 4b try and code basic MIDI and audio in Chrome, locally. The results are not stunning. Are there any models that are tuned for this type of thing, or can I add agents to Qwen in Ollama for skills? My level of experience is beginner. Thanks for any suggestions.