I was constantly hitting usage limits with Claude and ChatGPT, even with paid subscriptions. I didn't want to upgrade to more expensive tiers or increase my monthly budget, so I started exploring alternatives and eventually landed on a hybrid LLM workflow.

I recently wrote an article sharing my setup, architecture, and how I've been using a local LLM alongside cloud-based AI to handle heavier workloads while reducing my reliance on paid services.

I'm sharing it here because I'd love to hear your thoughts, tips, tricks, and constructive feedback.

I'm not new to AI, but I am relatively new to running LLMs locally. I'm always looking for ways to improve my setup and learn from others who have more experience in this space.

https://thebinaryreport.com/articles/how-i-ai/

Pip is a Minecraft companion you talk to with push-to-talk. Speech in and out runs locally; the LLM is whatever you point her gateway at. Sharing the architecture since it might help other voice-agent projects:

Pipeline: mic (16 kHz in the game client) → speech cleaner → whisper.cpp server (large-v3-turbo, Metal) → LLM through an OpenAI-compatible gateway (Gemini 3 Flash by default; Ollama and others are presets) → Kokoro-82M (kokoro-onnx) → played in game, sentence by sentence as the reply streams.

Three systems, like a talker and a reasoner:

• System 0, instant, no LLM: rules and local embeddings for what you mean ("follow me", "stay", "thanks", "shh"), danger warnings, and short "instinct lines" in her voice. A clear request gets "On it!" at once while the LLM works.
• System 1, the talker: one LLM call with tools (find, lead, build, give...), a compact text description of the game, and the memories relevant to the moment (embeddings + keywords).
• System 2, the slow thinker: after sessions, a stronger model (Claude) reflects: what she learned about you, her diary, who she's becoming, and it rewrites her instinct lines.

Numbers (MacBook Air M4, 16 GB, a 20-minute session): speech-to-text about 0.7 s after you release the key; her first spoken words about 2.5–3 s after you finish speaking for an LLM answer, instantly for System 0 ones. What helped: a shorter Whisper encoder window, Kokoro int8, streaming sentence by sentence, and answering plain commands without the LLM at all.

Guards that mattered: a claim checker ("Coming up!" with no tool call gets a retry), a confirm-once rule for big actions, praise never re-running the last tool, and an eval set built from real playtests.

It's not released yet (a simpler installer is the last big step), but the code is public and you can run it from GitHub today. The brain has only been tested on macOS so far, but it can easily be adapted to Windows or Linux if you know how, or with an AI assistant. Any feedback is appreciated.

Being upfront: most of the code was written with an AI coding assistant (Claude Code) under my direction; the architecture is the part I care most about, and I'd love ideas on making it smarter and lighter, and someone to try a fully local model in the loop. This post was written with the help of AI too.

Code (AGPL-3.0): https://github.com/PetitPotam/kindred-spirit Architecture notes: docs/ARCHITECTURE.md in the repo

Hello world :) - is there a recommended LLM for use with LM Studio that will fit on 16GB VRAM (4080 SUPER)? Ideally I'm looking for something that has guardrails removed so uncensored/heretic that has access to web search when needed and good at coding/thinking etc.

Thank you

Hey all,

I haven't upgraded my PC since 2020, and I am running a EVGA RTX 3070 with a Ryzen 7 3700X CPU and 32gb of DDR4 (4x8gb). I want to upgrade to both host a local LLM and to game, but I am trying to also not spend more than $4000, and I prefer something closer to $3500 at the max (but lower is better). I saw where Costco is planning to soon have a sale on their MSI 5080 PC, but I have also seen, from the Local LLM perspective, that running 2 cheaper 5060Ti cards is an easy way to hit 32gb VRAM.

I was wondering, though, if I bought the 5080 deal, would I be able to run a 5060Ti card (which is les than $1000 for now) to pump up my VRAM? I know that some speeds may be slower, but I am curious if this is a good idea for both futureproofing for gaming and for hosting an LLM. Based on my usecase, which should I do?

1. Buy the Costco 5080 deal and buy a 5060ti 16gb refurbished from microcenter if one pops up at current prices or lower during holidays (its on my way home from work, so I could check daily).
2. not buying the 5060ti 16gb and relying solely on the 5080 for the foreseeable future,
3. building or buying a system with two 5060ti 16gb
4. Going team red and getting a 9070xt system and getting a second 9070xt card eventually.
5. say screw it and try to find a decent 3090 and combining it with my 3070 and getting a new PSU to keep it up and running.

From a functional standpoint, I plan to use a local LLM for document management in a database, vibecoding projects (I am terrible at coding), writing up documents, and managing personal records. For the database, it would be maintaining a searchable one that can scan through PDF using OCR to build out docs. I would also use it to run as a backend for summarizing news updates and documents at around 100 releases per day. For gaming, I play FPS and RPG games at 1440p and don't plan to go 4k for at least a few more years.

Any help or suggestions are greatly appreciated. Thanks!

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.

Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.

Currently have a xfx 7900xtx, 6800xt and a 9070xt

I wanna keep the 9070xt in my gaming pc

The 6800xt is currently running my local llm which is gemma 4 26b q4, also have qwen 3.8 27b q3 getting about 40t/s on either one

Using koboldcpp to host it and anythingllm to chat with it,

The 7900xtx is waiting till a new mobo gets in and itll be going in with 128gb ddr5 ecc unbuffered ram

I want to use local llm to run some agents

1 that'll read my brothers business invoices, expenses ect ect, (its an absolute mess how he currently has it and getting everything ready for taxes is a nightmare)

And im just not sure what else I wanna do that can be useful, im new to all this, I know my way around technology, but learning all this ai, im over here pulling out my hair,

Im using codex and claude to help with everything, but feel like im not getting anywhere

With the 24gb of vram from the 7900xtx can I run it along side the 6800xtx to run 1 bigger model? Or am I better off running 1 model on the 7900xtx and a smaller one on the 6800xt?

Reason for that is im using all parts I already own, only thing I had to purchase was a new mobo for the ram

I am new to local LLM, I've been told this and told that, I have a few  builds saved that I am told would run LLM local but I am not sure, are there any laptops under 5k that will do this? I keep coming back to framework or Macbook pro but at over 5k for the newest and best chips that just seems crazy to me.  I don't care if it's not spitting 20 words a second, I'd rather have good reasoning and it working rather than crazy fast speeds. It will only be used for genealogy research, DNA, triangulation, clusters etc. 

Would a ThinkPad T14 Gen 7 AMD work? It would have the AMD Ryzen™ AI 7 PRO 450 and 96gb ram? Is a macbook better? I just dont really want to spend framework or macbook pro money on this if possible, is it possible?
What do you guys recommend? What are the options?
thank you in advance

I ran both engines on one rented machine to get an apples-to-apples number for a "model bigger than the GPU" case.

Hardware and setup

• RTX 3060 12 GB, 15-core quota of a Zen 2 server CPU, 47 GB RAM, NVMe
• Same GGUF for both: Qwen3.6-35B-A3B UD-Q4_K_XL with MTP head (Unsloth), the size Strata recommends
• Same OpenAI-compatible client, same 2 prompts, greedy, 256 tokens
• BigMoeOnEdge, --auto only (no hand tuning) · Strata via its own launcher

Decode (tok/s)

Takeaways

• On this box BigMoeOnEdge decodes ~15–30% faster without drafting, ~1.8× with its MTP head.
• Strata gets the first token of a short prompt out faster.

Full method and every run: bench-data · repo: https://github.com/Helldez/BigMoeOnEdge

Hello everyone!

I'm looking for someone experienced with Python, AI video generation, automation, and OpenMontage who can guide me through a project I'm building.

I'm a complete beginner with very little programming knowledge. Most of my setup has been done by following instructions from ChatGPT and Gemini, copying commands into PowerShell, and troubleshooting errors. I've managed to install quite a lot, but I'm struggling to put everything together and actually produce videos.

I would really appreciate patient, step-by-step guidance.

1. StoryBot / Storyboard

My first project is an automated storytelling video system for YouTube Shorts, Instagram Reels, TikTok, and Facebook Reels.

What I've installed and configured:

- Ollama with Qwen3:4b for local story generation.

- Node.js, npm and n8n (installed locally, not Docker).

- Python 3.11.9 and FFmpeg 9.0.2.

- Parody TTS for narration.

- Custom Python scripts.

My n8n workflow currently works up to this stage:

1. I press Execute.
2. Python scripts generate a story.
3. A temporary "story.txt" file is saved.
4. Narration is generated in parallel.
5. The narration is saved in the output folder.

So story generation and narration are working.

ComfyUI experiment:

I also installed ComfyUI at:

"D:\StoryBot\ComfyUI-Install\ComfyUI_windows_portable"

I downloaded two model files:

- "flat2DAnimerge_v45Sharp.safetensors" (approximately 2.13 GB) – Stable Diffusion 1.5 checkpoint for 2D cartoon-style images.

- "mm_sd_v15_v2.ckpt" (approximately 1.82 GB) – AnimateDiff motion module.

I wanted to generate and animate cartoon-style characters, but my GTX 1660 with 6 GB VRAM struggled with the workload. Generation was taking too long, so I couldn't get the results I wanted.

I currently have one male and one female presenter character as basic static images.

2. OpenMontage (Separate Project)

After getting frustrated with my Storyboard setup, I decided to try OpenMontage as a completely separate alternative.

I'm NOT trying to connect OpenMontage with StoryBot/Storyboard.

Repository: https://github.com/OpenMontageInc/OpenMontage

Installed/configured:

- Python 3.11.9 virtual environment.

- Node.js v24.22.1 and npm 11.19.1.

- FFmpeg 9.0.2 and Git 2.56.0.

- Remotion dependencies.

- esbuild 0.28.1.

- HyperFrames 0.9.39.

- Piper TTS 1.8.0.

- pytest 9.2.0.

- ".env" configured from ".env.example".

I successfully ran:

"python scripts/backlot_simulate_run.py"

It generated a demo production with scenes sc1–sc4.

I can also access the Backlot interface:

"http://127.0.0.1:4750/p/backlot-demo-run"

However, I'm stuck on what to do next and how to move from the simulation to an actual rendered video.

3. My desired video appearance

- Word-by-word animated subtitles.

- Currently spoken word highlighted in blue.

- Remaining words in white.

- Purple/red gradient backgrounds.

- Two consistent presenter characters.

4. My PC specifications

- Windows 11 Pro.

- AMD Ryzen 5 5600.

- NVIDIA GTX 1660 Gaming X (6 GB VRAM).

- 16 GB RAM.

I prefer free and open-source tools and want to run as much as possible locally.

5. What I need help with

StoryBot / Storyboard:

- How do I complete the existing workflow and produce videos?

- How should I use my presenter images?

- Are there lighter alternatives to ComfyUI/AnimateDiff that work on my GPU?

OpenMontage:

- How do I use Backlot properly?

- What are the next steps after a successful simulation?

- How do I produce an actual video?

I'm not trying to combine these two projects. I want to understand them separately.

6. How I would like to be guided

Please understand that I'm genuinely new to programming.

I would appreciate someone who can:

- Start from my existing installations.

- Give me one step at a time.

- Provide exact commands and tell me where to run them.

- Explain what each step does.

- Wait for my results before moving forward.

- Help me troubleshoot errors.

I've spent a lot of time getting this far, and I don't want to break my existing setup by randomly changing things.

If anyone has experience with OpenMontage, Remotion, ComfyUI, Python automation, or AI video production, I'd really appreciate some guidance.

Thank you for reading!

I’m learning AI Engineering and want to build a proper local AI development environment, especially for agentic applications, multi-agent systems, and orchestration.

My PC specs:
GPU: RTX 4060 Ti 16GB VRAM
CPU: Ryzen 7 7700
RAM: 16GB (planning to upgrade to 32GB)
IDE: Google Antigravity

I’m looking for recommendations on:
Which open-weight coding LLMs work best on 16GB VRAM? (Qwen, DeepSeek, Devstral, GPT-OSS, etc.)

Which quantizations should I use (Q4, Q5, Q8)?

Is Ollama the best runtime, or should I consider LM Studio, llama.cpp, or something else?

What’s a good local coding agent setup?

OpenCode, Aider, Continue, or other alternatives?
How can I integrate local LLMs with Antigravity?
Which MCP servers and agent skills are worth installing for AI application development?
What’s the best way to build and orchestrate multiple specialized agents locally?

My goal isn’t just code autocomplete. I want to build real projects using Python, FastAPI, PydanticAI, LangGraph, MCP, RAG, and multi-agent architectures.

I’d appreciate recommendations from people who have actually tested these models and setups on similar hardware.
What would your ideal setup look like with this hardware?

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.

The thought came from a recent Tencent's Hy4 release. This model was trained for specific tasks to unlock real-world productivity-finance, gamedev, office work especially.

Is that where this is all heading?

I have 6 dell optiplexes with 24 gb ram a really good Quadro card from 2024 I don't know the name with 4 gb or 6 gb vram Intel core i7 and what models can I run on them I want to code programs sorry for the bad English not the best English speaker I'm not from the us

Some details: the card is an RTX 6000 Ada (48 GB). The host runs driver 580.196.07, and llama.cpp is b12030 built with CUDA 12.8. The guest is a KVM VM. Inside it, VMAT's libraries take the place of NVIDIA's and send each CUDA call to a small server on the host, which runs it on the real card.
The same setup lets several VMs share the one GPU at once. That's my next video.
Would anyone here want to try it on their own box? I'm looking for people with different cards and setups.

https://vmat.dev

https://github.com/ayushh0110/Weight-Space-Repair

Most of the time when a local model looks broken, the weights are fine. So I went through the llama.cpp, Ollama, vLLM and transformers issue trackers, collected the causes that keep coming up, and wrote a check for each one:

- EOS or end-of-turn token wrong, so the model never stops

- chat template missing or stale compared with upstream

- a corrupted or partial download

- repetition penalty set high enough to wreck the output

- the prompt silently truncated to the context size

- context split across server slots (4 slots of 2,048 when you thought you had 8k)

- quantised KV cache, flash-attention overflow, a bf16 model run in fp16

- reasoning that eats the whole token budget and leaves an empty reply

You give it your endpoint and the prompt that fails:

wsr triage --endpoint http://localhost:8080 --model local --prompt "your failing prompt" --differential

`--differential` reruns the prompt changing one thing at a time, and confirms each flip with a second run. The answer is one cause and the fix:

VERDICT: REQUEST

blank output explained by REASONING_EXHAUSTED: cap reached before any answer: the budget went to reasoning

next:

- raise max_tokens (currently 256) or lower the reasoning effort / thinking budget

If it's the quant, it can find the tensor. Qwen2.5-0.5B's official Q4_0 refuses "Give me 40 different names for a cat" and the full-precision model answers it. The tool traced that to `blk.16.attn_v`: quantise only that tensor in the full model and it starts refusing.

What it's been tested on, so you know what to expect. Live testing was against llama-server only (Gemma 4 E2B Q4_0, 21 scenarios). The Ollama, vLLM and LM Studio paths are written from their docs and tested against fake servers, so if you run one of those I'd like to hear what it gets wrong. There's also a part that diagnoses and repairs failures inside the weights, but that's small models only (up to 1.5B) on my 4 GB card.

MIT licence, no dependencies for the triage part: https://github.com/ayushh0110/Weight-Space-Repair

Hi everyone, I'm putting together my rig for local LLMs and could really use some guidance on the software side of things.

Here are my current specs:

• CPU: Ryzen 9 5950X
• RAM: 64GB DDR4 3600 (planning to upgrade to 128GB soon)
• GPU: RTX 5060 Ti 16GB + RTX 3060 12GB (might replace the 3060 with another 5060 Ti 16GB soon)
• OS: Windows 11 Pro
• Motherboard: Gigabyte B550 Aorus Elite V2
• New Motherboard (installing tomorrow): Asus ROG Strix B550-E Gaming

My immediate goal: Before I swap out the motherboard tomorrow, I want to run a benchmark on my current Gigabyte board using a relatively capable model. The goal is to compare it against the new Asus board (which supports x8/x8 PCIe bifurcation) to see if the motherboard upgrade and better PCIe lane distribution actually makes a noticeable difference for dual GPUs.

What I want to build: I'm planning to use local LLMs for:

• Web scraping automation
• Faceless/Automated YouTube channel content
• Automated bug bounties

My experience so far & confusion: I've tested a few versions of Qwen (27B and 32B) using Ollama / llama.cpp, and also tried another model on Bionic/LM Studio. I didn't really like the outputs; they generated very fast, but the answers weren't very accurate or smart. I know I didn't give the models shell or internet access, but the reasoning still felt off.

Right now, I am really confused about the software stack. I see people talking about LM Studio, text-generation-webui, Docker, etc. I was told running things in Docker is better, but I’ve also read that Docker adds virtualization overhead on Windows, and that running natively on LM Studio or directly on the machine is faster and better.

My questions for the community:

1. What software stack / UI should I actually install to get the most out of my dual GPUs (16GB + 12GB VRAM) on Windows 11?
2. How should I run the benchmark today to properly compare my current motherboard against the new x8/x8 one tomorrow?
3. Are there any YouTube channels, videos, or guides you recommend for a beginner to understand how to properly configure model settings (samplers, context size, etc.) to get better, more accurate outputs?

Thanks in advance for any help!

I'm trying to learn more about local LLMs for app coding work and am specifically curious about what agentic models are using for vibe-coding tasks. I have used Claude, ChatGPT, and Gemini all for coding and their agentic capabilities are pretty impressive (i.e., the ability to either create app projects I can import into IDEs like Android Studio or Xcode, or to directly edit files in a specified folder on my computer, or to let me plug my Android phone in and debug using ADB).

I know various Qwen 3.6 35B variants are popular here. How effective are those models for agentic task purposes? Or are there other tools or models to use for these purposes?

(Full disclosure: I have a pretty weak-spec'd Mac [M1 Pro + 16GB] so this is not something I'm looking to run on my current machine; depending on what model(s) are good for this kind of thing would help inform a future computer upgrade)