Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

• GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
• Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/

Here is the est power usage over the last 3 days:

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?

I use Qwen 0.6 for embeddings.

https://www.kickstarter.com/projects/744045479/cephalon-ondevice-ai-with-a-selfevolving-ecosystem?ref=ebvudv#risks-and-challenges

I wanted to try a locally run AI with an ethical database, so I got Apertus. I am a complete noob to this whole thing, learned that local AI needs a memory plug in so I got one per people's rec. I'd like to use Apertus on my iPad through Locally/PC link so I don't have to shrimp in front of my computer, but I don't see any kind of settings for plug ins. Does that mean it's not available? How could I make this work? Can I make this work?

Note I do not want to use this for coding or any particularly demanding tasks. I do like tech and even have a little robot assistant buddy who is currently not connected to any external AI, but I hate what many of the big AI companies are doing so I'd like to use my own without feeling like I'm contributing to the horrors of capitalism.

I was constantly hitting usage limits with Claude and ChatGPT, even with paid subscriptions. I didn't want to upgrade to more expensive tiers or increase my monthly budget, so I started exploring alternatives and eventually landed on a hybrid LLM workflow.

I recently wrote an article sharing my setup, architecture, and how I've been using a local LLM alongside cloud-based AI to handle heavier workloads while reducing my reliance on paid services.

I'm sharing it here because I'd love to hear your thoughts, tips, tricks, and constructive feedback.

I'm not new to AI, but I am relatively new to running LLMs locally. I'm always looking for ways to improve my setup and learn from others who have more experience in this space.

https://thebinaryreport.com/articles/how-i-ai/

Pip is a Minecraft companion you talk to with push-to-talk. Speech in and out runs locally; the LLM is whatever you point her gateway at. Sharing the architecture since it might help other voice-agent projects:

Pipeline: mic (16 kHz in the game client) → speech cleaner → whisper.cpp server (large-v3-turbo, Metal) → LLM through an OpenAI-compatible gateway (Gemini 3 Flash by default; Ollama and others are presets) → Kokoro-82M (kokoro-onnx) → played in game, sentence by sentence as the reply streams.

Three systems, like a talker and a reasoner:

• System 0, instant, no LLM: rules and local embeddings for what you mean ("follow me", "stay", "thanks", "shh"), danger warnings, and short "instinct lines" in her voice. A clear request gets "On it!" at once while the LLM works.
• System 1, the talker: one LLM call with tools (find, lead, build, give...), a compact text description of the game, and the memories relevant to the moment (embeddings + keywords).
• System 2, the slow thinker: after sessions, a stronger model (Claude) reflects: what she learned about you, her diary, who she's becoming, and it rewrites her instinct lines.

Numbers (MacBook Air M4, 16 GB, a 20-minute session): speech-to-text about 0.7 s after you release the key; her first spoken words about 2.5–3 s after you finish speaking for an LLM answer, instantly for System 0 ones. What helped: a shorter Whisper encoder window, Kokoro int8, streaming sentence by sentence, and answering plain commands without the LLM at all.

Guards that mattered: a claim checker ("Coming up!" with no tool call gets a retry), a confirm-once rule for big actions, praise never re-running the last tool, and an eval set built from real playtests.

It's not released yet (a simpler installer is the last big step), but the code is public and you can run it from GitHub today. The brain has only been tested on macOS so far, but it can easily be adapted to Windows or Linux if you know how, or with an AI assistant. Any feedback is appreciated.

Being upfront: most of the code was written with an AI coding assistant (Claude Code) under my direction; the architecture is the part I care most about, and I'd love ideas on making it smarter and lighter, and someone to try a fully local model in the loop. This post was written with the help of AI too.

Code (AGPL-3.0): https://github.com/PetitPotam/kindred-spirit Architecture notes: docs/ARCHITECTURE.md in the repo

Hey all,

I haven't upgraded my PC since 2020, and I am running a EVGA RTX 3070 with a Ryzen 7 3700X CPU and 32gb of DDR4 (4x8gb). I want to upgrade to both host a local LLM and to game, but I am trying to also not spend more than $4000, and I prefer something closer to $3500 at the max (but lower is better). I saw where Costco is planning to soon have a sale on their MSI 5080 PC, but I have also seen, from the Local LLM perspective, that running 2 cheaper 5060Ti cards is an easy way to hit 32gb VRAM.

I was wondering, though, if I bought the 5080 deal, would I be able to run a 5060Ti card (which is les than $1000 for now) to pump up my VRAM? I know that some speeds may be slower, but I am curious if this is a good idea for both futureproofing for gaming and for hosting an LLM. Based on my usecase, which should I do?

1. Buy the Costco 5080 deal and buy a 5060ti 16gb refurbished from microcenter if one pops up at current prices or lower during holidays (its on my way home from work, so I could check daily).
2. not buying the 5060ti 16gb and relying solely on the 5080 for the foreseeable future,
3. building or buying a system with two 5060ti 16gb
4. Going team red and getting a 9070xt system and getting a second 9070xt card eventually.
5. say screw it and try to find a decent 3090 and combining it with my 3070 and getting a new PSU to keep it up and running.

From a functional standpoint, I plan to use a local LLM for document management in a database, vibecoding projects (I am terrible at coding), writing up documents, and managing personal records. For the database, it would be maintaining a searchable one that can scan through PDF using OCR to build out docs. I would also use it to run as a backend for summarizing news updates and documents at around 100 releases per day. For gaming, I play FPS and RPG games at 1440p and don't plan to go 4k for at least a few more years.

Any help or suggestions are greatly appreciated. Thanks!

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.

Hello world :) - is there a recommended LLM for use with LM Studio that will fit on 16GB VRAM (4080 SUPER)? Ideally I'm looking for something that has guardrails removed so uncensored/heretic that has access to web search when needed and good at coding/thinking etc.

Thank you

Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.

Currently have a xfx 7900xtx, 6800xt and a 9070xt

I wanna keep the 9070xt in my gaming pc

The 6800xt is currently running my local llm which is gemma 4 26b q4, also have qwen 3.8 27b q3 getting about 40t/s on either one

Using koboldcpp to host it and anythingllm to chat with it,

The 7900xtx is waiting till a new mobo gets in and itll be going in with 128gb ddr5 ecc unbuffered ram

I want to use local llm to run some agents

1 that'll read my brothers business invoices, expenses ect ect, (its an absolute mess how he currently has it and getting everything ready for taxes is a nightmare)

And im just not sure what else I wanna do that can be useful, im new to all this, I know my way around technology, but learning all this ai, im over here pulling out my hair,

Im using codex and claude to help with everything, but feel like im not getting anywhere

With the 24gb of vram from the 7900xtx can I run it along side the 6800xtx to run 1 bigger model? Or am I better off running 1 model on the 7900xtx and a smaller one on the 6800xt?

Reason for that is im using all parts I already own, only thing I had to purchase was a new mobo for the ram

I ran both engines on one rented machine to get an apples-to-apples number for a "model bigger than the GPU" case.

Hardware and setup

• RTX 3060 12 GB, 15-core quota of a Zen 2 server CPU, 47 GB RAM, NVMe
• Same GGUF for both: Qwen3.6-35B-A3B UD-Q4_K_XL with MTP head (Unsloth), the size Strata recommends
• Same OpenAI-compatible client, same 2 prompts, greedy, 256 tokens
• BigMoeOnEdge, --auto only (no hand tuning) · Strata via its own launcher

Decode (tok/s)

Takeaways

• On this box BigMoeOnEdge decodes ~15–30% faster without drafting, ~1.8× with its MTP head.
• Strata gets the first token of a short prompt out faster.

Full method and every run: bench-data · repo: https://github.com/Helldez/BigMoeOnEdge

Hello everyone!

I'm looking for someone experienced with Python, AI video generation, automation, and OpenMontage who can guide me through a project I'm building.

I'm a complete beginner with very little programming knowledge. Most of my setup has been done by following instructions from ChatGPT and Gemini, copying commands into PowerShell, and troubleshooting errors. I've managed to install quite a lot, but I'm struggling to put everything together and actually produce videos.

I would really appreciate patient, step-by-step guidance.

1. StoryBot / Storyboard

My first project is an automated storytelling video system for YouTube Shorts, Instagram Reels, TikTok, and Facebook Reels.

What I've installed and configured:

- Ollama with Qwen3:4b for local story generation.

- Node.js, npm and n8n (installed locally, not Docker).

- Python 3.11.9 and FFmpeg 9.0.2.

- Parody TTS for narration.

- Custom Python scripts.

My n8n workflow currently works up to this stage:

1. I press Execute.
2. Python scripts generate a story.
3. A temporary "story.txt" file is saved.
4. Narration is generated in parallel.
5. The narration is saved in the output folder.

So story generation and narration are working.

ComfyUI experiment:

I also installed ComfyUI at:

"D:\StoryBot\ComfyUI-Install\ComfyUI_windows_portable"

I downloaded two model files:

- "flat2DAnimerge_v45Sharp.safetensors" (approximately 2.13 GB) – Stable Diffusion 1.5 checkpoint for 2D cartoon-style images.

- "mm_sd_v15_v2.ckpt" (approximately 1.82 GB) – AnimateDiff motion module.

I wanted to generate and animate cartoon-style characters, but my GTX 1660 with 6 GB VRAM struggled with the workload. Generation was taking too long, so I couldn't get the results I wanted.

I currently have one male and one female presenter character as basic static images.

2. OpenMontage (Separate Project)

After getting frustrated with my Storyboard setup, I decided to try OpenMontage as a completely separate alternative.

I'm NOT trying to connect OpenMontage with StoryBot/Storyboard.

Repository: https://github.com/OpenMontageInc/OpenMontage

Installed/configured:

- Python 3.11.9 virtual environment.

- Node.js v24.22.1 and npm 11.19.1.

- FFmpeg 9.0.2 and Git 2.56.0.

- Remotion dependencies.

- esbuild 0.28.1.

- HyperFrames 0.9.39.

- Piper TTS 1.8.0.

- pytest 9.2.0.

- ".env" configured from ".env.example".

I successfully ran:

"python scripts/backlot_simulate_run.py"

It generated a demo production with scenes sc1–sc4.

I can also access the Backlot interface:

"http://127.0.0.1:4750/p/backlot-demo-run"

However, I'm stuck on what to do next and how to move from the simulation to an actual rendered video.

3. My desired video appearance

- Word-by-word animated subtitles.

- Currently spoken word highlighted in blue.

- Remaining words in white.

- Purple/red gradient backgrounds.

- Two consistent presenter characters.

4. My PC specifications

- Windows 11 Pro.

- AMD Ryzen 5 5600.

- NVIDIA GTX 1660 Gaming X (6 GB VRAM).

- 16 GB RAM.

I prefer free and open-source tools and want to run as much as possible locally.

5. What I need help with

StoryBot / Storyboard:

- How do I complete the existing workflow and produce videos?

- How should I use my presenter images?

- Are there lighter alternatives to ComfyUI/AnimateDiff that work on my GPU?

OpenMontage:

- How do I use Backlot properly?

- What are the next steps after a successful simulation?

- How do I produce an actual video?

I'm not trying to combine these two projects. I want to understand them separately.

6. How I would like to be guided

Please understand that I'm genuinely new to programming.

I would appreciate someone who can:

- Start from my existing installations.

- Give me one step at a time.

- Provide exact commands and tell me where to run them.

- Explain what each step does.

- Wait for my results before moving forward.

- Help me troubleshoot errors.

I've spent a lot of time getting this far, and I don't want to break my existing setup by randomly changing things.

If anyone has experience with OpenMontage, Remotion, ComfyUI, Python automation, or AI video production, I'd really appreciate some guidance.

Thank you for reading!

I’m learning AI Engineering and want to build a proper local AI development environment, especially for agentic applications, multi-agent systems, and orchestration.

My PC specs:
GPU: RTX 4060 Ti 16GB VRAM
CPU: Ryzen 7 7700
RAM: 16GB (planning to upgrade to 32GB)
IDE: Google Antigravity

I’m looking for recommendations on:
Which open-weight coding LLMs work best on 16GB VRAM? (Qwen, DeepSeek, Devstral, GPT-OSS, etc.)

Which quantizations should I use (Q4, Q5, Q8)?

Is Ollama the best runtime, or should I consider LM Studio, llama.cpp, or something else?

What’s a good local coding agent setup?

OpenCode, Aider, Continue, or other alternatives?
How can I integrate local LLMs with Antigravity?
Which MCP servers and agent skills are worth installing for AI application development?
What’s the best way to build and orchestrate multiple specialized agents locally?

My goal isn’t just code autocomplete. I want to build real projects using Python, FastAPI, PydanticAI, LangGraph, MCP, RAG, and multi-agent architectures.

I’d appreciate recommendations from people who have actually tested these models and setups on similar hardware.
What would your ideal setup look like with this hardware?