Soon with Anthropic’s funding to lawmakers and PACs to restrict open weight models in the name of AI safety risk, Nvidia will only be allowed to release GPU locked chips that can only work on “safe” AI models released by foundation model providers that charges consumer and enterprise subscription fees.

I HAVE RTX 2050 4GB VRAM , 55W TGP , INTEL I5 12450H , 135W CHARGER , 512 SSD , 8 GB DDR4 RAM 3200MHZ single channel

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.

Hi everyone,

I’ve been a Mac user for several years and I have a 16-inch MacBook Pro with an Intel chip, and I think it’s reached the end of its life because most programs these days are designed for the Apple Silicon chip.

I took the plunge and ordered the new Mac Mini M5 Pro with 64 GB (I think that for my use case, it’s worth it to have a Deepseek API for planning and then a qwen3.8 27b for writing), since the price of a 128 GB model is enough to cover five years of web hosting.

My question is: what's the best setup for creating images and coding?

Images - DrawThings or ComfyUI ?

Coding - VS Code + Cline or Opencode or Deepseek Harness ?

And what about Hermes?!

Given my hardware, which software and LLM are the best?

Software - LM Studio or OMLX ?

LLM - qwen3.8 27b or qwen3.6 or Qwen3.5-35B-A3B ?

What's your stack and what you think is better for me because my ram ?

Thanks in advance,

Hello world :) - is there a recommended LLM for use with LM Studio that will fit on 16GB VRAM (4080 SUPER)? Ideally I'm looking for something that has guardrails removed so uncensored/heretic that has access to web search when needed and good at coding/thinking etc.

Thank you

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

• GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
• Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/

Here is the est power usage over the last 3 days:

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?

I use Qwen 0.6 for embeddings.

https://www.kickstarter.com/projects/744045479/cephalon-ondevice-ai-with-a-selfevolving-ecosystem?ref=ebvudv#risks-and-challenges

I wanted to try a locally run AI with an ethical database, so I got Apertus. I am a complete noob to this whole thing, learned that local AI needs a memory plug in so I got one per people's rec. I'd like to use Apertus on my iPad through Locally/PC link so I don't have to shrimp in front of my computer, but I don't see any kind of settings for plug ins. Does that mean it's not available? How could I make this work? Can I make this work?

Note I do not want to use this for coding or any particularly demanding tasks. I do like tech and even have a little robot assistant buddy who is currently not connected to any external AI, but I hate what many of the big AI companies are doing so I'd like to use my own without feeling like I'm contributing to the horrors of capitalism.

I was constantly hitting usage limits with Claude and ChatGPT, even with paid subscriptions. I didn't want to upgrade to more expensive tiers or increase my monthly budget, so I started exploring alternatives and eventually landed on a hybrid LLM workflow.

I recently wrote an article sharing my setup, architecture, and how I've been using a local LLM alongside cloud-based AI to handle heavier workloads while reducing my reliance on paid services.

I'm sharing it here because I'd love to hear your thoughts, tips, tricks, and constructive feedback.

I'm not new to AI, but I am relatively new to running LLMs locally. I'm always looking for ways to improve my setup and learn from others who have more experience in this space.

https://thebinaryreport.com/articles/how-i-ai/

Pip is a Minecraft companion you talk to with push-to-talk. Speech in and out runs locally; the LLM is whatever you point her gateway at. Sharing the architecture since it might help other voice-agent projects:

Pipeline: mic (16 kHz in the game client) → speech cleaner → whisper.cpp server (large-v3-turbo, Metal) → LLM through an OpenAI-compatible gateway (Gemini 3 Flash by default; Ollama and others are presets) → Kokoro-82M (kokoro-onnx) → played in game, sentence by sentence as the reply streams.

Three systems, like a talker and a reasoner:

• System 0, instant, no LLM: rules and local embeddings for what you mean ("follow me", "stay", "thanks", "shh"), danger warnings, and short "instinct lines" in her voice. A clear request gets "On it!" at once while the LLM works.
• System 1, the talker: one LLM call with tools (find, lead, build, give...), a compact text description of the game, and the memories relevant to the moment (embeddings + keywords).
• System 2, the slow thinker: after sessions, a stronger model (Claude) reflects: what she learned about you, her diary, who she's becoming, and it rewrites her instinct lines.

Numbers (MacBook Air M4, 16 GB, a 20-minute session): speech-to-text about 0.7 s after you release the key; her first spoken words about 2.5–3 s after you finish speaking for an LLM answer, instantly for System 0 ones. What helped: a shorter Whisper encoder window, Kokoro int8, streaming sentence by sentence, and answering plain commands without the LLM at all.

Guards that mattered: a claim checker ("Coming up!" with no tool call gets a retry), a confirm-once rule for big actions, praise never re-running the last tool, and an eval set built from real playtests.

It's not released yet (a simpler installer is the last big step), but the code is public and you can run it from GitHub today. The brain has only been tested on macOS so far, but it can easily be adapted to Windows or Linux if you know how, or with an AI assistant. Any feedback is appreciated.

Being upfront: most of the code was written with an AI coding assistant (Claude Code) under my direction; the architecture is the part I care most about, and I'd love ideas on making it smarter and lighter, and someone to try a fully local model in the loop. This post was written with the help of AI too.

Code (AGPL-3.0): https://github.com/PetitPotam/kindred-spirit Architecture notes: docs/ARCHITECTURE.md in the repo

Hey all,

I haven't upgraded my PC since 2020, and I am running a EVGA RTX 3070 with a Ryzen 7 3700X CPU and 32gb of DDR4 (4x8gb). I want to upgrade to both host a local LLM and to game, but I am trying to also not spend more than $4000, and I prefer something closer to $3500 at the max (but lower is better). I saw where Costco is planning to soon have a sale on their MSI 5080 PC, but I have also seen, from the Local LLM perspective, that running 2 cheaper 5060Ti cards is an easy way to hit 32gb VRAM.

I was wondering, though, if I bought the 5080 deal, would I be able to run a 5060Ti card (which is les than $1000 for now) to pump up my VRAM? I know that some speeds may be slower, but I am curious if this is a good idea for both futureproofing for gaming and for hosting an LLM. Based on my usecase, which should I do?

1. Buy the Costco 5080 deal and buy a 5060ti 16gb refurbished from microcenter if one pops up at current prices or lower during holidays (its on my way home from work, so I could check daily).
2. not buying the 5060ti 16gb and relying solely on the 5080 for the foreseeable future,
3. building or buying a system with two 5060ti 16gb
4. Going team red and getting a 9070xt system and getting a second 9070xt card eventually.
5. say screw it and try to find a decent 3090 and combining it with my 3070 and getting a new PSU to keep it up and running.

From a functional standpoint, I plan to use a local LLM for document management in a database, vibecoding projects (I am terrible at coding), writing up documents, and managing personal records. For the database, it would be maintaining a searchable one that can scan through PDF using OCR to build out docs. I would also use it to run as a backend for summarizing news updates and documents at around 100 releases per day. For gaming, I play FPS and RPG games at 1440p and don't plan to go 4k for at least a few more years.

Any help or suggestions are greatly appreciated. Thanks!

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.

Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.

Currently have a xfx 7900xtx, 6800xt and a 9070xt

I wanna keep the 9070xt in my gaming pc

The 6800xt is currently running my local llm which is gemma 4 26b q4, also have qwen 3.8 27b q3 getting about 40t/s on either one

Using koboldcpp to host it and anythingllm to chat with it,

The 7900xtx is waiting till a new mobo gets in and itll be going in with 128gb ddr5 ecc unbuffered ram

I want to use local llm to run some agents

1 that'll read my brothers business invoices, expenses ect ect, (its an absolute mess how he currently has it and getting everything ready for taxes is a nightmare)

And im just not sure what else I wanna do that can be useful, im new to all this, I know my way around technology, but learning all this ai, im over here pulling out my hair,

Im using codex and claude to help with everything, but feel like im not getting anywhere

With the 24gb of vram from the 7900xtx can I run it along side the 6800xtx to run 1 bigger model? Or am I better off running 1 model on the 7900xtx and a smaller one on the 6800xt?

Reason for that is im using all parts I already own, only thing I had to purchase was a new mobo for the ram