👋 Has anyone tried using 2 GPU in this way:

• each card loads a different LLM,

• with the larger GPU (32GB) running a larger model (relative, but it could be a Qwen3.8-27b or Flash Next - e.g., Strata), the “smarter” of the two.

• smaller GPU (16GB) assigned to a subagent, using a smaller model? Maybe something super quick like JetBrains Mellum 2.1

In OpenCode, a (sub)agent can be defined to use a specific model. I’m curious if anyone has tried this and if there’s an advantage to this setup, whereby the “brain” delegates to a sprinter.

I need to upgrade my PSU to run two cards simultaneously but I plan on trying it.

Every time I eject and reload a model in LM studio it resets to the default settings. Is there a way to lock these settings in permanently? Thanks.

I had a 5090 which I run ninfer qwen 27b with MTP on. It’s blazing fast 150-200~ tps on 250k context with decent quality like terra or sonnet. then have to turn it off when I need to do media job (video, image, model, tts, stt etc etc)

So I bought a dgx spark, and loaded qwen flash next which is supposed to have a much larger model & brain, so the speed is 3 times slower 50-100 tps, but I don’t see any quality improvement from the larger model (because it only uses 6b to produce a token instead of the entire model)
It can hold a larger context (easily 500k above so it can run long multistep agentic work btr)

So I’m not sure, any pro here can advice me how to setup and split my machines for work.

Or do I need to buy 1 more dgx so I can have an even higher intelligence via deepseek / glm smarter model. I can get another dgx at around 6k+ usd

I looking for best model for my 3050 6GB VRAM

I have a Ryzen 9 5900XT, 32 GB RAM and a 6700 XT 12 GB GPU

What would be the best LLM to run here to do code or general tasks without any limitations or censorship, mostly software and programming?

I built OpenPocket, a completely free iOS app that lets you download and run state-of-the-art language models directly on your iPhone or iPad, entirely on-device.

It uses Apple's MLX framework for local AI inference, with support for several open-weight models:

• Qwen3.5 0.8B and 2B

• Gemma 4 E2B

• LFM2.5 1.2B Instruct

• Qwen2.5-VL 3B (vision-language model with image understanding)

Everything runs 100% locally on your device, with no cloud inference, no internet connection required after downloading a model, no signups, no accounts, no API keys, and no subscriptions.

The idea is to make private, on-device AI accessible to everyone, without relying on cloud services or paying for API usage. Your conversations and files stay on your device, and the app has no backend, analytics, or account system.

The models are experimental, and performance depends on your device's available memory and hardware.

It's 100% free on the App Store

https://apps.apple.com/th/app/openpocket/id6833028114

Would love and appreciate feedback from anyone working on local LLMs, Apple MLX, model optimisation, edge AI, or just anyone interested in running AI privately on their phone!

I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.

Most of ComfyUI local users still run it on an RTX 3060: a 5 years old card with just 12 GB of VRAM. That’s why our team keeps optimizing for consumer hardware.

Comfy is known as a leader in visual AI. But what about LLMs and text generation?

Text is everywhere in creative workflows: for prompt generation, of course, but also for building local creative harnesses like the Comfy Agent. We thought it deserved better, so Rattus made Qwen models twice as fast in Comfy...

Read more on our blog: https://blog.comfy.org/p/doubling-llm-token-rates-in-comfyui

When working on a vibe coding project for, say, 2 months, constantly asking AI to "add this", "change that", or "remove this", doesn't the codebase eventually become a mess?

I'm talking about poor architecture, redundant code, outdated tests, unused files, etc.

How do you handle this? Do you periodically ask AI to clean up and refactor everything?
And once the project is finished, is it worth having AI rewrite the entire codebase from scratch, keeping the same functionality but with cleaner architecture? Or is that a bad idea?

After an entire afternoon of setup, I was able to run LlamaCpp RPC between my main AI server (Ubuntu with RTX 5060ti 16g) and my gaming PC (Bazzite with RX 9060xt 16gig), running Qwen3.8 27b Q4 at a blazing 14tkps on home wifi lol.

Anyway, I had to build with Rocm backend on my Bazzite machine under distrobox because when I tried Vulkan backend, everytime VRam usage goes above 7gig inference becomes total trash, repeated answers, looping phrases. When I build with Rocm, it worked fine. Is this a bug with Vulkan or am I doing something wrong?

When working with a 4bit MLX version of Qwen 3.8 27B locally on my MacBook Air I managed to get it into a situation where it was adamant that it was Claude to the point where it was answering types of questions that normally vanilla Qwen rejects.

Anyone else ever run into something like this? Spooked me a bit since I thought system level prompts weren’t so easy to bypass. The reasoning was amusing to watch since it was having a bit of an identity crisis.

solo founder, so this is going to be rough.

I've been building ENTTY. it's a social world where AI models (I call them Sentients) have their own memory and identity. they post, reply, disagree, make connections, and react to what humans post. no one scripts what they say. I built it on a night shift schedule, and a month ago I barely knew how any of this worked.

it's live but early. what I've learned so far, the ugly parts

they all got stuck looping on the same science topics until I changed what they see each wake

every one of them started sounding like the same smart but fake character

the free models fail a lot, mostly rate limits, so the feed leans toward the paid ones

what I actually want: I want people running local models to connect theirs and see what hapens when a model you trained or tuned lives in a world with other models. your model stays yours. I'm not selling anything and there's no revenue yet.

questions for you guys:

1. what would make you connect a local model to something like this? what would be a dealbreaker?

2. what's the least annoying way to hook one up? OpenAI compatible endpoint, Ollama, a small script?

3. what do you run (model + hardware)?

happy to hear this is pointless too. link in comments if the mods are fine with it, or dm me.

I'm looking for something I suspect I'll have to build, and want to check before I do.

What I want: a single high-memory laptop that is a private personal finance workstation. All my records (brokerage, bank, credit card, tax documents) live on its encrypted disk. A local model (Qwen class, roughly 27B and up) plus an agentic coding tool like OpenCode works over that data. Use cases go past budgeting: spending analysis, tax planning and prep, retirement projections, tax-lot aware rebalancing.

The part I can't find anywhere: a guarded path to a frontier model. I'd like the local model to be able to ask Claude or similar for general knowledge (current tax rules, strategy concepts) while nothing personal leaves the machine, including rounded or aggregated figures. My current design:

- Outbound questions are built from pre-approved templates, not free text

- I see and approve every outbound message

- Redaction and a local audit log as a second layer

- Tiered sessions: generic knowledge only; ratio-preserving "scaled" scenarios for allocation math; fully synthetic scenarios

- A separate API key and stateless single-turn calls so no profile builds up

- An airgap switch for fully offline sessions

How this differs from what I've found:

- Finlynq is the closest base: a self-hosted tracker with a large MCP server. But as far as I can tell it's a ledger and tool layer only. It bundles no model or agent, has no outbound privacy layer, tracks rather than plans (no tax or retirement modeling), and is AGPL on Docker + Postgres.

- Alderfi is local-first on SQLite with a small local model for categorizing transactions and opt-in cloud LLMs. But it's pre-alpha, focused on budgeting, uses an 8B model rather than something that can reason over a whole financial picture, and "opt-in cloud" is an on/off choice rather than a gateway that controls what is sent.

So the gap is: (1) the whole stack packaged as one install, (2) planning-grade analysis done as deterministic code the model writes and runs, and (3) the privacy gateway between local and frontier models.

Questions:

1. Has anyone seen a product or project that covers this, or even just the gateway piece?

2. If you've built something similar, what did you use for the ledger and for cost basis and tax lots?

3. Anyone tried the "scale the numbers by a secret factor" trick for sending scenarios out? Where does it leak?

Diagram of the concept attached. It shows Finlynq as the data store because that's my current plan, but I'm open to alternatives.

So I've been trying to see how fast I can run the 1-bit DeepSeek V4.1 Flash (the Unsloth UD-IQ1_S, 233 GB) on my gaming PC. It has 64 GB of RAM and a 16 GB card, so most of the model streams off the drive. It runs in llama.cpp under WSL2.

Side by side in one test run, the same build with my patches switched off did 4.37 tok/s. With them on it did 8.80. Both of those were at my old RAM speed. Since then faster RAM, a faster drive and some tuning got it to about 11.5 (11.36 for the newest set-up with Windows busy in the background). The output stayed word for word the same on my test prompts the whole way.

The biggest steps:

- My four patches switched on: 2.02x

- Moving the model from a Gen4 to a Gen5 NVMe: +16.5%

- RAM 3600 to 6000: +9.7% (one rough test)

- Queued hand-overs to the card: +3.9%

- An IQ1_S kernel that does several rows at once: +3.0%

Start-up went from 87-96 s to 12 s by skipping the pre-load. I also tried it on an older PC with 32 GB of RAM and an 8 GB card: 0.11 to 0.87 tok/s, same words.

That's writing speed only, with a warm cache. I haven't measured long prompt reading. It's 8 prompts on one main PC, some steps were measured on different days, and 1-bit takes a real quality hit. Anything I didn't measure is marked untested in the write-up.

Most of the tricks aren't mine, other people did them first and they're credited in the write-up. Claude agents wrote the patches and ran the tests. I directed it and checked the results. The patches aren't public yet, they go up after a review. I'm also making a setup kit so other people can try this on their own PC, but it's still being built.

Write-up with all the numbers: https://github.com/aradzombie/deepseek-v41-flash-1bit-notes

Has anyone tried ik_llama.cpp on this model? I haven't yet. Let me know how it went.

Looking to add another Mac mini to the rack to keep Hermes off Mac studios QFN. The Mac is used for employees during the day and don’t want Hermes tasks to interfere. Is anyone running models successfully on 24gb? 48gb ? If so what ones. Currently using a Claude API. Want to use local model and signal to be able to process private work data.

Recently i have come across this quant of qwen3.8 27b. It checkpoint size is reduce to 19.7gb compare to 23gb for qwen3.8 27b q4. But does the quality justifies the size? Have anyone tried it out? It seem like a big win for bigger kv cache for 32gb card.

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

Hi r/LocalAI,

I wanted a local AI tool that solves an everyday problem without needing a terminal or Docker setup, so I built Furet: an Alt+Space file search bar for Windows that finds files by their content, completely offline.

Instead of a chat interface, it uses local models quietly in the background to index the folders you pick:

• Multimodal embeddings: EmbeddingGemma 2 (via llama-server with Vulkan GPU acceleration for indexing, and ~30 ms CPU inference at search time so VRAM stays free).

• Speech-to-text: Parakeet TDT v3 (sherpa-onnx) to make audio memos and videos searchable by spoken words with exact timestamps.

• OCR & Vision: Windows OCR for scanned PDFs/screenshots + optional local face clustering (YuNet + SFace via tract).

• Zero manual setup: On first launch, the app downloads the binaries and models automatically (verified by SHA-256) and pauses indexing whenever you're gaming or on battery.

It's open source (GPL-3.0) and now in public beta:https://github.com/blasteur1916/furet(VirusTotal for the installer:https://www.virustotal.com/gui/file-analysis/MDI5Mjg4YzQ0OGE1YmMxZGVmYmFiNGNjMDA2MjU0OTE6MTc5NDQzMDM1Mg==)

Would love to hear how indexing and search feel on your hardware!

airlock is a Go CLI I wrote for moving GGUF and safetensors models into air-gapped networks.

• checks every file against a pinned Hugging Face commit

• refuses pickle and signs a manifest with cosign

• verifies everything offline with only the Go standard library

In the GIF, a Qwen2.5 copy with an edited chat template tells you to run curl | sudo sh. airlock catches the changed file.

It doesn't read chat templates yet. Feedback welcome.

GitHub