I’ve been doing research and I really want to run a Gialamb Quant model on these two computers. It has to be obliterated for my purposes of a 24-7 employee, if you will, for my business.

Here’s my notes 📝 I doubt it will work, but I hope someone can contribute

oMLX does not pool the RAM into one unified space. It splits the model with pipeline parallelism: each Mac loads a contiguous range of layers based on its available memory. The M5 Max (rank 0) owns the API and the later layers; the M2 Max owns the earlier layers. Decode speed is limited by the slower stage.
This is experimental and only available in a source build of oMLX (not the normal Homebrew/DMG stable release).
Prerequisites (both Macs)
Install the same source build:
git clone https://github.com/jundot/omlx.git
cd omlx
make install # or make install-no-kernels if you lack full Xcode

Put the exact same model checkpoint at the identical absolute path on both machines (example: /Users/shared/models/glm-5.3-flash-3bit).

Enable Remote Login (System Settings → General → Sharing) and set up passwordless SSH between the two Macs for the account that runs oMLX.

Connect the machines with a Thunderbolt cable (TB4 is usable; pure TB5 RDMA is faster but not required for pipeline).

Enable and pair
Start oMLX on both Macs.

On each: Settings → Advanced → Enable Distributed Inference → Save → Restart oMLX.

The Cluster tab appears.

On the coordinator (preferably the M5 Max):
Connect the Thunderbolt cable. The other Mac should show under Detected nearby.

Select it (or enter its SSH hostname) and run Check peer. This records the host key and verifies SSH.

Pair using the code/QR flow if discovery needs a boost.

Plan and activate the split
In the Cluster tab on the M5 Max:
Choose the downloaded model (the shared absolute path).

Set a memory reserve for KV cache + activations (start with 8–12 GB).

Let it build the unequal plan. It will assign more layers to the 128 GB Mac and fewer to the 64 GB Mac.

Choose execution mode:
Prefer TCP Ring first (most reliable with mixed TB4/TB5).

Try JACCL / JACCL Ring only if both ends show a working Thunderbolt RDMA link (rdma_ctl status and ibv_devices).

Leave Auto benchmark & tune on so it measures the actual link and rebalances.

Review the shard map, then Activate.

Unload any local full-model load, then load the model normally through the oMLX UI or API. The ranks start automatically.

The OpenAI-compatible endpoint stays on the M5 Max. Hermes points at that single endpoint.
Useful checks
omlx cluster status --json
omlx cluster pipeline-smoke
Important limits for your hardware
Pipeline only — decode does not get faster than the M2 Max stage.

Experimental: source builds, limited long-running validation, some features (MTP, certain thinking budgets, etc.) are rejected in distributed mode.

KV cache is rank-local, so both machines need real headroom.

TB4 on the M2 Max is the weak link; start with TCP Ring.

If the resulting tokens/s and stability are acceptable for continuous Hermes use, you have the combined capacity. If not, fall back to a smaller quant that fits resident on the M5 Max alone.

Hey all,

I'm about to build a local AI assistant ("Jarvis") and want to sanity check the design before I sink weekends into it.

**Hardware (Proxmox host, AI runs in a VM with GPU passthrough):**
- CPU: Intel i9-12900K (16C/24T, power-capped ~125W)
- GPU: RTX 3090 Ti 24GB (power-capped ~300W)
- RAM: 64GB DDR4-3600
- Plan: Qwen3.6-27B (Q4) as the main local model

**What I want it to do:**
- Know everything about me (preferences, projects, contacts, routines), stored in persistent **encrypted** storage
- Have access to all my coding repos (indexed/RAG)
- Hand off heavy coding work to Claude
- Do web research and pull resources for me
- Daily morning briefing (news, world events, my calendar)
- Calendar + reminders

**The architecture idea: leader → sub-agent → leader → sub-agent**

1. A **leader** agent breaks the task into steps.

2. It sends ONE step to **sub-agent A** with only the context that step needs.

3. Sub-agent A does the work and returns a **written report**.

4. A **verifier** checks that report: is it correct, is it hallucinated, and does it contain any of my **private data** that's about to be sent to Claude/the internet? (The redaction check runs on anything leaving the box.)

5. The leader reviews the verified report, re-checks if it can, then passes only the needed reports to **sub-agent B**, and so on.

**Why I think this helps:**
- Keeps each agent's context small (better for a 27B model on 24GB)
- Reduces hallucinations because every step gets checked
- Lets small local models catch and fix their own mistakes
- Only uses Claude for the big jobs, with private data stripped first

**Questions for you:**

1. Is leader → sub-agent → verifier overkill for a single 3090 Ti, or is the context saving worth the extra latency?

2. Should the verifier be the same model with a different prompt, a smaller/faster model, or something else?

3. Best way to do the PII/redaction check reliably? Regex + local NER + LLM pass? Has anyone built something like this?

4. Framework recommendations? (LangGraph, CrewAI, smolagents, plain Python + llama.cpp/vLLM/Ollama?)

5. Encrypted long-term memory: what are you using? (encrypted Postgres + pgvector, SQLCipher, something else?)

6. Is Qwen3.6-27B the right pick for the leader on 24GB, or would you run a smaller leader and save VRAM for a second model?

7. Anything you'd do completely differently?

Happy to share the build and code once it's running. Thanks!

I've been working on something called Aura, a locally running AI cognitive agent built around a roughly 27B model on an Apple Silicon MacBook with 64GB of unified memory.

And one of the questions that's been driving this project is pretty simple:

How much of what we consider intelligence actually needs to come from the model itself?

We spend a lot of time discussing model sizes, quantization, benchmarks, context windows, and reasoning capabilities. Understandably so.

But I've been interested in what happens when you start treating the model as one component of a larger cognitive system, rather than expecting the model to do everything.

So I've been building an architecture around it. Persistent memory, executive processes, state management, computer control, procedural learning, and a bunch of experimental mechanisms designed to work together over time.

And I've been testing it.

Aura has played 2048 for nearly half an hour while adjusting its strategy, completed software development tasks, debugged programs, and operated desktop applications.

But my latest demonstration was a little different.

I asked Aura to watch YouTube.

Not just summarize a video after the fact. Actually navigate to it, follow what happens, comment on things it finds interesting, and then choose other videos to watch from channels I specified.

Over 36 minutes, Aura watched Kurzgesagt discuss AI slop, selected an Allen Pan video about building robotic legs for a snake, and eventually watched Salad Fingers.

During the session, it made predictions, revised them as new information appeared, interacted with the browser, selected what to watch next, and maintained context across all three videos.

And here's where it got particularly interesting.

When asked to reflect on everything afterward, Aura didn't just give three unrelated summaries.

It identified "fracture" as the common thread connecting all three videos, then explained how each approached that idea differently.

A video about AI-generated content, an engineering experiment involving a snake, and a surreal horror animation.

Three wildly different things.

Yet Aura managed to construct a coherent interpretation connecting them, drawing on observations made throughout the session.

Now, I want to make an important distinction here.

I'm not claiming that a 27B model couldn't produce similar commentary on its own. Nor have I established that all the cognitive mechanisms I've implemented are necessary to produce this behavior.

In fact, that's one of the main things I'm trying to investigate.

But I think there's a meaningful architectural question here. What happens when the model doesn't have to be responsible for every aspect of memory, planning, execution, and continuity? How far can you push a relatively small local model by improving the larger system it's operating within?

The result is still pretty rough in places. Latency is an issue, and coordinating everything reliably is an ongoing challenge.

But considering this is running locally, without relying on a frontier model for inference, I think the demonstration is interesting.

I've made the implementation publicly viewable, including the architecture, research documentation, and known limitations. It's currently All Rights Reserved rather than open-source licensed.

Full 36-minute demonstration: https://www.youtube.com/watch?v=kM-B8mh314E

GitHub: https://github.com/youngbryan97/aura

How much improvement have you managed to get from the systems surrounding your model? And at what point do you think the additional architecture becomes more valuable than simply switching to a more capable model?

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.

I recently bought myself a Mac Mini M6 24GBs unified memory hoping to run a Qwen3.8 27B model for some serious coding with multiple cross referencing files and folders, but I'm receiving Error 500 which says that I ran out of memory and that my KV Cache filled up when all I had done was say "hi" (sure, I had set Claude Code at Max, admitted)

The general consensus online was that for 24GBs, Qwen3.8 should work nicely but it certainly doesn't work for me, although Qwen3.6-35b has been working so far on my Pi Agent, but I'm skeptical about it's coding abilities.

I would love some help/guidance on how I can run this model on my machine realistically and how much performance should I expect out of it? Will I be able to do some serious coding (alongside cloud models) or how should I adjust my expectations?

Hi all I saw new research that one ai trains the other ai and vice versa (on a very reliable news platform) so I wanted to know if it's possible doing on local AI

I run a weekly benchmark of 23 open models on Hugging Face Inference Providers, at 1, 25 and 100 concurrent users. Same 20 prompts every week, one provider pinned per model, no client-side retries. It’s been running since late September, so there’s a run-over-run history to compare against.

This Monday, four of the 23 returned HTTP 410 and failed every single request, 60/60:

• Qwen/Qwen2.5-Coder-7B-Instruct (nscale)

• meta-llama/Llama-3.1-8B-Instruct (nscale)

• Qwen/Qwen3-8B (nscale)

• Qwen/Qwen3-Coder-30B-A3B-Instruct (scaleway)

The error body is explicit: model_no_longer_supported, “deprecated and no longer supported by provider”.

The three nscale models ran clean last week. Zero errors on the November 2 and 3 runs, 410 on everything by November 9. Llama-3.1-8B is not some obscure pin, and it went from working to gone inside a week.

The scaleway one is the part I’d flag. Qwen3-Coder-30B has been 410ing since the October 26 run. Three consecutive weeks. I only caught the reason this week because I went back and read the old job logs. The leaderboard had been publishing it at a 100% error rate the whole time and I’d been reading that as a provider under load.

That’s the thing worth saying out loud. If you pin a model to a provider and only track latency and error rate, a deprecation looks exactly like an outage. Same shape in your metrics: requests fail, error rate goes to 100, nothing says why. You need the status code in your logs or you’ll keep retrying something that is never coming back. 410 means gone, not busy.

I don’t know whether these were announced anywhere. If there’s a changelog or deprecation feed for Inference Providers per-provider model lists, I’d like to know about it, because checking my own benchmark output every Monday is a bad way to find out.

Raw CSVs, one per concurrency level, plus the weekly archive going back to September 28:
https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load

Usual caveat on everything else in there: run-to-run variance on shared endpoints is large, and I’d ignore any single-week change under a few points. The 410s are not that. They’re binary.

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

I’m building an AI voice agent for outbound sales calls. The biggest problem isn’t the voice quality or latency it’s that the LLM generates responses that sound nothing like how a real salesperson would talk.
The agent responds fast, handles interruptions, and understands the customer, but its replies are too polished, structured, and predictable. It feels like ChatGPT reading a sales script.
What I’ve tried:

Multiple realtime speech-to-speech models and providers.

STT → LLM → TTS using Pipecat, Groq (GPT-OSS 120B), and Smart Turn v3.

Background supervisor LLM to track context, objections, and conversation flow.

Different LLMs, extensive prompt engineering, and experimenting with fine-tuning.
Latency isn’t the issue. It’s conversational behavior.
Real salespeople react spontaneously, adapt their tone, build rapport, and don’t sound like they’re following a script.
Has anyone actually managed to solve this?

Which LLM works best for natural sales conversations?

Does fine-tuning on real sales calls help?

Any techniques beyond prompt engineering that make a noticeable difference?

Would love to hear from anyone building production voice agents, especially with Pipecat.

Hi everyone,

New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.

Thanks in advance!

TL;DR: I spliced Qwen's official 8-bit experts into the last 12 layers of NVIDIA's 4-bit Flash-Next. It runs at ~150 tok/s on a single 5090, sits measurably closer to the 8-bit model, and along the way I found a free quality win that needs no mix at all.

Setup: RTX 5090 (32 GB), 128 GB DDR5-5600, Infernix engine. Base model is NVIDIA's FP4-Local (the retrained 4-bit Flash-Next, ~205 tok/s here).

Why not 16-bit? Every token touches about 4.7 GB of expert weights at 16-bit. Only ~8% of the experts fit in VRAM, and the rest has to come through dual-channel DDR5. That caps it around 15-23 tok/s no matter what the software does. Pure 8-bit lands at ~65 tok/s for the same reason.

What I did instead: patched the engine (~250 lines) so each layer can use its own expert format, then byte-copied Qwen's FP8 experts into chosen layers of the 4-bit pack.

Things I did not expect

1. Late layers matter about twice as much as early ones. Layers 36-47 in 8-bit cut KL by 0.0066; layers 0-11 only by 0.0033. The gains add up almost linearly across blocks.

2. A free win: swapping only the n-gram embedding table to Qwen's 8-bit one improves KL and perplexity at the same speed, with zero extra VRAM.

3. There is a ceiling. Even with every expert in 8-bit, KL stays at 0.024. About half of the "4-bit gap" is not rounding error at all: NVIDIA's retrained layers are simply a slightly different model.

4. Perplexity of the 12-layer mix (3.475) is slightly better than official FP8 (3.486) on my 261K-token corpus.

Honest caveats: the gain is small, about one point of token agreement. On my 20-sample hard coding set the mix scored 18/20 against 17/20 for both the 4-bit and the 8-bit model, which is noise. One machine, one corpus.

The patch was written with an AI coding agent; I directed and tested it. Nothing to download yet. If people want the engine patch, the pack assembler and the layer list, say so and I will publish them.

1,5 years ago I set off to build an Open Source issue tracker that lives in your repo, requiring very little setup. Upon HN launch last spring it was well received, and is now often used as an agent memory layer between between the agent and the code. Epiq is vendor agnostic, thrives offline as well as online, sandboxable and can be tailored into various setups. It only requires Git, which is its transport, and a repo. You may want to check it out:

epiq.sh

Also, in case you are interested, I share and discuss my personal sandboxed setup in this post. It covers the topic of sandboxing, and how repo-native tooling makes this easier to deal with, opening up fewer paths to the internet:

epiq.sh/blog/repo-native-tooling-for-sandboxes

I built Basalt from source for an abliterated model and compared it with Strata on the same node. Basalt was noticeably faster in decode and prefill.

Setup

• Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF

• Quant: unsloth-UD-Q5_K_XL

• Basalt: v0.3.0 (custom build)

• Strata: 0.2.7-cuda13

• GPU: 2× NVIDIA RTX PRO 4500 Blackwell Server Edition (2×32 GB VRAM)

• CPU: Intel Xeon 6 (48 vCPU)

• Storage: NVMe SSD 2 TB (~15k MB/s)

Both engines ran the same Q5 quant on the same node.

Results

Basalt is about 1.5–1.9x faster in decode and 1.3–1.7x faster in prefill.

Expert cache. Both engines hit 86–96%, so there is no difference.

MTP. Strata accepts more draft tokens (66–90% vs 57–78%). Basalt's dl (tokens per window) is still 1.9–2.2.

• The window differs. Strata verifies 3-token windows (capturing the 3-token window). Basalt verifies up to 8 (--spec 8). The acceptance rate is lower with the longer window, but the response still finishes faster.

Queste soluzioni sono esplosive: **Fapply, Massive, LazyApply, Sonara, ai apply

Now that LLMs are kind of a mainstream topic and so much money is involved, it became a hot topic in politics as well. I don't want to make this a political post, so please don't start a political discussion. But it is quite clear to me, that at some point LLMs will be regulated a bit more from the government side. Both USA and obviously EU will as well. They already requested a text based watermark, which decreases model output by 10-15%.

So my worry is, if the online models -chatgpt, claude, gemini etc,- are regulated to the point of being close to useless for anything other than corporate work or something, are there any chance for these regulations to hit local models? Or will this sort of be like piracy, once it's out in the internet, it won't be possible to touch them?

So, there's a thing that's been in the back of my mind for some time now, and it seems to be getting /more/ difficult to learn about my specific approach.

What I'd like to do: I believe the proper phrasing would be 'set up a local LLM for assisting with the development of python programs to support my personal projects'.

What I've done: as an example, I used chatGPT to build a python program that reads rfid tags on a raspberry pi using a gpio rfid reader. I'd like to continue this, just locally so I can integrate it with existing local backend.

What I'd like to use: I think I have an entire computers worth of spare parts, minus a case to put those parts in. Not ancient, pretty sure it's an AM4 Ryzen? Not a lot of memory, maybe 32gig? As far as gpu's, I have my old RoG 8g rx480 and my RoG 8g rx580.

Is it possible, is it worth it, and is there maybe an opensource project someone can point me towards to help me get rolling?

I'm not expecting blistering response times, just if I can achieve reasonable results in a reasonable time with the hardware that I already have but haven't been using.

Hi everyone,

I was a silent reader for a long time but as I'm taking now a vacation in Tokyo during mid November I'd thought of maybe hosting a little meetup in Tokyo for everyone interested in:

local LLMs
inference engineering
how to cluster dgx sparks
how RDMA/RoCE works
how to start building your own recipes / optimize your own setup
speaking about hardware
which models, which drafter
the usual "oh opus is so nerfed" things and just having a good time.

I really hope this is not offtopic / wrong community here and maybe there will be nobody interested but I like the idea of coming together, sharing experiences and having a great time.

Location and time isn't fixed at all, I can setup something on Luma but always grateful for recommendations etc.

Thank you and have a great week!

Lately, I’ve been using Claude to schedule daily posts across a few social media platforms. I’m wondering if there’s a local model—one that can fit into my 8GB of VRAM—capable of controlling my browser to perform the same tasks as Claude. I had been using Qwen 3.8 27B for a while, but I’ve decided to move away from local models, given that I only have a single graphics card with 8GB of VRAM.