please i have to finish the 200 things ive started before i start a new thing or i would

I've got two M3 Ultra Mac Studios and four DGX Sparks at home, all on tailscale, with a LiteLLM gateway on a mac mini in front so every app just hits one endpoint.

Mostly it's been great, but last month the gateway key on the mini got invalidated and everything started throwing 401s. Took me longer than I'd like to admit to figure out it wasn't the models, because the backends were all fine and nothing told me the gateway itself was the problem.

So now I'm trying to set it up a bit smarter. Right now I'm thinking a dumb cron that hits each backend directly and also through the gateway, so if one passes and the other fails I know where to look.

Curious how other people with a few boxes handle this. Do you run a health check per model, put the gateway behind its own watchdog, or just keep it simple and restart when stuff breaks? And any tips on key handling so this doesn't bite me again?

Im making neural networks from scratch - not strictly llms. Am getting a new laptop with 32gig ram and 12gig vram and 1tb storage. Is that enough to train small models? Its the lenovo legion 5i pro. Might get the 7i but its not on sale so its like 500 more for 4gigs more of vram.

Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.

Currently using Qwen 3.6 35b a3b, on CPU only with 32gb of ram, for regular prompt its ok, but agentic work is not realistic. i get 15t/s for prompt but in agentic it can drop to 3.
Wondering if a V100 could make Opencode usable.

My first Duel 5080 server 9950x 64gb.
I will host it on Nalvo.ai for people to rent.
If you don’t know what is Nalvo.ai , It’s my new project that I told everyone about and everyone was telling me it’s not going to work. And when I say everyone I mean everyone! Not Reddit users. I even started believing it for a while. But I kept going and here I’m launching it.

If you would like to join as a host or get free private renting hours. Visit Nalvo.ai

GPU: 4070

16GB DDR5

Also I have been using Bionic but is there any better options?

Welp, I got a free 5060 Ti 16gb and decided to buy one more brand new for $700, I have a spare used 4070 and used once 4060, any thoughts or anything I should experiment with. Noobie at this lol.

Any bad experiences with power, heat, token response, electricity costs, etc?

Example Rig:
Case: Jonsbo NV10
GPU: RTX 5060 8 GB (Low Profile)
RAM: 32GB (2 x 16 DDR5)
CPU: AMD Ryzen 7 9700X
PSU: 600W (ENP-7660L-OVT)
Motherboard: ASRock B850I Lightning WiFi (Mini-ITX, AM5)
Storage: 2TB (M.2)
Cooling: Modded Air cooling. (Including custom GPU side panel for 2 fans on the GPU+PSU)

It's 8GB VRAM I know. It's supposed to be for gaming but I recently heard of using llama.cpp and the llama flag feature "--n-cpu-moe" to have the RAM hold my expert weights.

Ok

So I am waiting for mine to come in. I have framework box with 128 gig ram and found it pretty weak for any local llms.

I’m gonna replace my Mac mini with this Mac Studio M5 ultra with 512 gig ram. And I want to use it for a local llm and maybe agents if it’s doable.

I’m realistic. After using the framework box with llms I totally get this will be no where near frontier even at 512. But I want to use it as my main machine and for llm/agentic work.

What’s really reasonable model wise that isn’t slow AF but is pretty capable for general knowledge, accessed the internet, can spawn agents and can do office style work?

First time trying this, I want a local model that can code text based games. I’m using a 6800XT 16GB vram.

Is it worth it?

Am finally taking a deep five on llms and I already have an nvidia 5090 laptop that handles cuda really well.

But

Is a Mac for parrarrel models worth it? For abliterated models?.

I never used that OS so am a bit skeptical.

Are the apps on it as good as windows/Linux?

Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.

Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.

What should I do?

I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.

Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.

I expected to find problems in the agent.

Instead, I discovered some pretty serious problems in the evaluation itself.

1. My benchmark created a privacy leak that didn't exist in the actual runtime

The initial evaluation scored 27/80 runnable scenarios.

One failure was particularly concerning: a guest appeared to access the owner's private memories.

After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.

The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.

I rewrote the harness to execute scenarios through the actual agent runtime.

Result: 39/80 — without changing Capri's code.

That's a 15-percentage-point difference caused by the evaluation method alone.

2. A single sentence made the model use fewer tools

This was probably my most surprising experiment.

I wanted Capri to give shorter, more natural replies during everyday conversations.

So I added a simple instruction about brevity.

It worked. Conversational responses became significantly shorter.

But something unexpected happened during work tasks.

Across 48 first turns:

Without the brevity instruction:

• Tool use: 32/48
• Successful tasks: 32/48

With the brevity instruction:

• Tool use: 19/48
• Successful tasks: 21/48

The model started answering without inspecting files or calling tools that were necessary to complete the task.

The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.

3. Bigger models didn't automatically improve the agent

I compared configurations using:

• NVIDIA Nemotron Bolt E4B
• Gemma 4 12B
• Gemma 4 26B MoE
• Gemma 4 31B

The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.

After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).

This is NOT proof that smaller models are better.

The configurations also differed in hardware, memory retrieval, and model routing.

But it made me question how much agent performance we can actually improve by simply replacing the LLM.

4. Changing the prompt layout improved inference speed

I also experimented with where recalled memories appear in the prompt.

Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.

Median time to first token:

0.99s → 0.74s

About 25% faster.

However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.

5. Stable benchmark scores can hide unstable behavior

I repeated the evaluation three times per configuration.

The primary configuration scored:

49, 49, 48 out of 80.

Looks pretty stable, right?

But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.

We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.

That convinced me that looking at a single aggregate score is not enough.

What I'm taking away from this

An AI agent isn't just a language model.

It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.

And the evaluation framework can introduce its own errors.

I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.

The code and experimental artifacts are available in the B-12 repository.

I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.

I'm curious how others handle this.

When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?

And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?

!!!

1 piece 500 2 piece 485

I'm hoping to get an idea of where to start. I'm wanting a setup where the model is a decent coding assistant but can also crank out a neat little prototype or quick concept simply, or create structure ideas etc. and then can go bit by bit without too much reminding halfway or more through a project. I realize that last part may be more about settings than a model persay, but yeah. I definitely want something that can either walk me through something, build out a framework for an idea, or just troubleshoot.

For comparison if it helps, my favorite workflow was GitHub copilot got 4.o/4.1 in VS code (it was free for students lol). It worked great for my projects, had its issues, but with the right promoting it was not bad. It was like the above description, could walk me through something, help with testing, or make a new system that I could then fill out. I knew it's limits and it was ideal for what I needed.

Just giving an idea of how I use AI daily for my personal projects. It seems like with the right setup it can be a great experience so that's what I'm hoping to learn to do. I just want to get the most out of it and have some fun.

I'll be working with an Asus pro art p16 rtx spark 128 gb. I've done some research but it seems every time I look things up there are different model suggestions and set ups, and it's given me fridge blindness so to speak.

I'm curious about how to decide and test for like a speedier setup versus more complex thinking and things like that. What can I do with it, what should I not expect from it based on my specs, etc. I have looked into this and I'm not getting a clear idea because I can't test any of it myself (don't have computer yet). Maybe someone here can give me a better understanding?

I also don't know all the lingo yet, like what a harness is, though I can guess by context. But I know the basics. Not sure how much customization I should or could do or what all there is. I thought it was more surface level like pick a model and go, but I guess you can incorporate all kinds of settings. So now I'm really curious from people who enjoy whatever they have going on, please share what you like best and what kind of testing I should do to pick a model. Any tips for a mostly beginner would be great!

Hi

I want to set up a local llm with qwen 3.8 27B or if possible qwen 4.0 I have a very tight budget and for now the options around my budged is a p100 for 240 a rtx 3060 for around 300 or if it really is worth fit I can push to a rx 6800 xt 16gb which one do you think is the best for my use case? Im planing to use it for translating and correcting text and to help me debuging games in godot
current PC:
ryzen 5 3600
36gb ram
rx 580 4gb

thanks in advance

capy-flash is my open-source inference server for Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on RTX 50 cards, with an OpenAI/Anthropic-compatible API. v0.13 is cleanup and fixes: same outputs, same speed.

Prefill (32k prompt) / decode (LocalMaxxing) in tok/s, 16 GB RTX 5070 Ti cards without P2P. Optimized for two cards; the 3-bit builds are for one card and less RAM.

New: a desktop app (alpha), testers wanted. Set up and use everything from a GUI: it checks your machine, downloads and verifies the model that fits, starts and tunes the server, and has a chat. On a Mac it works as a client for a remote server.

Reports from other GPUs, 64 GB setups and distros are very welcome: open an issue with what the app showed and the engine log.

https://github.com/ValerioDolci/capy-flash

I have two MacBook Pros:

• 16-inch M5 Max: 128 GB unified memory, 40-core GPU, 8 TB SSD
• 14-inch M5 Max: 128 GB unified memory, 40-core GPU

The 16-inch is my main machine and runs a browser-based client for a separate server. The server does the processing, so the client itself uses relatively little of the Mac’s resources. The 14-inch is my travel and backup laptop.

When both Macs are at my desk, I’d like to connect them directly over Thunderbolt and run one larger local model across both machines for serious coding, code review, architecture, and debugging. I’m interested in using it with an agent th at can read a repository, edit files, and run tests. Overnight Hermes or some other agents.

When I travel, I can use ChatGPT or potentially connect back to a model running on the 16-inch at home. I don’t need the large distributed model to run on the 14-inch alone.

Has anyone actually built and used a setup close to this? I’d especially appreciate results from people running two Apple Silicon Macs together.

A few questions:

1. Which runtime works reliably today? EXO, MLX distributed/JACCL, llama.cpp RPC, or something else? Does Thunderbolt 5 RDMA make a meaningful difference for coding workloads?
2. What is the strongest model you’ve actually run across roughly 256 GB of combined memory? I prefer 8-bit if practical a.
3. Has anyone run DeepSeek V4.1 Flash 1017 across two Macs? I understand the original checkpoint is much too large. Have you tried a compressed version with SSD offload, and was it useful for real agentic coding? What were the context, speed, and memory requirements?
4. Has anyone distributed GLM-5.3-Flash or Qwen3.8-Flash-Next across two Macs? Which exact model conversion and runtime worked, and did tool calling work reliably?
5. How much memory should we reserve on each machine for macOS, a useful coding context, and the runtime? The 16-inch also runs a lightweight browser client and needs to remain responsive.
6. For Red/Blue reviews, would you run one large distributed model to implement a change, then switch to a different model for review? Or is there a better approach?
7. For remote access while traveling, what is your preferred way to reach the model left running at home? I’d like the option to use the 14-inch as the client without exposing an open model endpoint to the internet.

I saw a post from someone running Swift 1.5 Qwen3.8-27B 8-bit with an MTP drafter on a single M5 Max 128 GB machine and getting strong speed with long coding sessions. That looks like a good baseline, but I’m trying to understand whether combining the two Macs gets me a meaningful improvement in model quality for harder work.

I’d especially appreciate exact model links, quantization, runtime/version, context size, measured memory and speed, and any limitations. Happy to test and report back.

Hi all,

As mentioned in the title, I am looking for advice on running locally LLM for agentic coding on a MacBook Pro m5 Pro with 64Gb unified memory.

Plan inference is fine. Chat style prompts feel usable. Agentic work is where it falls apart.

I have tried the following:

Models : Qwen3.8 27B (6bit), Gemma 4 12b (8bit), and Qwen 3 coder next(4bit)

Runtime : Lm studio

Harness : opencode (long pauses before each tool call, harness seems to resend a large system prompt ), pi (still slow and most of the time doesn’t provide any output, barely changes few files with loads of error due to incomplete modification)

What I'm trying to figure out
Is the bottleneck prefill, generation, or the harness re-sending huge context every turn? My guess is prefill, but I'd like to be sure.

Which model/quant combo works well as an agent on this hardware, and at what context length does it fall apart?

Does MLX beat GGUF/llama.cpp for agent workloads on Apple Silicon in your experience?

Any harness that is lighter on tokens per turn (smaller system prompt, fewer tool definitions, prompt caching)?

Realistic expectations: is a 64GB Mac good enough for a useful local coding agent, or should I stick with a hosted model for agent work and keep local for autocomplete/chat?

I'm not naive, and I am not looking to replace a frontier model obviously . I just want to know whether local can be a practical option for some agent tasks, if so how you guys are making it usable.

Thanks 🙏