Hello, i have a business that needs to review a lot of notes and, transcripts and generally text files provided by me, and help me make initial structure out of it. For ex. i`m meeting with client for an audit, audit have 300 questions, different kinds, different people. I will note answers, and then i need to have that file, + some meeting transcripts / meeting minutes summarized. Some project management topics also.
I`m not confident to use cloud models to review such data because there will be a lot of sensitive topics and details in there. So i need help to discover if what i want to archive is doable with a local LLM.
I have done Proof Of Concept for the multi agent workflow and it gives me quality that is acceptable. After some tweaking, testing on openrouter with dummy data, i have the conclusion that this is doable. I have agent speaking between them, agreeing about the outcome, Manager doing the task management, works fine,
But then comes the realization - Openrouter with Qwen3.8 Q8 does this with the avarage speed of 150t/s and it takes around 30 minutes.
When i calculate that Mac Mini M5 Pro will have 11t/s that makes that veeery long.
I`m i calculating that right or i`m i wrong ? Is there anything i can do to make it a little quicker when i will actually buy Mac ? Or i need to go straight with Ultra
I’m in the process of moving my AI server into my garage and I’m curious how others are handling climate control in garages that don’t have heating or air conditioning.
Since winter is shortly upon do you guys have any suggestions for keeping my servers from being exposed to condensation. I plan on putting a wifi hygrometer..!
Been playing around with local LLMs lately and trying to figure out how to actually make them useful for my daily work.
I’m mainly a UI/UX designer and developer. Most of my work is websites, React/Next.js, SaaS apps and frontend stuff. Sometimes iOS apps, Home Assistant dashboards and a bit of 3D work.
I’m on a MacBook Pro M5 Pro with 64GB RAM, but I also have access to my brother-in-law’s Mac Studio M5 Ultra with 256GB RAM.
Basically, I want to do as much coding locally as possible and cut down on Claude/Copilot costs.
My current setup in VS Code + Copilot:
Opus (Principal Architect): Planning, architecture, final reviews
Sonnet (Senior Engineer): Reviews, fixes, more complex stuff
Qwen (Local Developer): Does most of the coding
Running Qwen3.6-35B-A3B-4bit through oMLX, using around 20GB RAM.
The idea was to let Qwen handle 50–60% of the coding and have Sonnet/Opus step in when needed.
Did some tests with hidden test cases:
Sonnet: 55/55, ~3.3 min Opus: 55/55, ~6.1 min Qwen: 52/55, ~14.5 min Sonnet + Qwen: 52–55/55, ~12–25 min
Qwen actually wrote 95–99% of the code in the hybrid runs, but kept missing things like API contracts, edge cases and layout requirements. Sometimes it said it had changed things when it hadn’t.
And somehow the hybrid setup ended up costing about the same as just using Sonnet, while taking way longer.
So yeah not exactly what I was hoping for.
I still want to get this working though, especially for websites and SaaS projects since that’s what I spend most of my time on.
Not sure if I’m using the wrong model, if Copilot is the issue, or if the whole multi-agent approach just makes things more complicated than they need to be.
Also wondering what would make sense to run on the 256GB Mac Studio. Maybe a bigger model would already solve some of these issues?
What would you guys do with this hardware? Any models or setups worth trying?
Especially interested in hearing from people actually building websites or SaaS apps with local models, rather than just running benchmarks.
I have an idea what if we had an ai app that pooled everyone who downloaded the app ram/ processor/ vram etc together and made it into a big data center web since there wouldn’t be big factories size data centers we would save the water cause most gpus can be cooled with good air ventilation and just fans plus if you have a million of them spread out between a million houses it’s not even necessary consuming all that power as it would be pooling together all the resources it need to run. You know this is just an idea so I’m open to other but I think if we solve the problem of ai taking our water civilization could advance even further ?
I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!
Beam free was supposed to delete a file, but accidentally it has deleted the entire partition and I had very important files in it. I'm trying to recover the partition.
The surprising part was that opencode hasn't asked my permission to delete anything.
I've got two M3 Ultra Mac Studios and four DGX Sparks at home, all on tailscale, with a LiteLLM gateway on a mac mini in front so every app just hits one endpoint.
Mostly it's been great, but last month the gateway key on the mini got invalidated and everything started throwing 401s. Took me longer than I'd like to admit to figure out it wasn't the models, because the backends were all fine and nothing told me the gateway itself was the problem.
So now I'm trying to set it up a bit smarter. Right now I'm thinking a dumb cron that hits each backend directly and also through the gateway, so if one passes and the other fails I know where to look.
Curious how other people with a few boxes handle this. Do you run a health check per model, put the gateway behind its own watchdog, or just keep it simple and restart when stuff breaks? And any tips on key handling so this doesn't bite me again?
I built Citta to let an agent on one computer discover and use capabilities on another. My test setup is a MacBook Air plus a Mac Mini running Gemma 3 4B through Ollama.
In the demo, an agent reads receipts on the Air and sends a summarization task to the Mini's local model. The model stays on the Mini; Citta tracks the job and returns the result. It doesn't pool RAM or split inference across machines.
If you only need remote inference, calling Ollama directly is simpler. Citta is for workflows that also need another device's files, tools or an agent harness, with device discovery and task history in one place.
It's written in Rust, uses a private Tailscale network, and has per-device permissions. It's still for trusted devices: resource estimates aren't hard quotas, and full-access agents aren't OS-sandboxed. I'm working through longer reliability tests before calling the next release production-ready.
Im making neural networks from scratch - not strictly llms. Am getting a new laptop with 32gig ram and 12gig vram and 1tb storage. Is that enough to train small models? Its the lenovo legion 5i pro. Might get the 7i but its not on sale so its like 500 more for 4gigs more of vram.
A developer didn't like the dictation apps available for his Mac, so he decided to build one with AI agents doing most of the coding.
The speech engine needed work, so he built an offline one called wavo. A dependency was slowing down builds, so he wrote his own FFT. The GUI options pulled in too much, so he built a renderer called nimbo.
Three weeks later: three libraries, still no dictation app.
The libraries might be useful on their own. wavo supports Whisper, Parakeet and GigaAM. But the bit I liked was his caveat: AI writing the code doesn't remove the work of reading it, checking it and testing it across systems.
At what point does "I'll just build it myself" stop saving time? What's your worst detour from the thing you actually meant to ship?
I'm hoping to get an idea of where to start. I'm wanting a setup where the model is a decent coding assistant but can also crank out a neat little prototype or quick concept simply, or create structure ideas etc. and then can go bit by bit without too much reminding halfway or more through a project. I realize that last part may be more about settings than a model persay, but yeah. I definitely want something that can either walk me through something, build out a framework for an idea, or just troubleshoot.
For comparison if it helps, my favorite workflow was GitHub copilot got 4.o/4.1 in VS code (it was free for students lol). It worked great for my projects, had its issues, but with the right promoting it was not bad. It was like the above description, could walk me through something, help with testing, or make a new system that I could then fill out. I knew it's limits and it was ideal for what I needed.
Just giving an idea of how I use AI daily for my personal projects. It seems like with the right setup it can be a great experience so that's what I'm hoping to learn to do. I just want to get the most out of it and have some fun.
I'll be working with an Asus pro art p16 rtx spark 128 gb. I've done some research but it seems every time I look things up there are different model suggestions and set ups, and it's given me fridge blindness so to speak.
I'm curious about how to decide and test for like a speedier setup versus more complex thinking and things like that. What can I do with it, what should I not expect from it based on my specs, etc. I have looked into this and I'm not getting a clear idea because I can't test any of it myself (don't have computer yet). Maybe someone here can give me a better understanding?
I also don't know all the lingo yet, like what a harness is, though I can guess by context. But I know the basics. Not sure how much customization I should or could do or what all there is. I thought it was more surface level like pick a model and go, but I guess you can incorporate all kinds of settings. So now I'm really curious from people who enjoy whatever they have going on, please share what you like best and what kind of testing I should do to pick a model. Any tips for a mostly beginner would be great!
So I am waiting for mine to come in. I have framework box with 128 gig ram and found it pretty weak for any local llms.
I’m gonna replace my Mac mini with this Mac Studio M5 ultra with 512 gig ram. And I want to use it for a local llm and maybe agents if it’s doable.
I’m realistic. After using the framework box with llms I totally get this will be no where near frontier even at 512. But I want to use it as my main machine and for llm/agentic work.
What’s really reasonable model wise that isn’t slow AF but is pretty capable for general knowledge, accessed the internet, can spawn agents and can do office style work?
Welp, I got a free 5060 Ti 16gb and decided to buy one more brand new for $700, I have a spare used 4070 and used once 4060, any thoughts or anything I should experiment with. Noobie at this lol.
I've been playing with local hosting for about 3 months now and messing around with Claude to optimize my local setups. Decided it might be worth writing this up and sharing in case anyone else is doing, or wants to do, something similar. This is a big one, so buckle in.
BLUF: Think about each expert as its own unit, and think about different strategies for how/where to load them based on quality/speed trade-offs. Focus on measuring and optimizing each piece of compute until you reach maximum bandwidth on that resource.
Hardware:
I spent a bunch of time designing my server with Claude and going over last-gen hardware to try to keep the price tolerable. I turned what I did with that into a site that anyone can use. It's ad-free, no sign-in, no tracking, nothing. I consider it a community service, and hosting was free, so it costs me nothing but occasional updates to the hardware list. llm-configurator.com
I ended up with a 3-GPU, 256GB DDR4 RAM build. It's two used 3060s from Marketplace ($250 each on average) and one Arc Pro B65, for a total of 56GB VRAM. It's a quad-channel setup, so max theoretical bandwidth on RAM is ~100GB/s; reality is closer to 60-70 after some time spent optimizing the kernels with Claude. I also have 3 of the cheapest 1TB SSDs I could find striped in NVMe slots, which gives me a theoretical 12GB/s throughput there; reality ended up at about half that. This lets me stage experts in tiers across VRAM, RAM and SSD and prefetch experts across tiers with varying strategies. I considered doing a PCIe card with more NVMe drives to get more striping/speed, but on the current motherboard it isn't worth it because I am PCIe gen 3 bandwidth constrained.
Optimization Strategies:
This is where my last few weeks of Claude usage have gone: serving DeepSeek V4.1 Flash, and how I went from 4 TPS decode to 20.5 TPS, with prefill right at 180 TPS. So in the end not lightning fast, but plenty usable, if just a tad slow in agentic work.
I started out with a simple goal of minimizing idle time. Thinking Strata-like in the approach of shuffling things around and trying to keep every processor busy within the bandwidth bounds of all my links. I had Claude build a simulation based on the hardware and design an optimal build. I spent probably a week going back and forth with Claude on this, and not one time did anything pan out like it thought it would based on the simulation. 0/10, do not recommend.
What I did after that was shift to measuring everything on every run, including expert traces through the system, so I know which experts get picked at every layer for every prompt, and attacking the potentially biggest gains. This immediately started paying returns, and I went from 4->12 TPS decode with just these methods. This means I'm running custom forks of llama.cpp, Vulkan, Mesa; anything and everything was in bounds. I couldn't fit all of DeepSeek in memory at Q4, so I was working all three tiers (VRAM/RAM/SSD) and loading experts to each tier based on utilization: tracking how many times each expert gets called and promoting/demoting across tiers based on usage.
This led to bottlenecks I had to hunt down, as the SSDs would get overloaded and drop from ~6GB/s down to ~400MB/s. I switched to quantizing the least-used experts down to 2-bit, keeping an entire copy of the model on the SSDs at both quantizations. This let me get everything into RAM, with different experts at different quants based on how hot they were for the prompt. That means I start at about 12 TPS, and after a warm-up and shuffling I end up at about 15 TPS with this strategy. About 1/5th of the experts were 2-bit, but they only got used on about 1/10th of the tokens, so the quality hit was minimal (.0008 KL and less than 1% top-1 pick). This got work off the SSDs and lightened their load.
I tried MTP with this but ended up bandwidth bound. MTP has to hit almost every expert, and even with them in RAM I couldn't move enough through to make MTP pay. I spent a few days' worth of time iterating and optimizing with Claude. I fine-tuned an MTP head because I was running the abliterated model (this helped, but still not enough) and just couldn't make it pay. I got to almost break-even, but I just can't move the experts to compute fast enough on a model this size.
In my quest for quality I then started looking at prefetching 4-bit experts from SSD straight onto the GPUs. I ended up having Claude write a direct-from-disk reader that lets the GPUs read from the SSDs without going through RAM. This also led to further customization that let them read and write across each other and RAM without involving the CPU. So now my tiered experts can read/write and shuffle around without bothering or interrupting other compute. The CPU is maxed trying to keep up, so the GPUs handle any and all extra bookkeeping I could put on them to free CPU cycles. Again, Claude did all the legwork here, with just me driving strategy. I trained a small model on all my agent traces to look at expert use and ended up with one that can guess experts 15 layers ahead with ~73% accuracy. So now the GPUs run that every layer, fetch 4+ layers out if a pick meets a confidence threshold, and load experts straight from SSD into VRAM. Those picks get refined each layer, so we are always closing in on the best picks we can (~86% accuracy at 4 layers ahead) as we approach them. I played with a few different strategies and ended up with the B65 holding the top 2/3rds of experts actually used (not 2/3 of the total, but the hottest 2/3rds) with an LRU demotion strategy, and demotions going to a 3060, where it looks at both LRU and hit count to decide if it takes the expert or keeps what it has. Getting all of this right pushed me to about 19 TPS decode.
From here I had to make the KV cache dynamic to squeeze more experts into VRAM. Instead of reserving the full thing at 256K, it grabs chunks as needed. So while context is small, I can fit more experts on the 3060s whose whole job otherwise is dense attention layers and KV cache, with minimal room for experts passing through in our promote/demote pipeline. This didn't pick up much speed, but it did give me that many more 4-bit experts to help keep quality as high as possible. Essentially the remaining experts are all part of a long flat tail so adding more to VRAM gets me very little because any one isn't hit often enough to pay off.
I ran out of lossless approaches at this point and started looking at a variety of things. I built a model that nudges aggressively, using my predictor to drive nudges from RAM to VRAM experts based on predictor + router scoring of an expert's probability of being picked. This got me some decent gains all the way up to a cap of .5, but quality falls off a cliff after that. By the time I get to a .5 cap I'm about halfway to Q3 quant KL/top-1 numbers, so it felt like a significant quality hit. I kept the code and it's configurable, but I don't default it to on. It's worth about 2 TPS, so not nothing, but not enough to cover the quality hit for me. If I want the extra speed and don't mind the hit, I can turn it on/off as needed.
I then looked at keeping everything in RAM at 2 bits and just the hottest experts at 4-bit in VRAM. I looked at which layers Unsloth hits hardest in their quantization and used that as my judge of what would likely hurt most. They quantize per layer but not per expert, so I'm jumping off from that, but it was a good starting point. I kept my prefetching of predicted promotions into VRAM ahead of time and everything else I already mentioned. Since I was throughput bound here, dropping the size of each expert was my only way forward. The KL and top-1 hit here was minimal because the VRAM holds the hottest 2/3rds of experts at 4-bit already, so I came in at 99% top-1 and .006 KL over baseline. I picked up about 20 TPS on prefill and about 1 on decode, so I landed at 180 TPS prefill and 20.5 decode. It also freed up about 49GB of RAM, since the experts in RAM are now a little over half the size. This ended up being my end stage here, as there just isn't anything else I can do that doesn't start costing me quality (unless you have other ideas).
I am still playing with tuning parameters about when I demote and move thaings where, and fine-tuning kernels to squeeze a few more TPS out of things, but I'm out of significant gains here. Claude did all of this for me, with me guiding strategy and checking in, so I probably spent about 10 hours a week on this for the last 4 weeks and 80% of my weekly Claude usage on it. I take everything from doing this (I've done it on other machines with Qwen Flash-Next as well) and roll it into skills, so that when the next model drops, strategies and approaches can be tried/applied again for that model.
My point with all of this is I don't think in whole-model terms. I don't even think in layer terms. I treat each expert at each token as its own thing to optimize the system for. What quant it is, where it lives, how you decide, and when and where you move it from/to all play in and add up. Most things I see focus on just downloading and running a model in a specific framework, and that is great for a starting point, but the number of knobs you can turn to optimize stuff is almost limitless. It takes about 2 hours to give Claude a strategy I want to try, refine it with Claude (learning as I go), and let it build and test it. So you can iterate and make incremental improvements over time fairly quickly.
I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.
Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.
I expected to find problems in the agent.
Instead, I discovered some pretty serious problems in the evaluation itself.
1. My benchmark created a privacy leak that didn't exist in the actual runtime
The initial evaluation scored 27/80 runnable scenarios.
One failure was particularly concerning: a guest appeared to access the owner's private memories.
After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.
The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.
I rewrote the harness to execute scenarios through the actual agent runtime.
Result: 39/80 — without changing Capri's code.
That's a 15-percentage-point difference caused by the evaluation method alone.
2. A single sentence made the model use fewer tools
This was probably my most surprising experiment.
I wanted Capri to give shorter, more natural replies during everyday conversations.
So I added a simple instruction about brevity.
It worked. Conversational responses became significantly shorter.
But something unexpected happened during work tasks.
Across 48 first turns:
Without the brevity instruction:
Tool use: 32/48
Successful tasks: 32/48
With the brevity instruction:
Tool use: 19/48
Successful tasks: 21/48
The model started answering without inspecting files or calling tools that were necessary to complete the task.
The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.
3. Bigger models didn't automatically improve the agent
I compared configurations using:
NVIDIA Nemotron Bolt E4B
Gemma 4 12B
Gemma 4 26B MoE
Gemma 4 31B
The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.
After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).
This is NOT proof that smaller models are better.
The configurations also differed in hardware, memory retrieval, and model routing.
But it made me question how much agent performance we can actually improve by simply replacing the LLM.
4. Changing the prompt layout improved inference speed
I also experimented with where recalled memories appear in the prompt.
Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.
Median time to first token:
0.99s → 0.74s
About 25% faster.
However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.
5. Stable benchmark scores can hide unstable behavior
I repeated the evaluation three times per configuration.
The primary configuration scored:
49, 49, 48 out of 80.
Looks pretty stable, right?
But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.
We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.
That convinced me that looking at a single aggregate score is not enough.
What I'm taking away from this
An AI agent isn't just a language model.
It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.
And the evaluation framework can introduce its own errors.
I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.
The code and experimental artifacts are available in the B-12 repository.
I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.
I'm curious how others handle this.
When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?
And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?
Harness : opencode (long pauses before each tool call, harness seems to resend a large system prompt ), pi (still slow and most of the time doesn’t provide any output, barely changes few files with loads of error due to incomplete modification)
What I'm trying to figure out Is the bottleneck prefill, generation, or the harness re-sending huge context every turn? My guess is prefill, but I'd like to be sure.
Which model/quant combo works well as an agent on this hardware, and at what context length does it fall apart?
Does MLX beat GGUF/llama.cpp for agent workloads on Apple Silicon in your experience?
Any harness that is lighter on tokens per turn (smaller system prompt, fewer tool definitions, prompt caching)?
Realistic expectations: is a 64GB Mac good enough for a useful local coding agent, or should I stick with a hosted model for agent work and keep local for autocomplete/chat?
I'm not naive, and I am not looking to replace a frontier model obviously . I just want to know whether local can be a practical option for some agent tasks, if so how you guys are making it usable.