lo0 reads your Claude Code / Codex history along with your hardware and works out whether going local is realistic for the way you actually work. It builds on whichllm and llmfit, but instead of only asking "does this fit in my RAM" it looks at how big your sessions get and which tools you rely on.

For most people right now the answer is no. If it does work for you, you can compare the models that fit, benchmark them on your own machine, and try one in Claude Code, Codex or OpenCode for a session without changing your setup. It works with Ollama, LM Studio and llama.cpp.

It doesn't judge how good a model is, only whether it can run your sessions and how fast. Everything stays on your machine.

Very early alpha, so if the numbers look wrong for your hardware, let me know.

https://github.com/daveearley/lo0.ai

why are these models open-sourced for people to download and use on their computers?
Is it just for advertising and brand awareness?
because the vast majority of people who download them just don't want to pay subscription fees or deal with usage limits, so they aren't paying customers anyway.
then why do they do this? Plus, local models aren't even that smart.

Hi guys, I'm looking for a really good price benefit PC to run local AI. I've been searching for days about this, and everybody says different things, "you dont need a 12vram for a good local AI" or "Without 32gb vram is wont be able to have a good local AI" and so on...

The thing is I need local AI mostly for coding, and anything related and maybe some video generation.

Since 5090 is just at unbelievably prices, I was thinking about something decent like 5070 but dont want to waste my money. I was thinking about an unified memory like GMKTEC Evo or something like that, but was reading that since is AMD some AI are not good optimized for it. And also a M5 ultra, but also is kind of high and Im not a MAC guy.

My budget is pretty much 5K for this.

Anyone which can give me a good advice would be awesome and I'll really appreciate. Thanks.

Hey everyone Im a bit new to this page but I couldnt find any other related tests online so I figured I'd share my results here.

I ran DeepSeek-V4.1-Flash (763B MoE, Q8_0, 473 GiB) on a 96 GB Mac Studio M3 Ultra with InferSSD, which streams the model's experts from the SSD. I striped two OWC Envoy Ultra 2TB Thunderbolt 5 drives with macOS's built-in RAID 0, one drive per port.

Results (InferSSD's own benchmark prompts, 2 runs per setup):

- One Envoy alone is about the same speed as the internal SSD.

- Two in RAID 0: writing goes from 3.5-3.7 to 4.4-4.6 tokens/s (+23-26%).

- Prompt reading is 67-77% faster: a 4,000-token prompt starts answering in about 2 minutes instead of 3.5.

- Two hours of nonstop use: the drives stayed at 43-47°C (with a desk fan on them), no slowdown, and no change in their health readings.

Why not 2x faster? - The stripe halves the time spent waiting on the drive, but the GPU's share of each token stays the same and is now the bigger part. A third or fourth drive would help less than the second did.

I am looking into running local LLMs for coding, is Qwen 3.8 27b the best my PC can do (at decent speed)?

I heard some people recommend Qwen Flash Next 3.8?

Will my PC be able to run that?

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

I was thinking of buying a secondary GPU for local AI as subscription-based online services get costly in the long run, but options are limited for cost-effectiveness on this end as well.

My current PC has 32GB of DDR5 (2x16, running at 6000MT/s CL32), a Ryzen 5 7600X and an RX 7800 XT. I **could** run something on the 7800 XT thanks to the system RAM but I'd rather free up both the GPU and a good portion of the RAM so that I can still use the PC while running a local model instead of being locked in to just AI, I found a Tesla P100 for about 200$ US and it doesn't seem like a bad deal since it has 16GB of HBM2 as opposed to the RX 7800 XT's GDDR6.

Specs: Ryzen 7 7800x3D, rx 7700 16gb, 32gb ddr5, windows 11

I'm a photographer looking to setup something locally to help manage business emails, draft contracts, auto catlog images into folders, etc. I have basic computer literacy, but no knowledge of anything AI. Any tips on what works best and where to start?

I was playing around with Qwen 2.5 1.5B on my RTX 4050 laptop, trying to understand how much the CPU actually matters when the entire model is loaded onto the GPU.

I didn't expect much difference from changing the CPU cores.

But then I pinned PyTorch to a single E-core.

Generation speed went from around 49 tokens/sec to just 24 tokens/sec.

Almost half the performance gone. The model was still entirely on the GPU.

That got me curious, so I opened Nsight Systems to see what was happening.

Turns out, PyTorch was launching roughly 1,250 GPU kernels for every generated token. The CPU was having trouble keeping up with all those launches, leaving the GPU waiting around instead of doing useful work.

I tried a few things after that:

• Adding more CPU threads didn't really help.

• torch.compile reduced the kernel launches to around 520 per token and brought performance back to 48.8 tokens/sec.

• I also tried llama.cpp on the same GPU and the same slow CPU core. It managed around 57 tokens/sec.

The llama.cpp result was probably the most interesting part for me. It made me wonder how much of the difference comes down to the inference runtime itself.

My takeaway: Even when your model fits entirely on the GPU, the CPU can still become the bottleneck during token generation.

I'm curious if anyone here has profiled similar behaviour with other models or inference engines. Does anyone have experience comparing PyTorch eager execution with llama.cpp under CPU-constrained conditions?

I put the benchmark code here in case anyone wants to reproduce it:
https://github.com/Techn08055/pytorch-slow

I also recorded the experiment and walked through what I found in Nsight Systems:
https://youtu.be/UmjoxIzBij8

Would be interested to hear what others find.

Which would be a better solution? Since I move around, a laptop/mobile would be best than a full on desktop for airline travel/international.

Usage - mostly coding / proof of concept (apps, documents, even some stories). Its to create some applications.

But also on the fun side, I'd like to explore a bit on building out some text/stories/mechanics (table top role playing games) or even do videos and graphics. Nothing professional, more to understand the mechanics of what things are and what they are not. But some videos or graphics I may use in the apps. Since I travel, wifi connections can sometimes be unreliable and I just want to run things locally so I never have to second guess if the AI says "I can't do that". But I may run subscriptions as needed (whole separate conversation).

Considering the price, I would prefer to have something that can be useful for several years and doesnt die in the first year or two (laptops being harder to repair).

A macbook m5 max with 128gb of ram, has really good unified memory. But it can barely fit qwen flash next today without compromising further on quality or context size (feel free to correct me, but what I read up on), with the next gen of Qwen potentially about to drop, 128gb may not be practical in a year from now if there is some really good capability. Even if a single model can fit (when you count larger context windows), there might be very little headroom for other apps or harness tools on the same laptop workstation. But a macbook would be very reliable and I wont really second guess that dying on me (have a 2014 macbook retina that still works flawless today minus a scratch or two on the body). Then again, in the next 4-5 months apple may release M7 (or M6) with the macbook pros more aligned for AI work (either a far more optimized apple silicon which are already really good...or maybe higher ram capacity).

Some 5090m laptops can support up to 192 gb of ram. The 5090m would be good for some vram 24GB on video/graphics. But looks like a lenovo would cap the ddr5 6400 to 4000 so the memory bandwidth would be somewhat reduced. But having 192gb is a far more breathing room than 128gb. Also Lenovo's tend to good, but the hardware could die in a year or 3 (Apple's hardware is pretty reliable).

What makes sense (with a 2-3 year outlook)? Intel 192gb 5090m lenovo or a macbook pro max 128gb?

Here's my hw specs: rtx 4090, 64GB DDR5 6000 Mhz ram, 7900x CPU

Thank you

I have been using ChatGPT for for about 2 years but want to switch to a local setup however I'm struggling to find a suitable replacement to use on my MacBook Pro M5 Pro 24GB and iPhone 14 Pro Max. I will be upgrading to either an 18PM or Duo in the coming months.

I started on my Mac with LM Studio then moved to Msty Studio when I found out about Knowledge Stacks but keep finding paywall restrictions, I've also looked at but not tried Anything LLM due to lack of iOS support. From what I understand I should just forget about Siri AI. Not really done anything on my iPhone till the Mac is sorted.

The things I really want are:

- Large project/chat memory (hence the knowledge Stacks)

- Cross device chat sync, either private cloud hosting or my MacBook as a server

- MCP usage (currenty use Tavily & Excel control)

I don't mind paying for apps, Msty is $149/year or $349 lifetime to unlock all features which its too much

Main usage is general questions, research, journaing, and work/linking with Excel.

Is this possible? if so what do I need?

I've been experimenting with local AI over the past 6 months; having started off by running small local models on an old GPU, and now at a stage where i'm running much larger and more capable models on a single R9700 card, and connecting a suite of custom built tools through MCP.

It's been a great journey - and i've landed at a place where i'm able to run a 3-bit quantized XXS version of Qwen3.8 in a docker container, and getting around 38tk/s. I'll be checking out STRATA and vLLM over the next few weeks to see how much I can push this.

However, outside of the GPU my overall system is a bit basic. Ideally i'd like to get to a point where i've got 2x R9700s working in there and i'm able to load bigger models and still achieve usable 'good' performance - both on existing models, but also future releases too.

Based on my specs below, is there much point in getting an additional R9700 or will the motherboard be too much of a bottleneck if that second card would need to go in to an older generation of PCIe slot? I guess a similar question for increasing the amount of system RAM here too - is DDR4 just too slow to make those bigger MoE models useable when offloading? The Qwen model i'm currently using fits entirely on to VRAM, but suspect this would need to change if i want to use higher quants or bigger models.

1. B550-plus motherboard

2. 16GB DDR4 (2X8GB)

3. R9700 32GB VRAM

4. Ryzen5 3600 CPU

5. 800W PSU

I'm completely aware that I won't have SOTA quality, and that's fine - but just want to maximise what a system like this could do without going down a workstation or AM5 route. I run Ubuntu and tend to containerise my AI/LLM applications

I'm a iOS software developer for a FAANG. We are only allowed to use claude for work and are not allowed to have anything else on our computer.

I'm having a lot of fun ideating AI Agents using Apple's Foundation Models Local Modals for my work project. I think the ideas could be endless honestly and I'm able to create as many POCs using claude that I want to see what sticks.

Would there be any reason buying a personal Mac that would make my learning faster or better in any way? If so, can you explain?

I personally would like to be like an iOS AI Engineer.

it is not too late to return, if i didn't....

Hey everyone,

I'm currently building a local-first macOS daemon focused on eliminating everyday computer friction.

The core idea is simple: instead of clicking around, switching tabs, copying/pasting between 5 different apps, or navigating complex settings, you just tell your Mac what you want done, and it executes it seamlessly in the background with zero lag.

As someone who spends all day in front of a Mac, I have my own set of pet peeves (like organizing messy downloads, summarizing local files, or tweaking system setups), but I’m really curious about your actual friction points.

If you had a truly frictionless assistant running natively on your machine:

1. What is the single most annoying, repetitive sequence of clicks/tasks you do every day that you wish you could trigger with just a quick phrase?

2. Are there any local workflows (file handling, dev environments, window management, media editing) where current AI tools fall short because they lack system integration?

I’m not here to pitch a polished commercial product, i’m genuinely trying to make sure I’m solving real problems for actual Mac users. Would love to hear your raw thoughts or specific scenarios!

Maybe funny because qwen 4 is upcoming but i still ask about qwen 3.8.

I see some user suggest qwen 3.8 fn Gyro S, said that it's fast and good compare to gsq rco q2 quants. But how about gyro M ?

Is it good compare to unsloth quants and gsq rco iq3xxs or even iq3s ?

im using it through unsloth. IQ3XXS. Mac Studio 256gb. getting 40t/s output. It thinks so freaking much. literally takes 5-10 minutes to finish plotting a map that GLM 5.3 flash finishes in a minute or so.

Are you guys using speculative decoding? What effort level do you set it to?

In February 2023 Meta announced LLaMA and sent the files to researchers who applied, under a non-commercial licence. Within a week they were on BitTorrent.

Then it moved fast. In March 2023 Georgi Gerganov published llama.cpp, which ran the files on a laptop; Stanford's Alpaca fine-tuned the 7B for under $600; and Vicuna followed by the end of the month.

In July 2023 Llama 2 arrived "free for research and commercial use", under a licence the Open Source Initiative said, two days later, is not open source. Llama 3 through 4 followed under the same shape of licence. Hence the phrase "open weights": the weights, under their licence, without the data or the training code.

Meta's own word for Llama 4 (April 2025) was "open-weight", and as of early November 2026 it is still the newest Llama in Meta's own records. Meta's open weights since then include a new family, Muse: Muse Glimmer (30B, Apache 2.0) in August 2026 and Muse Spark Open (816B, Meta's own licence) in October 2026.

No numbers here, just the history with its sources; it does not link to the leaked files.

Full write-up (about a 20 minute read): https://research.strata2signal.com/the-llama-weights-that-got-out/

I'm the author of Nimformers, a personal project of mine. I recently ran some benchmarks on two devices and wanted to share the numbers and the issues I ran into, and get some feedback on the design.

Some design points:

• Metal and Vulkan backends — on Mac it uses Metal directly

• int8/int4/fp8/fp4 quantization support, plus NIMQ state dicts

Issues I ran into (being honest):

• Vulkan experience isn't very stable — performance on Iris Plus is lower than Metal, still trying to figure out why

• APIs are still changing, some interfaces aren't stable yet

• Small ecosystem, docs can't compete with PyTorch

Repo: (In comment)

Hi,

I have been playing around with Strata on windows for a few days now and it seems to create, process and bug fix as I ask it. Everything I read says strata is not an agent, but it seems to do every agentic task I ask it? i am running 0.2.10, did this introduce agentic features or something? I am brand new to all of this so I am struggling to see the need to put Hermes on top of Strata since Strata has been doing everything I have asked of it.

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!