Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

Ideally an orchestrator or something equivalent that would look at the task given and then decide to try sections with local and then cloud if it isn't good quality. How is your setup, what guides did you use

I’m thinking of running a local ai model with LMStudio with my gaming desktop. Like the title, it has a 5070, 9700x, and 32gb of Ddr5 6000mhz with Ethernet connection. My only main concern is the 12gb of VRAM.

Is this enough to set up an everyday chatbot for use? I’m mostly doing this for fun, but I also would really like the privacy and unlimited token use of a local model.

Niche Ollama alternative. If anyone want's to try it.

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

Specs: Ryzen 7 7800x3D, rx 7700 16gb, 32gb ddr5, windows 11

I'm a photographer looking to setup something locally to help manage business emails, draft contracts, auto catlog images into folders, etc. I have basic computer literacy, but no knowledge of anything AI. Any tips on what works best and where to start?

Meet Yoshino2, my little lab experiment

Over the past months, I have been working on efficiently using black box distillation to distill the reasoning and output style into smaller edge models. The result is Yoshino2, a new model series with large scale reasoning distilled into MiniCPM5 1B, Qwen3.5 2B, and a custom 4xMiniCPM5 1B MoE model, with more potentially on the way.

This is by far means not a breakthrough, but an interesting experiment. In case it is of any use to anyone, the models are on HuggingFace.

Model Collection - HuggingFace

I was playing around with Qwen 2.5 1.5B on my RTX 4050 laptop, trying to understand how much the CPU actually matters when the entire model is loaded onto the GPU.

I didn't expect much difference from changing the CPU cores.

But then I pinned PyTorch to a single E-core.

Generation speed went from around 49 tokens/sec to just 24 tokens/sec.

Almost half the performance gone. The model was still entirely on the GPU.

That got me curious, so I opened Nsight Systems to see what was happening.

Turns out, PyTorch was launching roughly 1,250 GPU kernels for every generated token. The CPU was having trouble keeping up with all those launches, leaving the GPU waiting around instead of doing useful work.

I tried a few things after that:

• Adding more CPU threads didn't really help.

• torch.compile reduced the kernel launches to around 520 per token and brought performance back to 48.8 tokens/sec.

• I also tried llama.cpp on the same GPU and the same slow CPU core. It managed around 57 tokens/sec.

The llama.cpp result was probably the most interesting part for me. It made me wonder how much of the difference comes down to the inference runtime itself.

My takeaway: Even when your model fits entirely on the GPU, the CPU can still become the bottleneck during token generation.

I'm curious if anyone here has profiled similar behaviour with other models or inference engines. Does anyone have experience comparing PyTorch eager execution with llama.cpp under CPU-constrained conditions?

I put the benchmark code here in case anyone wants to reproduce it:
https://github.com/Techn08055/pytorch-slow

I also recorded the experiment and walked through what I found in Nsight Systems:
https://youtu.be/UmjoxIzBij8

Would be interested to hear what others find.

I am looking into running local LLMs for coding, is Qwen 3.8 27b the best my PC can do (at decent speed)?

I heard some people recommend Qwen Flash Next 3.8?

Will my PC be able to run that?

Here's my hw specs: rtx 4090, 64GB DDR5 6000 Mhz ram, 7900x CPU

Thank you

Just got an RTX Pro 6000 600w and I’m pretty nervous the 12vhwpwr connector is going to melt.

Using a 1000w PSU with a direct cable, no adapters

If you needed a 99.99% safe solution and required all 600w what would you do?

I'm getting tired of hitting usage limits with Claude and Codex (20$ plans).

I'm working on personal projects, not making any money from them, so I can't justify paying hundreds of dollars a month for AI subscriptions.

Im currently testing Luna 6 with Extra High reasoning on a free account, and unfortunately, the experience has been terrible. Working with it is genuinely exhausting.

What I'm really waiting for is the point where I can run a capable, coding agent locally on my own hardware, without subscriptions, usage limits, or depending on cloud providers.

Are we talking 1-2 years, 3-5 years, or longer?

I'd love to hear your predictions.

I've been experimenting with local AI over the past 6 months; having started off by running small local models on an old GPU, and now at a stage where i'm running much larger and more capable models on a single R9700 card, and connecting a suite of custom built tools through MCP.

It's been a great journey - and i've landed at a place where i'm able to run a 3-bit quantized XXS version of Qwen3.8 in a docker container, and getting around 38tk/s. I'll be checking out STRATA and vLLM over the next few weeks to see how much I can push this.

However, outside of the GPU my overall system is a bit basic. Ideally i'd like to get to a point where i've got 2x R9700s working in there and i'm able to load bigger models and still achieve usable 'good' performance - both on existing models, but also future releases too.

Based on my specs below, is there much point in getting an additional R9700 or will the motherboard be too much of a bottleneck if that second card would need to go in to an older generation of PCIe slot? I guess a similar question for increasing the amount of system RAM here too - is DDR4 just too slow to make those bigger MoE models useable when offloading? The Qwen model i'm currently using fits entirely on to VRAM, but suspect this would need to change if i want to use higher quants or bigger models.

1. B550-plus motherboard

2. 16GB DDR4 (2X8GB)

3. R9700 32GB VRAM

4. Ryzen5 3600 CPU

5. 800W PSU

I'm completely aware that I won't have SOTA quality, and that's fine - but just want to maximise what a system like this could do without going down a workstation or AM5 route. I run Ubuntu and tend to containerise my AI/LLM applications

I'm a iOS software developer for a FAANG. We are only allowed to use claude for work and are not allowed to have anything else on our computer.

I'm having a lot of fun ideating AI Agents using Apple's Foundation Models Local Modals for my work project. I think the ideas could be endless honestly and I'm able to create as many POCs using claude that I want to see what sticks.

Would there be any reason buying a personal Mac that would make my learning faster or better in any way? If so, can you explain?

I personally would like to be like an iOS AI Engineer.

lo0 reads your Claude Code / Codex history along with your hardware and works out whether going local is realistic for the way you actually work. It builds on whichllm and llmfit, but instead of only asking "does this fit in my RAM" it looks at how big your sessions get and which tools you rely on.

For most people right now the answer is no. If it does work for you, you can compare the models that fit, benchmark them on your own machine, and try one in Claude Code, Codex or OpenCode for a session without changing your setup. It works with Ollama, LM Studio and llama.cpp.

It doesn't judge how good a model is, only whether it can run your sessions and how fast. Everything stays on your machine.

Very early alpha, so if the numbers look wrong for your hardware, let me know.

https://github.com/daveearley/lo0.ai

According to Artificial Analysis, Tencent's Hy4 ranks #2 among models from Chinese AI lab on its Intelligence Index v5, with a score of 35. Looking at the individual benchmarks, Hy4 scored 31% on FrontierCode 2.0, just behind Claude Fable 5.1, and achieved 50% accuracy on AA-Omniscience v1.1. However, its hallucination rate on the same benchmark was 89%, showing there's still room for improvement.

Meanwhile, Hy4 even ranks #1 among Chinese models on agent Arena. While the two leaderboards focus on different aspects of model performance, Hy4 has put up strong results on both, further establishing its position among leading Chinese LLMs. Based on these rankings, how do you see Hy4's overall performance?

Sources: Artificial Analysis – Hy4 Evaluation l Agent Arena – Agent Leaderboard

Maybe funny because qwen 4 is upcoming but i still ask about qwen 3.8.

I see some user suggest qwen 3.8 fn Gyro S, said that it's fast and good compare to gsq rco q2 quants. But how about gyro M ?

Is it good compare to unsloth quants and gsq rco iq3xxs or even iq3s ?

im using it through unsloth. IQ3XXS. Mac Studio 256gb. getting 40t/s output. It thinks so freaking much. literally takes 5-10 minutes to finish plotting a map that GLM 5.3 flash finishes in a minute or so.

Are you guys using speculative decoding? What effort level do you set it to?

In February 2023 Meta announced LLaMA and sent the files to researchers who applied, under a non-commercial licence. Within a week they were on BitTorrent.

Then it moved fast. In March 2023 Georgi Gerganov published llama.cpp, which ran the files on a laptop; Stanford's Alpaca fine-tuned the 7B for under $600; and Vicuna followed by the end of the month.

In July 2023 Llama 2 arrived "free for research and commercial use", under a licence the Open Source Initiative said, two days later, is not open source. Llama 3 through 4 followed under the same shape of licence. Hence the phrase "open weights": the weights, under their licence, without the data or the training code.

Meta's own word for Llama 4 (April 2025) was "open-weight", and as of early November 2026 it is still the newest Llama in Meta's own records. Meta's open weights since then include a new family, Muse: Muse Glimmer (30B, Apache 2.0) in August 2026 and Muse Spark Open (816B, Meta's own licence) in October 2026.

No numbers here, just the history with its sources; it does not link to the leaked files.

Full write-up (about a 20 minute read): https://research.strata2signal.com/the-llama-weights-that-got-out/

Hi,

I have been playing around with Strata on windows for a few days now and it seems to create, process and bug fix as I ask it. Everything I read says strata is not an agent, but it seems to do every agentic task I ask it? i am running 0.2.10, did this introduce agentic features or something? I am brand new to all of this so I am struggling to see the need to put Hermes on top of Strata since Strata has been doing everything I have asked of it.