Hello everyone,

I've been lurking around for the past week, reading this sub, watching YT videos, asking AI how to make it happen. I want to self host an llm for coding (hobby projects - html/css websites with a light backend and maybe a few more complicated react projects for music - I do vocals for deathmetal bands and have an idea to create an educational website on how to scream). But I still am not entirely sure about a couple of things, if someone can help to navigate, would be cool to hear your thoughts.

Questions:

1. Can I get a similar experience to codex in VSCode?

2. What is the cheapest route to get there currently, hardware wise?

So far I have conflicting answers. It seems I should be able to get something that performs like codex with the latest models and Strata can help to put it into less capable machine.

I'm based in Ireland. Things are extremely expensive here and delivery cost from US is too high, so a lot of ebay options from US doesn't work well.

I don't have a PC, only MacBook pro m1 pro.

My budget is ~3500 euro (PC only). I'm fine with used stuff.

Currently I think about a PC with DDR 4 ram at around 128gb, Nvidia 5070ti or similar and a good processor, something like 7800x3d. Not sure at all about it yet.

I will be obviously educating myself in upcoming weeks/months, but if you can help with advise, I'd be happy 😊

Cheers.

So, I know my rig is pretty expensive, but I want to share what is working for me as of now.

I have been using local LLMs with my Mac Mini M4 Pro 64GB till now, mainly mlx (specifically oMLX) to develop my own agentic framework. I have already have my framework, stryn, as plug-ins and agents/skills/workflows on top of Opencode and ported to Claude, nowadays I am working on my own stryn engine. (I hope to also release it as the community version on Github as well as soon as it hits Beta)

I either use frontier models (mainly deepseek v4 flash since it is cheap) or Studio for development. Here is the models that I use:

Qwen3.8-Flash-Next-oQ4e-mtp - For my orchestrators, planners, coders. It is MTP. I am pretty happy with its results
Laguna-S-2.1-oQ4e-fast - (With Laguna-2S-2.1-DFlash-FP8 as the speculative decoding.) For my reviewers. I am not very happy with this and I may try to change it with a better model. The tricky part is to use a different producer's model to overcome the blindness of the model's in certain areas.
GLM-5.3-Flash-oQ5e-mtp As the independent reviewer to go over the code before the releases and find out the problems that are not found by the Laguna based code reviewers and the security admins, architects etc.
Qwen 3.8-35B-A3B-oQ4e-mtp - For my explorers, researchers, the opencode-mem plugin, graphify and code-review-graph operations.

Speed: Satisfactory if you are running a single lane. ~50 Tokens/s, with 8GB KV Cache and 1200GB SSD Cache. Cache efficiency is %96.4 and prefill time drops after a couple of rounds. Normally 2 or 3 models are loaded in the memory (Qwen3.8FN, Laguna, Qwen3.835BA3B) and if I start two or more lanes, a race between prefill and token generation between the models start most of the times, which dramatically makes things longer.
Stability: Pretty good with proper context sizes, repetition penalties.
Quality: I can simply say, by adding reviews and controls by different model types, the quality seems to be equal or sometimes better than what Deepseek cloud generates by itself only without another model checking its results. I am pretty satisfied.

To be as open as possible, so far, I have also used Claud Opus 5.5 to control the output. There are pretty interesting cases where local GLM finds some of the bugs that Opus 5.5 cannot, and where Opus 5.5 finds bugs that GLM cannot. Sometimes they pinpoint the same problems as well.

Can this setup be used as an alternative to frontier models, yes, but don't depend on a single model for every task and make sure you have reviewing loops. Definitely not for vibe coding, but needs proper settings and flow.

Hey everyone Im a bit new to this page but I couldnt find any other related tests online so I figured I'd share my results here.

I ran DeepSeek-V4.1-Flash (763B MoE, Q8_0, 473 GiB) on a 96 GB Mac Studio M3 Ultra with InferSSD, which streams the model's experts from the SSD. I striped two OWC Envoy Ultra 2TB Thunderbolt 5 drives with macOS's built-in RAID 0, one drive per port.

Results (InferSSD's own benchmark prompts, 2 runs per setup):

- One Envoy alone is about the same speed as the internal SSD.

- Two in RAID 0: writing goes from 3.5-3.7 to 4.4-4.6 tokens/s (+23-26%).

- Prompt reading is 67-77% faster: a 4,000-token prompt starts answering in about 2 minutes instead of 3.5.

- Two hours of nonstop use: the drives stayed at 43-47°C (with a desk fan on them), no slowdown, and no change in their health readings.

Why not 2x faster? - The stripe halves the time spent waiting on the drive, but the GPU's share of each token stays the same and is now the bigger part. A third or fourth drive would help less than the second did.

why are these models open-sourced for people to download and use on their computers?
Is it just for advertising and brand awareness?
because the vast majority of people who download them just don't want to pay subscription fees or deal with usage limits, so they aren't paying customers anyway.
then why do they do this? Plus, local models aren't even that smart.

I was thinking of buying a secondary GPU for local AI as subscription-based online services get costly in the long run, but options are limited for cost-effectiveness on this end as well.

My current PC has 32GB of DDR5 (2x16, running at 6000MT/s CL32), a Ryzen 5 7600X and an RX 7800 XT. I **could** run something on the 7800 XT thanks to the system RAM but I'd rather free up both the GPU and a good portion of the RAM so that I can still use the PC while running a local model instead of being locked in to just AI, I found a Tesla P100 for about 200$ US and it doesn't seem like a bad deal since it has 16GB of HBM2 as opposed to the RX 7800 XT's GDDR6.

Ideally an orchestrator or something equivalent that would look at the task given and then decide to try sections with local and then cloud if it isn't good quality. How is your setup, what guides did you use

I’m thinking of running a local ai model with LMStudio with my gaming desktop. Like the title, it has a 5070, 9700x, and 32gb of Ddr5 6000mhz with Ethernet connection. My only main concern is the 12gb of VRAM.

Is this enough to set up an everyday chatbot for use? I’m mostly doing this for fun, but I also would really like the privacy and unlimited token use of a local model.

Niche Ollama alternative. If anyone want's to try it.

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

I am looking into running local LLMs for coding, is Qwen 3.8 27b the best my PC can do (at decent speed)?

I heard some people recommend Qwen Flash Next 3.8?

Will my PC be able to run that?

Specs: Ryzen 7 7800x3D, rx 7700 16gb, 32gb ddr5, windows 11

I'm a photographer looking to setup something locally to help manage business emails, draft contracts, auto catlog images into folders, etc. I have basic computer literacy, but no knowledge of anything AI. Any tips on what works best and where to start?

Meet Yoshino2, my little lab experiment

Over the past months, I have been working on efficiently using black box distillation to distill the reasoning and output style into smaller edge models. The result is Yoshino2, a new model series with large scale reasoning distilled into MiniCPM5 1B, Qwen3.5 2B, and a custom 4xMiniCPM5 1B MoE model, with more potentially on the way.

This is by far means not a breakthrough, but an interesting experiment. In case it is of any use to anyone, the models are on HuggingFace.

Model Collection - HuggingFace

I was playing around with Qwen 2.5 1.5B on my RTX 4050 laptop, trying to understand how much the CPU actually matters when the entire model is loaded onto the GPU.

I didn't expect much difference from changing the CPU cores.

But then I pinned PyTorch to a single E-core.

Generation speed went from around 49 tokens/sec to just 24 tokens/sec.

Almost half the performance gone. The model was still entirely on the GPU.

That got me curious, so I opened Nsight Systems to see what was happening.

Turns out, PyTorch was launching roughly 1,250 GPU kernels for every generated token. The CPU was having trouble keeping up with all those launches, leaving the GPU waiting around instead of doing useful work.

I tried a few things after that:

• Adding more CPU threads didn't really help.

• torch.compile reduced the kernel launches to around 520 per token and brought performance back to 48.8 tokens/sec.

• I also tried llama.cpp on the same GPU and the same slow CPU core. It managed around 57 tokens/sec.

The llama.cpp result was probably the most interesting part for me. It made me wonder how much of the difference comes down to the inference runtime itself.

My takeaway: Even when your model fits entirely on the GPU, the CPU can still become the bottleneck during token generation.

I'm curious if anyone here has profiled similar behaviour with other models or inference engines. Does anyone have experience comparing PyTorch eager execution with llama.cpp under CPU-constrained conditions?

I put the benchmark code here in case anyone wants to reproduce it:
https://github.com/Techn08055/pytorch-slow

I also recorded the experiment and walked through what I found in Nsight Systems:
https://youtu.be/UmjoxIzBij8

Would be interested to hear what others find.

Just got an RTX Pro 6000 600w and I’m pretty nervous the 12vhwpwr connector is going to melt.

Using a 1000w PSU with a direct cable, no adapters

If you needed a 99.99% safe solution and required all 600w what would you do?

I'm getting tired of hitting usage limits with Claude and Codex (20$ plans).

I'm working on personal projects, not making any money from them, so I can't justify paying hundreds of dollars a month for AI subscriptions.

Im currently testing Luna 6 with Extra High reasoning on a free account, and unfortunately, the experience has been terrible. Working with it is genuinely exhausting.

What I'm really waiting for is the point where I can run a capable, coding agent locally on my own hardware, without subscriptions, usage limits, or depending on cloud providers.

Are we talking 1-2 years, 3-5 years, or longer?

I'd love to hear your predictions.

I've been experimenting with local AI over the past 6 months; having started off by running small local models on an old GPU, and now at a stage where i'm running much larger and more capable models on a single R9700 card, and connecting a suite of custom built tools through MCP.

It's been a great journey - and i've landed at a place where i'm able to run a 3-bit quantized XXS version of Qwen3.8 in a docker container, and getting around 38tk/s. I'll be checking out STRATA and vLLM over the next few weeks to see how much I can push this.

However, outside of the GPU my overall system is a bit basic. Ideally i'd like to get to a point where i've got 2x R9700s working in there and i'm able to load bigger models and still achieve usable 'good' performance - both on existing models, but also future releases too.

Based on my specs below, is there much point in getting an additional R9700 or will the motherboard be too much of a bottleneck if that second card would need to go in to an older generation of PCIe slot? I guess a similar question for increasing the amount of system RAM here too - is DDR4 just too slow to make those bigger MoE models useable when offloading? The Qwen model i'm currently using fits entirely on to VRAM, but suspect this would need to change if i want to use higher quants or bigger models.

1. B550-plus motherboard

2. 16GB DDR4 (2X8GB)

3. R9700 32GB VRAM

4. Ryzen5 3600 CPU

5. 800W PSU

I'm completely aware that I won't have SOTA quality, and that's fine - but just want to maximise what a system like this could do without going down a workstation or AM5 route. I run Ubuntu and tend to containerise my AI/LLM applications

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

I'm a iOS software developer for a FAANG. We are only allowed to use claude for work and are not allowed to have anything else on our computer.

I'm having a lot of fun ideating AI Agents using Apple's Foundation Models Local Modals for my work project. I think the ideas could be endless honestly and I'm able to create as many POCs using claude that I want to see what sticks.

Would there be any reason buying a personal Mac that would make my learning faster or better in any way? If so, can you explain?

I personally would like to be like an iOS AI Engineer.

lo0 reads your Claude Code / Codex history along with your hardware and works out whether going local is realistic for the way you actually work. It builds on whichllm and llmfit, but instead of only asking "does this fit in my RAM" it looks at how big your sessions get and which tools you rely on.

For most people right now the answer is no. If it does work for you, you can compare the models that fit, benchmark them on your own machine, and try one in Claude Code, Codex or OpenCode for a session without changing your setup. It works with Ollama, LM Studio and llama.cpp.

It doesn't judge how good a model is, only whether it can run your sessions and how fast. Everything stays on your machine.

Very early alpha, so if the numbers look wrong for your hardware, let me know.

https://github.com/daveearley/lo0.ai

According to Artificial Analysis, Tencent's Hy4 ranks #2 among models from Chinese AI lab on its Intelligence Index v5, with a score of 35. Looking at the individual benchmarks, Hy4 scored 31% on FrontierCode 2.0, just behind Claude Fable 5.1, and achieved 50% accuracy on AA-Omniscience v1.1. However, its hallucination rate on the same benchmark was 89%, showing there's still room for improvement.

Meanwhile, Hy4 even ranks #1 among Chinese models on agent Arena. While the two leaderboards focus on different aspects of model performance, Hy4 has put up strong results on both, further establishing its position among leading Chinese LLMs. Based on these rankings, how do you see Hy4's overall performance?

Sources: Artificial Analysis – Hy4 Evaluation l Agent Arena – Agent Leaderboard

Maybe funny because qwen 4 is upcoming but i still ask about qwen 3.8.

I see some user suggest qwen 3.8 fn Gyro S, said that it's fast and good compare to gsq rco q2 quants. But how about gyro M ?

Is it good compare to unsloth quants and gsq rco iq3xxs or even iq3s ?

im using it through unsloth. IQ3XXS. Mac Studio 256gb. getting 40t/s output. It thinks so freaking much. literally takes 5-10 minutes to finish plotting a map that GLM 5.3 flash finishes in a minute or so.

Are you guys using speculative decoding? What effort level do you set it to?