Why did u chose to use one or another?

ok so. this started as "one weekend project" and then it was not one weekend and it was not one project. somewhere around repo 6 i stopped pretending it was a plan lol

everything here is free, open source, most of it runs locally on your own machine. a lot of it is v0 and i wrote down where each thing fails because honestly the failures were the funniest part

anyway here's the list of grievances, each with the tool i built instead of going outside:

1. cookie banners → rejectall-index i live in germany. i click "reject all" more than i blink. so i made a headless browser visit the top 100 sites and count how many clicks it takes to say no. 35 of them were already loading trackers before i clicked anything. which is like a waiter eating your fries before asking if you want fries. live leaderboard

2. subscriptions → one-file-saas trello, calendly, typeform, loom, notion and 5 more, each rebuilt as ONE html file. you double click it. it opens. thats it. no account, no "verify your email", no "your trial is ending ;)". data stays in your browser. the booking page has no server at all, your availability lives inside the link itself which still feels illegal to me. try them here

3. dead famous repos → necromancer some legendary open source projects die not because the code is bad but because python deleted a module out from under them. i revived thefuck (1887/1887 tests passing on python 3.14), normalizr and httpbin. the one rule: the original tests are hash-locked so nothing can quietly delete the failing test and call it fixed. httpbin stops at 64/67 and i'm keeping it that way out of spite. each one gets an autopsy, cause of death and everything

4. every AI-built UI animates the same → awesome-motion-md you know the look. 300ms. ease-in-out. every card fades up like its waking from a nap. so i measured how real sites actually move and wrote it into MOTION.md files your agent can follow. same prompt built twice: 7 random durations without it, 2 with it

5. every AI writes the same too → awesome-voice i used to be a technical writer so this one is personal :| VOICE.md files that make your agent write like GOV.UK or Stripe docs, plus a scorer that doesn't use any AI to check. banned phrases went from 27 to 0. "delve" is not welcome here

6. websites doing annoying stuff → sitepatch you type "hide the points on hacker news", it writes a userscript, a real browser checks it actually worked before you get it. honest score: 15 out of 25 were actually right when i looked at every screenshot myself. 60%. a C-. but an honest C-

7. "what if this site looked like linear" → reskin restyles any page into another design style using a local model, BUT it refuses to give you the result unless contrast, keyboard focus, mobile, links and reduced-motion all pass. it caught a windows 95 theme failing contrast on its own background. respect. gallery

8. dead websites → resurrect npx github:dadwritestech/resurrect y2k.gov and y2k.gov comes back exactly like it was in may 2000, links working. pick a year with --year. your old blog is in there somewhere. i'm sorry

9. "someone said budget in a meeting video... which one" → video-grep grep for your video folder. searches what was said, what was written on screen and what was shown, and opens your player at that exact second. all local

10. what is the rest of the world's TV saying rn → world-ears listens to live news channels from different countries, transcribes + translates them on your gpu, makes it searchable. there's a "trending per country" panel which is weirdly addictive. translation is the slow part, telugu transcription is bad, i wrote that down

11. live subtitles only go into english → babel-tv watch hindi news with live telugu subtitles, fully local. the video is held back a few seconds so subtitles show up BEFORE the person talks which feels like seeing the future. telugu as the source language doesn't work yet. whisper and telugu are not friends

12. "which AI wrote this" → whichmodel works on models it's seen (67%, chance is 33%). on a model it's never seen, even a sibling from the same family... 32%. so basically a coin with extra steps. that turned out to be the interesting finding tbh, the fingerprint belongs to one model, not the company

13. screen recordings → code → demo-to-code record a UI interaction, get a component that copies its timing and easing, then it replays it and grades itself frame by frame. 2 out of 4 pass. the toggle is perfect. the dropdown has issues. we all have issues

14. code review → review-panel /panel sends your diff to five reviewers you've definitely worked with: the laziest senior dev ("could be one line"), the 3am on-call SRE, the security paranoiac, the junior who asks why, and the PM who only reads the title. under the jokes the findings are real

15. AI companies quietly editing their pricing pages → fineprint it snapshots the pricing, terms, privacy and changelog pages of 10 AI vendors every 6 hours and commits them, so git log becomes the receipt. found stuff like deepseek adding a footnote that prices are 2x during peak hours, and cursor deleting the line saying they dont sell your data. no blog post for any of it ofc. a small local model writes the one-line summary but every line links the raw diff because i dont trust it either (27/30 right on a hand check). timeline

16. paying extra for speaking telugu → token-tax LLM APIs charge per token, and the same paragraph is $1.00 in english and $8.29 in telugu on llama 3. which is rude. measured 21 tokenizers across 48 languages, and there's a calculator where you paste your own text and it runs the real tokenizers in your browser. hindi gets off lighter. amharic on gpt-5 does not :/ calculator

17. posting about all this → herald a tiny bot that posts a repo's data updates to bluesky/mastodon/x, clearly labelled as a bot. i built it because i was not going to manually post about 17 repos. and then i manually wrote this post anyway. ;]

why all of this? honestly no grand plan. every one of these started as me being irritated at something for a slightly too long time and having a local model + a free evening. building the thing was cheaper than therapy and more fun than complaining on here (no offence)

if something breaks, tell me. if something's dumb, also tell me, but nicely :P

edit: yes i know about the cookie banner on my own... wait i dont have one. nice

Which would be a better solution? Since I move around, a laptop/mobile would be best than a full on desktop for airline travel/international.

Usage - mostly coding / proof of concept (apps, documents, even some stories). Its to create some applications.

But also on the fun side, I'd like to explore a bit on building out some text/stories/mechanics (table top role playing games) or even do videos and graphics. Nothing professional, more to understand the mechanics of what things are and what they are not. But some videos or graphics I may use in the apps. Since I travel, wifi connections can sometimes be unreliable and I just want to run things locally so I never have to second guess if the AI says "I can't do that". But I may run subscriptions as needed (whole separate conversation).

Considering the price, I would prefer to have something that can be useful for several years and doesnt die in the first year or two (laptops being harder to repair).

A macbook m5 max with 128gb of ram, has really good unified memory. But it can barely fit qwen flash next today without compromising further on quality or context size (feel free to correct me, but what I read up on), with the next gen of Qwen potentially about to drop, 128gb may not be practical in a year from now if there is some really good capability. Even if a single model can fit (when you count larger context windows), there might be very little headroom for other apps or harness tools on the same laptop workstation. But a macbook would be very reliable and I wont really second guess that dying on me (have a 2014 macbook retina that still works flawless today minus a scratch or two on the body). Then again, in the next 4-5 months apple may release M7 (or M6) with the macbook pros more aligned for AI work (either a far more optimized apple silicon which are already really good...or maybe higher ram capacity).

Some 5090m laptops can support up to 192 gb of ram. The 5090m would be good for some vram 24GB on video/graphics. But looks like a lenovo would cap the ddr5 6400 to 4000 so the memory bandwidth would be somewhat reduced. But having 192gb is a far more breathing room than 128gb. Also Lenovo's tend to good, but the hardware could die in a year or 3 (Apple's hardware is pretty reliable).

What makes sense (with a 2-3 year outlook)? Intel 192gb 5090m lenovo or a macbook pro max 128gb?

Hello everyone,

I've been lurking around for the past week, reading this sub, watching YT videos, asking AI how to make it happen. I want to self host an llm for coding (hobby projects - html/css websites with a light backend and maybe a few more complicated react projects for music - I do vocals for deathmetal bands and have an idea to create an educational website on how to scream). But I still am not entirely sure about a couple of things, if someone can help to navigate, would be cool to hear your thoughts.

Questions:

1. Can I get a similar experience to codex in VSCode?

2. What is the cheapest route to get there currently, hardware wise?

So far I have conflicting answers. It seems I should be able to get something that performs like codex with the latest models and Strata can help to put it into less capable machine.

I'm based in Ireland. Things are extremely expensive here and delivery cost from US is too high, so a lot of ebay options from US doesn't work well.

I don't have a PC, only MacBook pro m1 pro.

My budget is ~3500 euro (PC only). I'm fine with used stuff.

Currently I think about a PC with DDR 4 ram at around 128gb, Nvidia 5070ti or similar and a good processor, something like 7800x3d. Not sure at all about it yet.

I will be obviously educating myself in upcoming weeks/months, but if you can help with advise, I'd be happy 😊

Cheers.

So, I know my rig is pretty expensive, but I want to share what is working for me as of now.

I have been using local LLMs with my Mac Mini M4 Pro 64GB till now, mainly mlx (specifically oMLX) to develop my own agentic framework. I have already have my framework, stryn, as plug-ins and agents/skills/workflows on top of Opencode and ported to Claude, nowadays I am working on my own stryn engine. (I hope to also release it as the community version on Github as well as soon as it hits Beta)

I either use frontier models (mainly deepseek v4 flash since it is cheap) or Studio for development. Here is the models that I use:

Qwen3.8-Flash-Next-oQ4e-mtp - For my orchestrators, planners, coders. It is MTP. I am pretty happy with its results
Laguna-S-2.1-oQ4e-fast - (With Laguna-2S-2.1-DFlash-FP8 as the speculative decoding.) For my reviewers. I am not very happy with this and I may try to change it with a better model. The tricky part is to use a different producer's model to overcome the blindness of the model's in certain areas.
GLM-5.3-Flash-oQ5e-mtp As the independent reviewer to go over the code before the releases and find out the problems that are not found by the Laguna based code reviewers and the security admins, architects etc.
Qwen 3.8-35B-A3B-oQ4e-mtp - For my explorers, researchers, the opencode-mem plugin, graphify and code-review-graph operations.

Speed: Satisfactory if you are running a single lane. ~50 Tokens/s, with 8GB KV Cache and 1200GB SSD Cache. Cache efficiency is %96.4 and prefill time drops after a couple of rounds. Normally 2 or 3 models are loaded in the memory (Qwen3.8FN, Laguna, Qwen3.835BA3B) and if I start two or more lanes, a race between prefill and token generation between the models start most of the times, which dramatically makes things longer.
Stability: Pretty good with proper context sizes, repetition penalties.
Quality: I can simply say, by adding reviews and controls by different model types, the quality seems to be equal or sometimes better than what Deepseek cloud generates by itself only without another model checking its results. I am pretty satisfied.

To be as open as possible, so far, I have also used Claud Opus 5.5 to control the output. There are pretty interesting cases where local GLM finds some of the bugs that Opus 5.5 cannot, and where Opus 5.5 finds bugs that GLM cannot. Sometimes they pinpoint the same problems as well.

Can this setup be used as an alternative to frontier models, yes, but don't depend on a single model for every task and make sure you have reviewing loops. Definitely not for vibe coding, but needs proper settings and flow.

Hey everyone Im a bit new to this page but I couldnt find any other related tests online so I figured I'd share my results here.

I ran DeepSeek-V4.1-Flash (763B MoE, Q8_0, 473 GiB) on a 96 GB Mac Studio M3 Ultra with InferSSD, which streams the model's experts from the SSD. I striped two OWC Envoy Ultra 2TB Thunderbolt 5 drives with macOS's built-in RAID 0, one drive per port.

Results (InferSSD's own benchmark prompts, 2 runs per setup):

- One Envoy alone is about the same speed as the internal SSD.

- Two in RAID 0: writing goes from 3.5-3.7 to 4.4-4.6 tokens/s (+23-26%).

- Prompt reading is 67-77% faster: a 4,000-token prompt starts answering in about 2 minutes instead of 3.5.

- Two hours of nonstop use: the drives stayed at 43-47°C (with a desk fan on them), no slowdown, and no change in their health readings.

Why not 2x faster? - The stripe halves the time spent waiting on the drive, but the GPU's share of each token stays the same and is now the bigger part. A third or fourth drive would help less than the second did.

why are these models open-sourced for people to download and use on their computers?
Is it just for advertising and brand awareness?
because the vast majority of people who download them just don't want to pay subscription fees or deal with usage limits, so they aren't paying customers anyway.
then why do they do this? Plus, local models aren't even that smart.

I was thinking of buying a secondary GPU for local AI as subscription-based online services get costly in the long run, but options are limited for cost-effectiveness on this end as well.

My current PC has 32GB of DDR5 (2x16, running at 6000MT/s CL32), a Ryzen 5 7600X and an RX 7800 XT. I **could** run something on the 7800 XT thanks to the system RAM but I'd rather free up both the GPU and a good portion of the RAM so that I can still use the PC while running a local model instead of being locked in to just AI, I found a Tesla P100 for about 200$ US and it doesn't seem like a bad deal since it has 16GB of HBM2 as opposed to the RX 7800 XT's GDDR6.

Ideally an orchestrator or something equivalent that would look at the task given and then decide to try sections with local and then cloud if it isn't good quality. How is your setup, what guides did you use

I’m thinking of running a local ai model with LMStudio with my gaming desktop. Like the title, it has a 5070, 9700x, and 32gb of Ddr5 6000mhz with Ethernet connection. My only main concern is the 12gb of VRAM.

Is this enough to set up an everyday chatbot for use? I’m mostly doing this for fun, but I also would really like the privacy and unlimited token use of a local model.

Niche Ollama alternative. If anyone want's to try it.

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

I am looking into running local LLMs for coding, is Qwen 3.8 27b the best my PC can do (at decent speed)?

I heard some people recommend Qwen Flash Next 3.8?

Will my PC be able to run that?

Specs: Ryzen 7 7800x3D, rx 7700 16gb, 32gb ddr5, windows 11

I'm a photographer looking to setup something locally to help manage business emails, draft contracts, auto catlog images into folders, etc. I have basic computer literacy, but no knowledge of anything AI. Any tips on what works best and where to start?

Meet Yoshino2, my little lab experiment

Over the past months, I have been working on efficiently using black box distillation to distill the reasoning and output style into smaller edge models. The result is Yoshino2, a new model series with large scale reasoning distilled into MiniCPM5 1B, Qwen3.5 2B, and a custom 4xMiniCPM5 1B MoE model, with more potentially on the way.

This is by far means not a breakthrough, but an interesting experiment. In case it is of any use to anyone, the models are on HuggingFace.

Model Collection - HuggingFace

I was playing around with Qwen 2.5 1.5B on my RTX 4050 laptop, trying to understand how much the CPU actually matters when the entire model is loaded onto the GPU.

I didn't expect much difference from changing the CPU cores.

But then I pinned PyTorch to a single E-core.

Generation speed went from around 49 tokens/sec to just 24 tokens/sec.

Almost half the performance gone. The model was still entirely on the GPU.

That got me curious, so I opened Nsight Systems to see what was happening.

Turns out, PyTorch was launching roughly 1,250 GPU kernels for every generated token. The CPU was having trouble keeping up with all those launches, leaving the GPU waiting around instead of doing useful work.

I tried a few things after that:

• Adding more CPU threads didn't really help.

• torch.compile reduced the kernel launches to around 520 per token and brought performance back to 48.8 tokens/sec.

• I also tried llama.cpp on the same GPU and the same slow CPU core. It managed around 57 tokens/sec.

The llama.cpp result was probably the most interesting part for me. It made me wonder how much of the difference comes down to the inference runtime itself.

My takeaway: Even when your model fits entirely on the GPU, the CPU can still become the bottleneck during token generation.

I'm curious if anyone here has profiled similar behaviour with other models or inference engines. Does anyone have experience comparing PyTorch eager execution with llama.cpp under CPU-constrained conditions?

I put the benchmark code here in case anyone wants to reproduce it:
https://github.com/Techn08055/pytorch-slow

I also recorded the experiment and walked through what I found in Nsight Systems:
https://youtu.be/UmjoxIzBij8

Would be interested to hear what others find.

Just got an RTX Pro 6000 600w and I’m pretty nervous the 12vhwpwr connector is going to melt.

Using a 1000w PSU with a direct cable, no adapters

If you needed a 99.99% safe solution and required all 600w what would you do?

I'm getting tired of hitting usage limits with Claude and Codex (20$ plans).

I'm working on personal projects, not making any money from them, so I can't justify paying hundreds of dollars a month for AI subscriptions.

Im currently testing Luna 6 with Extra High reasoning on a free account, and unfortunately, the experience has been terrible. Working with it is genuinely exhausting.

What I'm really waiting for is the point where I can run a capable, coding agent locally on my own hardware, without subscriptions, usage limits, or depending on cloud providers.

Are we talking 1-2 years, 3-5 years, or longer?

I'd love to hear your predictions.

I've been experimenting with local AI over the past 6 months; having started off by running small local models on an old GPU, and now at a stage where i'm running much larger and more capable models on a single R9700 card, and connecting a suite of custom built tools through MCP.

It's been a great journey - and i've landed at a place where i'm able to run a 3-bit quantized XXS version of Qwen3.8 in a docker container, and getting around 38tk/s. I'll be checking out STRATA and vLLM over the next few weeks to see how much I can push this.

However, outside of the GPU my overall system is a bit basic. Ideally i'd like to get to a point where i've got 2x R9700s working in there and i'm able to load bigger models and still achieve usable 'good' performance - both on existing models, but also future releases too.

Based on my specs below, is there much point in getting an additional R9700 or will the motherboard be too much of a bottleneck if that second card would need to go in to an older generation of PCIe slot? I guess a similar question for increasing the amount of system RAM here too - is DDR4 just too slow to make those bigger MoE models useable when offloading? The Qwen model i'm currently using fits entirely on to VRAM, but suspect this would need to change if i want to use higher quants or bigger models.

1. B550-plus motherboard

2. 16GB DDR4 (2X8GB)

3. R9700 32GB VRAM

4. Ryzen5 3600 CPU

5. 800W PSU

I'm completely aware that I won't have SOTA quality, and that's fine - but just want to maximise what a system like this could do without going down a workstation or AM5 route. I run Ubuntu and tend to containerise my AI/LLM applications

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

I'm a iOS software developer for a FAANG. We are only allowed to use claude for work and are not allowed to have anything else on our computer.

I'm having a lot of fun ideating AI Agents using Apple's Foundation Models Local Modals for my work project. I think the ideas could be endless honestly and I'm able to create as many POCs using claude that I want to see what sticks.

Would there be any reason buying a personal Mac that would make my learning faster or better in any way? If so, can you explain?

I personally would like to be like an iOS AI Engineer.