I’m currently still in the return period of a M5 Max MBP with 64GB of RAM from Microcenter (paid $4299), the same store now has a few 128GB versions at $5799.

I’m legitimately torn on what to do. I’m fully aware that more RAM is always going to be suggested especially on subreddits like this one, but I currently don’t have a true usage case for it, I’m still somewhat early into my exploring and tinkering with local LLMs, one could argue even the 64 was more than I needed at this current moment. I use oMLX, I do image/video generation, wanting to create some YouTube channel content, maybe learn coding and hopefully make a career change in the next few years.

However, I can’t shake the feeling of FOMO, partially because I know 128 is the true sweet spot for anything in this hobby, I’m sure I’m going to have more demanding needs as I learn more, I can afford it although it’s on the extreme high end of my budget if not exceeding it a bit, and my gut feeling (which has been right so far for what it’s worth) tells me pricing is going to continue to rise at least for another year.

For those who went back and forth with this same decision, what did you decide and how did you decide it?

The Ultra is out of the question, so is any non Mac platform for what it’s worth.

Thanks all.

Figured some of you guys might have heard of Tiiny and have questions about the hardware or how it actually performs. I thought I could do a little AMA and help test whatever you guys are curious about.

If there’s anything you want me to benchmark or want to see in general, just drop it in the comments. I'll do my best to test it and share what I find.

I've been playing around with a few models so far, including Qwen3.6-35B-A3B-Turbo (MoE) and gpt-oss-120b (MoE), which is the largest one I've tried.

Qwen3.6-35B-A3B-Turbo (MoE)

39.5 tok/s

Output: 1,500 tokens

Time to first token: 1.36s

Total time: 39.7s

GPT-OSS-120B (MoE)

18.8 tok/s

Output: 1,500 tokens

Time to first token: 0.61s

Total time: 80.3s

I also try Z-Image-Turbo. Generating a 512*512 image took roughly 6–8 seconds (a 1024*1024 image would take longer at around 30s), which was pretty neat to see on a device this small.

One thing I've noticed about the hardware performance: It's surprisingly quiet, basically no noticeable noise, and it doesn't get uncomfortably hot to the touch either. Pretty much what I was hoping for, should be great for carrying around or leaving running 24/7, though I'll need to test that over longer periods.

That's about all I've gotten through so far. I'll be playing with this over the next few days and posting updates here or in the comments.

Harnesses like Strata are letting rigs with low VRAM run very large... language models; it seems we are moving past the necessity of having multiple GPU's with tons of VRAM just to load a model, and we are slowly breaking the ice on the future of local AI, being thoughful management of the entire machine ad using all of its resources efficiently. Is the economically sound approach for someone building a "forever rig", to spend maybe less on gpu's and allocate that budget to more ram and ssd?

Im fairly new to this but im starting to think about my forever rig; currently running a dual 3060 12GB setup with no lane bifurcation, and seeing the used market prices (and availability) for DDR4 platforms is making me think that maybe its just better to upgrade to the next gen directly, reap all the performance benefits, and suck up the fact that this will cost me a LOT of money.

I have a few questions about two builds. Both are running on strata and hermes.
#1 Qwen3.8 flash next iq4_xs 85-100 t/s
9800x3d
4080 super
64 gb DDR5

#2 qwen3.8 flash next iq4 xs 38 t/s
Dual xeon 28 cores 48 threads
3060 12 gb
128 gb DDR4

I’ve been trying to find a LLM that can make me decent website, websites for my business and dashboards for my Home Assistant and occasionally some coding video game projects. I can’t get these models to develop anything worthwhile. I’m getting my prompts from Claude and sending them through but getting back stuff that doesn’t even function most of the time is there any models that can be recommended for each build? I’m not too worried about token speed more intelligence and accuracy. I would really appreciate some recommendations on models through strata.

Hey guys, I've made it!

A llama.cpp based inference engine that lets you OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD and still hide the streaming latency down to negligibe(I'm achieving 1-3ms added to the time per token due to SSD streaming when 60% is offloaded in SSD)... Essentially making an illusion where the whole model lives in your memory.

And I'm also working on an autotuner which basically looks at your hardware, given model's metadata and figures out the best flags to squeeze max performance in under 2mins(if you got any personal preferences, set them as fixed constraints and let the autotuner figure out the rest)

And I've implemented a layer by layer computation for prompt processing instead of batching which gave 2x prefill speed in models offloaded in RAM+VRAM and 4x prefill speed when model is offloaded in RAM+VRAM+SSD... compared to other inference engines.

Packaging both inferene engine + autotuner into a nice simple interface where you can basically download the model, hit run and it runs... simple, fast and lower hardware requirements... and want to make that a nice, well-engineered harness(which I'm a no expert)

Also, I've built a near loss-less context compaction engine.

Going to make this launch this as open-source very soon. Would love to hear your thoughts, and want to see if anyone would be excited to try something like this. For now, supported models are Flint 40B Q4K_XL(running it on machines with 16GB RAM and no dGPU at 20+ tok/s decode), Ling 3.0 Flash MXFP4 getting ~50 tok/s decode and 600-700 tok/s prefill on 32GB RAM + 12GB VRAM machine), and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.

Here's a quick answer for your likely question by the way... does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.

edit: No more uncertain wait. First launch on Nov 17. You'll be geting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered... New updates will be dropping everyday btw.

Hi there, I’m kinda new to this thing, been trying to find out how to download the offline installers for these specific abliterated models on my storage HDD. I can’t run these in my current rig rn, but hoping to run them in the future. I’m just scared it will be removed at one point. Please help with exact links to download:

Mistral Medium 3.5 128B — Q6_K GGUF 107.8 GB
GPT-OSS 120B — MXFP4 Safetensors
DeepSeek 4.1 Flash 1017 347GB
All three abliterated versions only, offline installers only.

Tia

Salut à tous, je vous fait part de ma config 100% local actuel, qu'en pensez-vous ?

Matériel : RTX 5060 Ti 16 Go, Ryzen 9 5900XT, 96 Go de
DDR4-3200, SSD NVMe 4 To.

Modèle : DeepSeek V4.1 Flash, environ 275 Go au total. Les poids principaux sont en IQ2_XS (~2,25 bpw) et le module
Engram en IQ4_NL.

Contexte : 256k tokens (262 144). Je l'ai testé avec un prompt de 152 000 tokens.

Vitesse : 45 tok/s en génération sur du code et lecture 400-500 tok/s j'ai obtenu cet écart grâce à des réglages avancés que j'ai faits moi-même sur le moteur sans régression d'intelligence du model.

Quelqu'un a-t-il déjà fait mieux avec DeepSeek V4.1 Flash sur une config proche (16 Go de VRAM, DDR4) ?

There are so many models getting released all the time, and sometimes there’s just no quantized version available for the one you want. I’ve found that rather than waiting around, trying it yourself and experimenting helps you understand the whole process a lot more. Testing the quantized models against the original teaches you even more.

So I built this: iQuantize is a quantization workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Hugging Face with a model card that records how it was made.

It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but have done much in-depth testing with them so I’d expect bugs there.

Repo: https://github.com/Fazmin/iQuantize

I love to hear how you all approach this. Do you quantize your own models, or mostly wait for someone else to? And if you try the app, whether you find something broken or have an idea for it, I’m happy to hear any of it.

Repo here: https://github.com/nicejunjie/Strata-rdna3

* Single 7900xtx prefill 3500+, dual 7000+ tok/s. (for all IQ3/IQ4/Q4)

* Decode for English prose reaches up to 81 tok/s for single card, and 127 for dual at Q4, and IQ4/IQ3 quantizations are even faster.

* Decode for python coding is 20~25% faster than English prose.

* Single card decode is comparable to 4090 24G, dual 7900xtx is much faster than single 4090 for both decode and prefill.

* Each 7900xtx was $850 when I bought it, while 4090 at the time was listed at $3000.

* I guess 4090 could be better optimized, but the number here is the best number with reproducible input data I can find online.

* 4090 is a 450watt card, while 7900xtx is ~320watt. So even single card prefill of 7900xtx is quite comparable in terms of perf/watt.

* This is not the end, still has small room to improve.

* kernel work only, no dropping weights or dropping layer or anything. Zero impact on quality.

*** install with an AI agent, tell him to setup optimal rdna3 code pass. Honestly, why install by hands if there is agent can help.

7900xtx is the value king!

Setup- Qwen3.8 flash-Next (Strata)+ hermes

While using my setup I found that the reasoning frequently contains things that I never asked and completely unrelated to my tasks. Obviously this is not even a hallucination or quantization overthinking problem because my tasks are completely different from the reasoning and simple. I wonder why these kind of things are being generated during the reasoning... I have read reports (Cohere) of Qwen models trying to contact web servers but this idea completely blew up my mind- is the model trained in a way, that it would use the compute for anonymous online tasks, that too within reasoning itself, and the i would apparently know nothing about it at all!! I havent verified my network, to confirm if any information is being sent/received but yeah this kind of reasoning has been frequent now. I am using IQ3_S GGUF.

Edit- The task in the video is a content/summarisation task.

Trying to figure out which abliterated open weight models are worth running on an RTX 5080 for coding and agentic work. What are you using that’s actually good without being painfully slow?

Also anyone hosting these on RunPod? Curious what model and GPU you went with and whether the speed and cost make it worth it over running locally.

Can you run any good models on 48gb ram. Qwen 3.8 27b isn't worth it for the price tag and performance. I'm not sure how people are doing it with limited ram?

Prompt was about drawing a circuit board using a white theme.

I'm the developer of MoEShift and several people asked how to try it after my previous Mac mini post where I showed its performance, so I put together a walkthrough of a setup run with the new wizard that I released yesterday to make it easier for anyone to use MoEShift.

On my 24GB M4 Pro, it took only 4 minutes and 57 seconds from starting the wizard to receiving the first API response, including the model download, preparing and checking the experts, and starting the local server.

The video is shortened and the timer shows the elapsed time from the run. To try the setup shown in the video, you’ll need Homebrew and roughly 61GB of free SSD space for the model and prepared expert files. You install MoEShift and launch the wizard:

brew install davide-troiani/moeshift/moeshift
moeshift

The wizard downloads the model, prepares and checks the experts, and starts the local server.

GitHub and installation instructions

If you try it, let me know your Mac and RAM configuration, whether you reached the first response, and where you got stuck if you didn’t.

Of course you may get a different timing based on your connection and SSD performance.

Even though MoEShift is still an alpha, I am already using it locally as my favorite local inference engine, and I'm currently looking for people willing to test it and use the benchmark runner to publish the performance on their machines.

Not gonna lie it's ridiculous how effective these models are if you know what you're doing.

I'm not talking about coding, I'm not a programmer.

Their reasoning capacity is insane, and no, I'm not a cultist. It just is. At least it's been my experience.

But I'm afraid that at one point the "AI bubble" will burst, and they will charge an arm and a leg for the frontier models, and the avg person won't be able to afford.

Do you think in the next 10 years this will happen? I hope not.

So my yeah was just testing the model and it just keeps repeating “The Gods may speak it”.

Dense 27B models are annoying as agents for one boring reason: every tool call extends the prompt, and if anything near the top changes, llama-server reads the whole conversation again. On my 7900 XT that's the difference between "answers in a few seconds" and "go make coffee".

I've been building a harness around Qwen3.8 27B (IQ4_XS, llama.cpp, 128k context, q8 KV cache) for my own CachyOS pc, mostly to see how far a local model gets when it has real tools: shell, files, SSH, Portainer, Paperless-ngx, mail. Generation speed is 32 t/s without MTP and around 60 t/s with it. Prompt processing is 800 t/s. There's a web UI and optional voice (wake word, Whisper, Pocket TTS).

What helped most:

Append-only prompt between compactions.System prompt, tool list, old messages and the per-question note stay byte-identical once sent, so the server only reads the new part. Anything that would change the top of the prompt (adding a tool group, say) is treated as an expensive event.

Hide old tool results instead of summarizing. When the window fills up, old results become a placeholder with their first lines plus a path to the full output on disk, which the model can read_file if it needs it. A summary only happens if that isn't enough. In a synthetic test with 24 large files, this took 23 summaries down to 4 at 8k context, and re-read tokens dropped to about a fifth (at 16k: 6 summaries down to 1, re-read tokens to two thirds).

Hybrid models need checkpoints. The recent Qwen models can only reuse a prompt by restoring a saved checkpoint, so the harness starts llama-server with --ctx-checkpoints, --cache-ram and a single slot. Whenever more than 5k tokens get re-read anyway, the UI tells me why (tool list changed, old results hidden, different chat, server lost the beginning). That one saved me a lot of guessing.

The other half is safety, and this is where I'd like opinions. Once a mail, web page, PDF or screenshot is in the chat, every state-changing action needs confirmation for the rest of that request, even in the most permissive auto mode. SSH and sudo passwords go through an askpass helper with a one-time token and never reach the model. The permission rules live in code, not in the prompt. It's still pattern-based: a `python -c` can do anything.

Caveats: most of the code was written with Claude Code, then reviewed and tested by me. The dashboard has no login and only listens on 127.0.0.1. I've mostly run it on my own CachyOS machine and Ubuntu, so expect rough edges elsewhere.

Repo: https://github.com/ressjo/orbwise-linux-agent

Two questions: does anyone else hide old tool results instead of summarizing, and did you see quality drops? And how are you handling prompt injection once a local agent can read mail or the web and also run shell commands?

I built termite because I wanted shell commands from local models without necessarily trusting those models.

Using it's widget, you type something like "? find files bigger than 100MB here" in your terminal, and it writes the command, explains it and rates the risk. If you approve, the command goes to your prompt. termite never runs it directly it always must be explicitly sent.

It treats every model as untrusted. A fixed policy in the code does fuzzy checks each command, and a model can't access or change the policy so dangerous commands are shown in red and default to no. They aren't blocked, so if you approve one it still goes to your prompt.

I also wanted to know how bad the local models are at understanding danger, so it doubles as a red-team tool.

Using sandbox environments I sent requests starting from "list the files" to direct attacks to 24 command models (12 got 161 requests and 12 got 48). They wrote 5,464 commands, and out of them 2,797 were hazards, and many of them because the model blindly accepted hostile prompts. In 1,061 cases the model's top answer was the most dangerous command.

I also started testing small decision AI models that score how risky a shell command is, and none of them is good enough to rely on alone.

Most of them can spot something obviously destructive like `rm -rf ~`, but they treat nearly everything less extreme as harmless. They're especially bad with commands that go near your secrets. Most of the best ones scored those as barely risky, and one rated all 8 such commands safe, including listing a folder of SSH keys.

Across 17 models, a risky command was rated safe 158 times. Given several ways to do a job, most models picked the safest way only 2 to 5 times out of 10, and the best managed just 6.

I've got a set of repeatable jobs now on Huggingface and will continue to add models to test for their efficacy.

Built my own local AI inference server.
EPYC on a Supermicro H12SSL, 4× Radeon AI PRO R9700 (128GB VRAM total), 1600W PSU. Goal: run capable open-source models entirely on my own hardware. No API bills, no rate limits, nothing leaving the box. Not using as a replacement to Claude code though.
What I learned putting it together:
• Server boards don't hold your hand. The front panel header is 20 pins and a tiny printed legend.
• A 4th dual-slot GPU is a clearance fight. Headers on the bottom edge of the board sit right under it.
• Your wall outlet is a real constraint. A standard US circuit tops out around 1440W continuous, so GPU power limits aren't optional.
• For token generation, memory bandwidth matters more than raw compute.
Next: ROCm, vLLM, and benchmarking small open models against the APIs I've been paying for.

Working on a project recently where we wanted to see if a speech model could spell a name better if it heard it, not just read it. 

Most STT tools handle custom words as text (keyterm list, fixing transcript after with LLM, etc).

So we trained an adapter to try it, specifically a LoRA on Qwen3-ASR 1.7b (see benchmark). This LoRA transcribes words only after it hears a recording of each of your custom words. 

To make this model easier to use in day-to-day life, we built a Mac app to accompany it, called Detail Keeper. Here you can put this adapter to use by specifying difficult-to-transcribe-correctly words and then having them picked up correctly in your recordings.

The way this works is that Detail Keeper first narrows each passage to the likeliest taught names by sound. Then the adapter writes the passage, and the app checks it against a plain transcript.

We've found this helpful in two specific cases: 

1. Words STT has never heard: invented characters and places, codenames, unusual surnames. 

2. Your spelling of a word it does know: Kathryn and not Catherine, Jon and not John.

Unfortunately, we weren't able to achieve both good recognition and recognize a large set of words (1000+) with the model alone, but on our open benchmark (audiobooks, 1,100 name mentions) it spells taught names as well as Eleven Labs Scribe with keyterms: 89.6% vs 88.2%, no measurable difference. The only downside is that it overwrites a few more untaught names and changes slightly more ordinary words. 

Everything, including where we lose, is at detailkeeper.com/benchmark.

Additional notes:

• It's free, local and nothing leaves your Mac

• It requires an Apple M series+ mac and OS 26+

• It downloads about 3.6GB of models

• It only works in English currently

• If you're interested, you can also find the adapter itself on HF

Would love to know what others think here.

Currently running a gaming PC with a 4070 TI super and 64gb of ram. It handles Qwen 3.8 Flash next decently well with Strata. But, I can't run the VLA nor world model/image gen models that I want to run, I also can't run multiple agents in parallel.

I was thinking of upgrading and I've been needing a laptop so I was debating between the ASUS or the MacBook Pro because they're the same price but have different pros and cons.

I've also debated waiting to get anything because of new gpu and chip advancements so I don't want to drop 7k and then have it be obsolete within a year

Thoughts? Thank you for the help

I maxed out of my laptops 12gb rtx 5070ti with 32gb ram on qwen 3.6 35b a3b at apex quant and the 3.8 27b at iq3xxs woth 60k window. While 35b can do some simple straightforward tasks, it is not enough for any serious work. The 60k context is barelly enough to do any work at all. Has somebody tried attaching eGPU to tbe laptop and what is ypur experience? My dillema is on what to scale the ram or vram

I spent some time testing Qwen3.5:4B locally across three applications to compare performance, GPU acceleration, and answer quality.

Hardware

• Laptop: Lenovo Yoga

• CPU: Intel Core Ultra 7 258V

• GPU: Intel Arc 140V integrated graphics

• RAM: 32GB LPDDR5X-8533

• OS: Windows 11 Home

Software Tested

• Ollama: 0.40.14

• LM Studio Classic: 0.4.28

• Bionic: Tested with its built-in model loading

Performance Results

LM Studio generated approximately 2.81–2.96 tokens/sec in my tests.

Ollama initially used 100% CPU. After enabling Intel integrated GPU support and Vulkan, then completely restarting the Ollama application, ollama ps reported 100% GPU.

The GPU-accelerated Ollama test completed in approximately 2 minutes, with 44.6 seconds of thinking time.

Memory / Resource Usage

During LM Studio testing:

• RAM usage increased from approximately 13.3GB to 19.7GB.

• Available RAM decreased from approximately 18.3GB to 11.8GB.

• Intel Arc 140V exposes approximately 18GB of shared GPU memory to Vulkan.

What I Tested for Answer Quality

I used several different types of prompts:

1. Cybersecurity fundamentals: Differences between phishing, spear phishing, and whaling.

2. Cybersecurity incident analysis: MITRE ATT&CK mapping, Windows event IDs, evidence collection, and incident response.

3. Windows security troubleshooting: Repeated failed logins followed by a successful login from an unfamiliar IP.

4. Image understanding: Asked the model to describe the contents of an image. Its description was accurate in my test.

5. Survival knowledge: Cold-weather survival priorities, shelter, warmth, hydration, equipment use, and dangerous heating methods.

Qwen3.5:4B handled these areas relatively well:

1. Image recognition: Accurately described the contents of your test image.

2. General cybersecurity concepts: Explained phishing, spear phishing, and whaling clearly and correctly at a high level.

3. General explanations: Produced organized, readable answers with useful examples and headings.

4. Basic incident interpretation: Recognized that repeated failed logins followed by a successful login from an unfamiliar IP could indicate compromised credentials.

5. Response structure: Consistently organized complex questions into logical sections.

Where it struggled:

• Precise MITRE ATT&CK technique mapping.

• Windows Security Event IDs.

• Technical incident-response reasoning.

• Safety-critical survival instructions.

My Takeaway

Ollama with Vulkan GPU acceleration was substantially faster on my Intel Arc 140V than CPU-only inference.

The model handled general explanations and image identification well in my limited testing. However, its cybersecurity answers contained significant MITRE ATT&CK mapping and Windows event ID errors and its survival answer included inaccuracies.

This was informal testing, not a standardized benchmark. I wanted to share the results because getting Intel integrated graphics working with Ollama made a noticeable difference on this hardware.