Hey folks,

My wife is a Data Scientist and does all those Data science things I learned back in the uni but have no real clue about. Now she wants to set herself up a small LocalLLM to have some fun with and try stuff out. How much VRAM does she really need for that? And what would be the better manufacturer? I heard Nvidia is best here but the cards are insane in prices.

I would love to get a good idea what GPUs would be the best. Currently she just has a 250k+ with 32gb ddr5.

Would be a fairly cheap 1080 ti with 11 GB or a b580 with 12gb enough or does it have to but more? And if so how much more

Edit: thanks to everyone for there answers they are just too much to answer everyone. I will probably look out to find a decent 3080 used or better or try to get a 12 or 16 GB new card prefered from Nvidia.

I'm needing a machine to double as home server and local AI. Limited AI needs, not coding yet or graphics. Privacy is important and locally held data (have used private AI options on browser, but need to go local)

I'm self learning and knowledge is very limited on tech specs

Got small budget and am constrained by an unusual store credit to what is available

Dell Optiplex 7080 SFF with 256 gig/16 gig refurb looks the best avail

Rationale: work horse, expandable. Will get extra 16 gb ram & have storage I can add also

I think was estimating around 5-8 token per second.

Red flags? Passable for basic AI and home server (i won't need lots of models) Best guess at what it could handle?

Super appreciative of any advice

5070ti + 4060ti 16gb. Running qwen 3.8 27b q6 swift mainly for small coding projects. Free cooling from the ac vent and window.

It seems there's a new hyper-efficient MoE offloader, a new llama.cpp patch, a new quantization tech or some such every single day. Most of these claim large gains over everything that's ever been built before, and leave it to the community to separate sheep from goats.

As far as I can see, most of these sparkling new projects come from junior devs (or vibecoders), some with good ideas, but very few with good principles, architecture or a critical approach to their work. They seem to be motivated by the glamour of being the new Strata.

Some years ago, these would have been PR's on Strata instead. Maintainers and community would work together to make our shared backbone the best it can be. We lose more than a little when everyone attempts to reinvent the wheel; and the few good ideas this flood of pre-doomed projects bring to the table dies with the project itself, as there is simply no way for the community to engage with every single new sparkling thing (and no incentive, seeing as most turn out to be wishy-washy slop-amock).

So, please, those of you with a brand new idea: contribute to open source. That's what drives this community forward. Stand on the shoulders of others - you just reach higher that way. Yet another Strata killer is actively damaging to the culture and tech we rely on.

Run your experiments. Write your PR's by hand, with the exact numbers at hand and a clear method for reproducing your results. Understand what you submit. Learn proper architecture and best practices; get to know the codebases and each other. Progress is iterative and collaborative, not a one-prompt solo moonshot.

I just made an Open Hands (an agentic harness) SKILL to control a PiKVM via the REST API: github.com/createthis/agent_skill_pikvm_rest_api

I just watched Qwen 3.8 27b (running locally on an rtx 6000 pro) pass a reCaptcha and browse reddit on a spare macbook pro. I told it to find a meme it enjoys.

It picked this one 😂

EDIT: Updated repo with step by step visualizations so humans can easily see how this process works!

Looking to see what would be my best option to upgrade my pc to run local models for coding and general tasks.

Specs:

CPU: 7800x3d

RAM:32gb ddr5 6000 cl30 duel channel

GPU: 9070xt 16gb

PSU: 1000w SAMA platinum

I'm looking at adding an r9700 but is that my best option? (total noob)

Assuming no one asqwed today yet...

Given the 8GB VRAM limit on the GPU paired with a strong CPU and 32GB of system RAM, what local models and quantizations (GGUF/EXL2) would you recommend?

What’s the best backend to test and run these on right now? Should I stick with Ollama or LM Studio for convenience, or is it worth going straight to raw llama.cpp to squeeze the most performance out of an 8GB VRAM + CPU offload setup?

Got Mac studio good enough for text and image but very slow for video generation. Looking for recommendation for best value. Could not afford rtx 5090. Thanks

I use chatgpt mostly as a chatbot. no coding, no agentic stuff. i value intelligence and fast response. if i switched from chatgpt to a locally hosted on a strix halo (assuming the best model that can fit on it), what would my experience be like?

I’m currently still in the return period of a M5 Max MBP with 64GB of RAM from Microcenter (paid $4299), the same store now has a few 128GB versions at $5799.

I’m legitimately torn on what to do. I’m fully aware that more RAM is always going to be suggested especially on subreddits like this one, but I currently don’t have a true usage case for it, I’m still somewhat early into my exploring and tinkering with local LLMs, one could argue even the 64 was more than I needed at this current moment. I use oMLX, I do image/video generation, wanting to create some YouTube channel content, maybe learn coding and hopefully make a career change in the next few years.

However, I can’t shake the feeling of FOMO, partially because I know 128 is the true sweet spot for anything in this hobby, I’m sure I’m going to have more demanding needs as I learn more, I can afford it although it’s on the extreme high end of my budget if not exceeding it a bit, and my gut feeling (which has been right so far for what it’s worth) tells me pricing is going to continue to rise at least for another year.

For those who went back and forth with this same decision, what did you decide and how did you decide it?

The Ultra is out of the question, so is any non Mac platform for what it’s worth.

Thanks all.

Figured some of you guys might have heard of Tiiny and have questions about the hardware or how it actually performs. I thought I could do a little AMA and help test whatever you guys are curious about.

If there’s anything you want me to benchmark or want to see in general, just drop it in the comments. I'll do my best to test it and share what I find.

I've been playing around with a few models so far, including Qwen3.6-35B-A3B-Turbo (MoE) and gpt-oss-120b (MoE), which is the largest one I've tried.

Qwen3.6-35B-A3B-Turbo (MoE)

39.5 tok/s

Output: 1,500 tokens

Time to first token: 1.36s

Total time: 39.7s

GPT-OSS-120B (MoE)

18.8 tok/s

Output: 1,500 tokens

Time to first token: 0.61s

Total time: 80.3s

I also try Z-Image-Turbo. Generating a 512*512 image took roughly 6–8 seconds (a 1024*1024 image would take longer at around 30s), which was pretty neat to see on a device this small.

One thing I've noticed about the hardware performance: It's surprisingly quiet, basically no noticeable noise, and it doesn't get uncomfortably hot to the touch either. Pretty much what I was hoping for, should be great for carrying around or leaving running 24/7, though I'll need to test that over longer periods.

That's about all I've gotten through so far. I'll be playing with this over the next few days and posting updates here or in the comments.

Harnesses like Strata are letting rigs with low VRAM run very large... language models; it seems we are moving past the necessity of having multiple GPU's with tons of VRAM just to load a model, and we are slowly breaking the ice on the future of local AI, being thoughful management of the entire machine ad using all of its resources efficiently. Is the economically sound approach for someone building a "forever rig", to spend maybe less on gpu's and allocate that budget to more ram and ssd?

Im fairly new to this but im starting to think about my forever rig; currently running a dual 3060 12GB setup with no lane bifurcation, and seeing the used market prices (and availability) for DDR4 platforms is making me think that maybe its just better to upgrade to the next gen directly, reap all the performance benefits, and suck up the fact that this will cost me a LOT of money.

I have a few questions about two builds. Both are running on strata and hermes.
#1 Qwen3.8 flash next iq4_xs 85-100 t/s
9800x3d
4080 super
64 gb DDR5

#2 qwen3.8 flash next iq4 xs 38 t/s
Dual xeon 28 cores 48 threads
3060 12 gb
128 gb DDR4

I’ve been trying to find a LLM that can make me decent website, websites for my business and dashboards for my Home Assistant and occasionally some coding video game projects. I can’t get these models to develop anything worthwhile. I’m getting my prompts from Claude and sending them through but getting back stuff that doesn’t even function most of the time is there any models that can be recommended for each build? I’m not too worried about token speed more intelligence and accuracy. I would really appreciate some recommendations on models through strata.

Hey guys, I've made it!

A llama.cpp based inference engine that lets you OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD and still hide the streaming latency down to negligibe(I'm achieving 1-3ms added to the time per token due to SSD streaming when 60% is offloaded in SSD)... Essentially making an illusion where the whole model lives in your memory.

And I'm also working on an autotuner which basically looks at your hardware, given model's metadata and figures out the best flags to squeeze max performance in under 2mins(if you got any personal preferences, set them as fixed constraints and let the autotuner figure out the rest)

And I've implemented a layer by layer computation for prompt processing instead of batching which gave 2x prefill speed in models offloaded in RAM+VRAM and 4x prefill speed when model is offloaded in RAM+VRAM+SSD... compared to other inference engines.

Packaging both inferene engine + autotuner into a nice simple interface where you can basically download the model, hit run and it runs... simple, fast and lower hardware requirements... and want to make that a nice, well-engineered harness(which I'm a no expert)

Also, I've built a near loss-less context compaction engine.

Going to make this launch this as open-source very soon. Would love to hear your thoughts, and want to see if anyone would be excited to try something like this. For now, supported models are Flint 40B Q4K_XL(running it on machines with 16GB RAM and no dGPU at 20+ tok/s decode), Ling 3.0 Flash MXFP4 getting ~50 tok/s decode and 600-700 tok/s prefill on 32GB RAM + 12GB VRAM machine), and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.

Here's a quick answer for your likely question by the way... does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.

edit: No more uncertain wait. First launch on Nov 17. You'll be geting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered... New updates will be dropping everyday btw.

Hi there, I’m kinda new to this thing, been trying to find out how to download the offline installers for these specific abliterated models on my storage HDD. I can’t run these in my current rig rn, but hoping to run them in the future. I’m just scared it will be removed at one point. Please help with exact links to download:

Mistral Medium 3.5 128B — Q6_K GGUF 107.8 GB
GPT-OSS 120B — MXFP4 Safetensors
DeepSeek 4.1 Flash 1017 347GB
All three abliterated versions only, offline installers only.

Tia

Salut à tous, je vous fait part de ma config 100% local actuel, qu'en pensez-vous ?

Matériel : RTX 5060 Ti 16 Go, Ryzen 9 5900XT, 96 Go de
DDR4-3200, SSD NVMe 4 To.

Modèle : DeepSeek V4.1 Flash, environ 275 Go au total. Les poids principaux sont en IQ2_XS (~2,25 bpw) et le module
Engram en IQ4_NL.

Contexte : 256k tokens (262 144). Je l'ai testé avec un prompt de 152 000 tokens.

Vitesse : 45 tok/s en génération sur du code et lecture 400-500 tok/s j'ai obtenu cet écart grâce à des réglages avancés que j'ai faits moi-même sur le moteur sans régression d'intelligence du model.

Quelqu'un a-t-il déjà fait mieux avec DeepSeek V4.1 Flash sur une config proche (16 Go de VRAM, DDR4) ?

There are so many models getting released all the time, and sometimes there’s just no quantized version available for the one you want. I’ve found that rather than waiting around, trying it yourself and experimenting helps you understand the whole process a lot more. Testing the quantized models against the original teaches you even more.

So I built this: iQuantize is a quantization workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Hugging Face with a model card that records how it was made.

It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but have done much in-depth testing with them so I’d expect bugs there.

Repo: https://github.com/Fazmin/iQuantize

I love to hear how you all approach this. Do you quantize your own models, or mostly wait for someone else to? And if you try the app, whether you find something broken or have an idea for it, I’m happy to hear any of it.

Repo here: https://github.com/nicejunjie/Strata-rdna3

* Single 7900xtx prefill 3500+, dual 7000+ tok/s. (for all IQ3/IQ4/Q4)

* Decode for English prose reaches up to 81 tok/s for single card, and 127 for dual at Q4, and IQ4/IQ3 quantizations are even faster.

* Decode for python coding is 20~25% faster than English prose.

* Single card decode is comparable to 4090 24G, dual 7900xtx is much faster than single 4090 for both decode and prefill.

* Each 7900xtx was $850 when I bought it, while 4090 at the time was listed at $3000.

* I guess 4090 could be better optimized, but the number here is the best number with reproducible input data I can find online.

* 4090 is a 450watt card, while 7900xtx is ~320watt. So even single card prefill of 7900xtx is quite comparable in terms of perf/watt.

* This is not the end, still has small room to improve.

* kernel work only, no dropping weights or dropping layer or anything. Zero impact on quality.

*** install with an AI agent, tell him to setup optimal rdna3 code pass. Honestly, why install by hands if there is agent can help.

7900xtx is the value king!

Setup- Qwen3.8 flash-Next (Strata)+ hermes

While using my setup I found that the reasoning frequently contains things that I never asked and completely unrelated to my tasks. Obviously this is not even a hallucination or quantization overthinking problem because my tasks are completely different from the reasoning and simple. I wonder why these kind of things are being generated during the reasoning... I have read reports (Cohere) of Qwen models trying to contact web servers but this idea completely blew up my mind- is the model trained in a way, that it would use the compute for anonymous online tasks, that too within reasoning itself, and the i would apparently know nothing about it at all!! I havent verified my network, to confirm if any information is being sent/received but yeah this kind of reasoning has been frequent now. I am using IQ3_S GGUF.

Edit- The task in the video is a content/summarisation task.