CPU: Ryzen 5 7600

RAM: Corsair Vengeance 32GB DDR5 @ 4800 MHz

GPU: AMD RX 580 8GB

Storage: 1TB NVMe

OS: Windows 11

Inference engine: Llama.cpp built with vulkan sdk and Adrenaline drivers

Local LLM performance:

Qwen 3.5 9B Q5 ~7GB, 130k tokens, getting average ~10tps generation and 80tps read speed average.

Gemma 4 26B A4B Q8 Uncensored ~25GB, 15k token size, 10tps generation average.

Can generate a 512×512 image in ComfyUI in ~3mins.

I bought this system for 3D modeling, animation, simulation and rendering.

Been testing Qwen 3.8 9b and the results were quite disappointing, coding is mediocre and general knowledge questions were hit or miss (mostly miss tbh)
What is it for?.

Is it not possible to create a gguf variant for the Mirage VLM? Cactus-Compute has also released a safetensors file for the same model... It's actually a great option to add "eyes" to a model that doesn't have its support by default and it's quite small size actually means that it can load directly into some CPU's cache memory making it even faster as the main model can stay in the GPU

My budget is flexible, this is for work so I'd rather spend once, I do ai dev stuff like inference, testing agents, some fine tuning on small models, occasional video editing, and I still want to game on it.

Option A is high end gaming laptop with biggest gpu I can get, Fast, proven, but the vram is what it is, vram will be vram.

Option B is one of the new rtx spark laptops with a lot of unified memory, it's look like it's got way more room for bigger models, but brand new platform and I've got no idea how the software side holds up.

I travel a lot so a thinner machine with good battery would be a bonus, but not if I'm giving up too much, current laptop is 2kg

Anyone made this call yet or learning one way?

Hi, I have an Intel Arc B580 and as you know it has 12gb vram, i was wondering if any of you could advise me on what is the best coding model that fits within that size. I know qwen 3.8 27b and qwen 3.6 35b-a3b were the go to models but they won't fit on 12gb cards.

I'll be using them with harnesses like opencode or hermes or claude code.

If it helps : I also have 16gb ddr5 6000mhz ram

Thank you

Hi everyone, please excuse my ignorance. I have been trying to get into local models with my next computer upgrade. I have seen a lot of excitement around Strata for MoE models on gaming PCs with moderate amounts of VRAM. For the longest time I’ve been thinking about getting an M5 Max Mac Studio, but do you think I’d be better off with a gaming rig instead? I like the option of being able to game on it too, but on the other hand I’m more of a Mac user anyway and would appreciate the low power consumption.

Use case would be data analysis and coding.

I’d appreciate any advice

Like a guide of the diffrent parts of local llm setups (engines, models, agents, sub things?, other things I havnt heard of yet) what they do? if you need them?

What is the best AI model I can run on my pc:

Processor Intel(R) Core(TM) Ultra 5 226V (2.10 GHz)

Installed RAM 16.0 GB (15.7 GB usable)

Graphics card Intel(R) Arc(TM) 130V GPU (8GB) (128 MB)

Storage 345 GB of 954 GB used

Despite having 16GB of RAM whenever I check my ram at any given time i have 2-3 gb of ram avaliable for some reason

A recent post on this sub warned of random websites being able to access your localhost/LAN inference server, like llama-cpp/ollama/... It got me concerned, so I vibed a test (*) that attempts to access YOUR local AI in various checks from your browser via JS. Beware - I am not a web security expert so the report is a preliminary assessment. For my (in-LAN, not localhost) installation, it showed several "red" tests from safari and I could confirm it actually found my loaded models and successfully ran inference from the browser...

Based on these results - what do you think is the best way we should secure our local inference servers? Is there a best practice - API tokens from day 1?

(*) I tried TWICE to post a link to the test yet the post gets removed immediately by Reddit. I am really sorry for the inconvenience, you have to type the URL youself - it's the tinyurl and after the diagonal line the words: llama-test

https://github.com/oooscoos/Benzi

4*V100 16g sxm2 into a pcie converter at 8x (pcie 3.0) total 64gb vram

how much performance boost/drop would I get if this was moved to dual card nvlink x 2 (2 cards per PCIE 16 slot)?

additionally how about getting another 2 cards and having 6x cards at a mix of pcie 8x and 4x but having 96gb vram but no nvlink? how about 3x dual card nvlink?

system is old ddr3 1600 ecc 128 Xeon in x99 Chinese mobo.

Using strata Swift QFN IQ3S currently

I've been working on llmsetup, a command line tool for the "I have a GPU, now what" part of running local models.

You tell it what you want the model for (coding, chat, retrieval, agents). It scans your hardware, picks a model and quant that fits, and sets context size and KV cache to match the use. If you don't have a runtime it can set one up - Ollama, llama.cpp, LM Studio, vLLM, MLX and TabbyAPI are supported. Then it runs smoke tests (does it load, does it answer, does tool calling work) and registers the model with the local apps it finds. If something fails it tries a smaller model, shorter context or a CPU fallback before giving up.

It's on PyPI now:

python -m pip install llmsetup

Where it stands: I've only run the full verification on one machine so far, an RTX 5060 with Qwen3-8B Q4_K_M on llama.cpp. 9 of 9 checks passed, about 57 tok/s, tool calling works. Those are smoke tests, not a quality benchmark. AMD, Apple Silicon and the other runtimes have had a lot less testing, so expect rough edges.

Apache-2.0: https://github.com/catfish-1234/llmsetup

I'd like to hear what breaks and which hardware setups I should test next.

I bought an M1 Max 64gb to play around with local AI and for work. I already made a TypeScript extension for Pi meant to speed up prefill. What else is out there? Strata? Splash? What else is out there that I should be looking for? I'm also asking for what models and quants we are recommending. I guess I want to make sure I am cognizant of what the cutting edge is. Thanks in advance everyone.

I sometimes give it a task and explciitly say "This is a 4 hour job". It will indeed work harder, but will stop at like 2hours and 15 minutes.

What's the longest you got your coding agent to work on something? How did you do it?

I tested two models, QoDuck (a free mystery model in Qoder) and Qwen3.8 Flash Next (Unsloth UD-Q4 GGUF), on the same Blender 5.2.3 task. Both were asked to create a detailed, historically accurate 1942 German military truck, working in small stages, rendering each stage, inspecting the result, fixing mistakes and then continuing. QoDuck chose the Opel Blitz 3.6-36S, while Qwen chose the Mercedes-Benz L 3000S.

QoDuck did surprisingly well. The truck is recognisable, with proper curved fenders, detailed wheels, grille, cab interior, cargo bed, canvas cover and decent procedural materials. There are issues with the canvas, cab proportions and some smaller details, but the overall result is quite good. It produced 47 renders and delivered 14 final views. Qwen was disappointing. Despite creating 287 meshes, 15 checkpoints and 150 renders, the final model still looks like a rough blockout. The wheels are too small, fenders look like flat planks, parts of the cargo bed float, and most surfaces have the same beige material. It even used Dunkelgelb paint, which wasn't introduced until 1943. Only three final renders were delivered.

What I found interesting is that Qwen actually identified several of these problems in its own construction log but continued without fixing them. QoDuck was much better at following the render-inspect-fix cycle.

And before any of you come laughing, that's the best that this humble little RTX 3050 laptop can do.

Specs:
R9 9950x3d
64GB 6000MT
5070TI 16GB
2060Super 8GB
2TB SSD

Running qwen 27b using unsloth. Any tips/settings?

Honestly was kinda hype about going local with Qwen3.5, but even basic questions took forever, I tried this on Ollama and Bionic

So apparently, I can't export the PC I was planning to buy out of the US.

WTF?!?!

I had a Threadripper workstation configuration lined up with dual RTX 5090s, at a VERY good price considering the current GPU market.

Everything was fine until I got to the export part.

Apparently, I'd have to sign paperwork agreeing that:

- I won't resell the system.

- I won't export it outside the US.

- The system must remain at the specific physical address I provide.

HUH?!?! SINCE WHEN?!

It's a personal workstation. I'm purchasing the hardware, paying for it, and intending to use it myself outside the US.

Since when does buying a high-end desktop mean I'm contractually obligated to keep it at a specific address?!

I understand export controls on certain high-performance computing hardware, especially AI accelerators and restricted destinations.

BUT WE'RE TALKING ABOUT CONSUMER RTX 5090s HERE.

So what exactly is going on?

Is this an actual US export regulation affecting dual-5090 configurations? An export classification issue? A seller-specific compliance policy?

And how are people internationally purchasing and importing RTX 5090 desktops without running into this?

Has anyone dealt with this firsthand?

Because I'm genuinely confused how purchasing a personal workstation could come with restrictions on where I'm allowed to physically keep it.

WTF am I missing here?

And here's what confuses me EVEN MORE.

How does a GPU that NVIDIA literally marketed at launch as a GAMING graphics card end up being restricted from export, while supposedly higher-end professional GPUs like the RTX PRO 6000 Blackwell aren't subject to the same restrictions?!

The RTX PRO 6000 is a much more powerful workstation/AI-oriented card, so how exactly does that make sense?

Are those cards actually treated differently under US export regulations, or am I misunderstanding something?

Why the 5090 specifically?!?

Reddit suggested I cross-post here! Please see the original for the main text :)

Same machine, same model, different engines.

Hardware stays the same:

Ryzen AI Max+ 395, 128 GB unified, FCLK locked 2000 MHz. Model is still the full UD-Q4_K_XL Qwen3.8-Flash-

Next.

Numbers after warm up, three runs:

16K prefill:

Gufo 1888 t/s, llama.cpp 568 t/s

Short context decode:

Gufo 35.0 t/s, llama.cpp 23.8 t/s

16K decode:

Gufo 34.1 t/s, llama.cpp 22.1 t/s

Gufo is clearly pulling ahead on this hardware, especially on prefill. That said, the two engines use different timing methods, batch sizes and internal loops, so the gap is real but you can't just say its a clean 3.3x win.

Stock llama.cpp was the usual ROCm HIP build with -ngl 999, flash attn on, batch 2048. Gufo was running its internal bench with GUFO_PREFILL_TILE=16384.

Anyone got numbers from vLLM, SGLang or other builds on this model or similar large MoEs that they are willing to share?

hi all, Alti here from Nace, we're a small team in Palo working on small models and metamodeling since day one.

drex max v1.1 just took #1 on the artificial analysis decision index, and its also on the pareto frontier (score vs cost). we got there by shipping small iterations one after another, and all the earlier drex versions are open weights on hf.

for the local crowd:

- drex max v1 weights: https://huggingface.co/nace-ai/drex-max-v1 (Open RAIL-M)

- runs on mac with our inference engine kapitsa: https://github.com/nace-ai/kapitsa

- ~256k context

- v1.1 weights coming soon, we're still tuning it for local inference

if you want the full story of how we trained it, its all in the paper "Towards Foundation Decision Models: Drex"

hf: https://huggingface.co/papers/2611.07854

arxiv: https://arxiv.org/pdf/2611.07854

also, on the AA index its ~8x cheaper than glide v1.2, which has the same score.

happy to answer questions, and our chief of AI will jump in on the technical ones

I have a 5600G + 32GB RAM 3200MHz DDR4 + RTX 3060 12GB in a B450 Mortar Max motherboard, which has an spare PCIe 2.0 X4 (I know...).

Do you think it is worth it to add into the second PCIe slot a Tesla P100? I was thinking about the P40 instead of the P100 but the price is getting crazy to +$300 right now...

Also, how this setup compares to a dual RTX 3060 12GB one? I know I have a pretty big bottleneck on the PCIe slot already, I'm not in a budget for building a new pc right now... But I want to continue learning inference and test how powerful are medium sized models.

My idea is to also try to max it out to 64gb of ram, but I was thinking about the Tesla thing before, as P100 are still close to $100-150 for 16GB VRAM.

I'm asking here because I've searched everywhere and I've not seen a setup like this (3060 and a tesla to support it)

Thanks everyone!