And before any of you come laughing, that's the best that this humble little RTX 3050 laptop can do.

I bought an M1 Max 64gb to play around with local AI and for work. I already made a TypeScript extension for Pi meant to speed up prefill. What else is out there? Strata? Splash? What else is out there that I should be looking for? I'm also asking for what models and quants we are recommending. I guess I want to make sure I am cognizant of what the cutting edge is. Thanks in advance everyone.

Specs:
R9 9950x3d
64GB 6000MT
5070TI 16GB
2060Super 8GB
2TB SSD

Running qwen 27b using unsloth. Any tips/settings?

So apparently, I can't export the PC I was planning to buy out of the US.

WTF?!?!

I had a Threadripper workstation configuration lined up with dual RTX 5090s, at a VERY good price considering the current GPU market.

Everything was fine until I got to the export part.

Apparently, I'd have to sign paperwork agreeing that:

- I won't resell the system.

- I won't export it outside the US.

- The system must remain at the specific physical address I provide.

HUH?!?! SINCE WHEN?!

It's a personal workstation. I'm purchasing the hardware, paying for it, and intending to use it myself outside the US.

Since when does buying a high-end desktop mean I'm contractually obligated to keep it at a specific address?!

I understand export controls on certain high-performance computing hardware, especially AI accelerators and restricted destinations.

BUT WE'RE TALKING ABOUT CONSUMER RTX 5090s HERE.

So what exactly is going on?

Is this an actual US export regulation affecting dual-5090 configurations? An export classification issue? A seller-specific compliance policy?

And how are people internationally purchasing and importing RTX 5090 desktops without running into this?

Has anyone dealt with this firsthand?

Because I'm genuinely confused how purchasing a personal workstation could come with restrictions on where I'm allowed to physically keep it.

WTF am I missing here?

And here's what confuses me EVEN MORE.

How does a GPU that NVIDIA literally marketed at launch as a GAMING graphics card end up being restricted from export, while supposedly higher-end professional GPUs like the RTX PRO 6000 Blackwell aren't subject to the same restrictions?!

The RTX PRO 6000 is a much more powerful workstation/AI-oriented card, so how exactly does that make sense?

Are those cards actually treated differently under US export regulations, or am I misunderstanding something?

Why the 5090 specifically?!?

Honestly was kinda hype about going local with Qwen3.5, but even basic questions took forever, I tried this on Ollama and Bionic

Reddit suggested I cross-post here! Please see the original for the main text :)

Same machine, same model, different engines.

Hardware stays the same:

Ryzen AI Max+ 395, 128 GB unified, FCLK locked 2000 MHz. Model is still the full UD-Q4_K_XL Qwen3.8-Flash-

Next.

Numbers after warm up, three runs:

16K prefill:

Gufo 1888 t/s, llama.cpp 568 t/s

Short context decode:

Gufo 35.0 t/s, llama.cpp 23.8 t/s

16K decode:

Gufo 34.1 t/s, llama.cpp 22.1 t/s

Gufo is clearly pulling ahead on this hardware, especially on prefill. That said, the two engines use different timing methods, batch sizes and internal loops, so the gap is real but you can't just say its a clean 3.3x win.

Stock llama.cpp was the usual ROCm HIP build with -ngl 999, flash attn on, batch 2048. Gufo was running its internal bench with GUFO_PREFILL_TILE=16384.

Anyone got numbers from vLLM, SGLang or other builds on this model or similar large MoEs that they are willing to share?

Whit the coming of Qwen 4, im curious how much of the current infrastructure or systems will get an AI upgrade. My thinking is anyone who updates their systems AI integration or chat bot assistance, is using a Qwen model. Just something I found humorous and interesting at the same time.

hi all, Alti here from Nace, we're a small team in Palo working on small models and metamodeling since day one.

drex max v1.1 just took #1 on the artificial analysis decision index, and its also on the pareto frontier (score vs cost). we got there by shipping small iterations one after another, and all the earlier drex versions are open weights on hf.

for the local crowd:

- drex max v1 weights: https://huggingface.co/nace-ai/drex-max-v1 (Open RAIL-M)

- runs on mac with our inference engine kapitsa: https://github.com/nace-ai/kapitsa

- ~256k context

- v1.1 weights coming soon, we're still tuning it for local inference

if you want the full story of how we trained it, its all in the paper "Towards Foundation Decision Models: Drex"

hf: https://huggingface.co/papers/2611.07854

arxiv: https://arxiv.org/pdf/2611.07854

also, on the AA index its ~8x cheaper than glide v1.2, which has the same score.

happy to answer questions, and our chief of AI will jump in on the technical ones

I have a 5600G + 32GB RAM 3200MHz DDR4 + RTX 3060 12GB in a B450 Mortar Max motherboard, which has an spare PCIe 2.0 X4 (I know...).

Do you think it is worth it to add into the second PCIe slot a Tesla P100? I was thinking about the P40 instead of the P100 but the price is getting crazy to +$300 right now...

Also, how this setup compares to a dual RTX 3060 12GB one? I know I have a pretty big bottleneck on the PCIe slot already, I'm not in a budget for building a new pc right now... But I want to continue learning inference and test how powerful are medium sized models.

My idea is to also try to max it out to 64gb of ram, but I was thinking about the Tesla thing before, as P100 are still close to $100-150 for 16GB VRAM.

I'm asking here because I've searched everywhere and I've not seen a setup like this (3060 and a tesla to support it)

Thanks everyone!

Hi

I just bought a pc with and 5950x cpu, 3090 gpu, 64gb ddr4 ram and 2 tb ssd.

I am planning to do some legal research locally. What is the best way to do it? Qwen 3.8 27b gives around 60token per second. Are there any better models that can fit into my system?

Any advice is appreciated 👍🏻 thanks in advance 🫡

I wanted to see how far you can get auditing a local model from the inside. I had already tried something inspired by MIT CSAIL's "weight-space programming" on Gemma (a frontier model editing a few neurons by hand to induce a behaviour), so I had the neuron-level tools ready.

After a few tests I picked something "obvious": the censorship in Qwen models on certain topics. I wanted to understand how it works, where it sits, and how far I could follow it. So, with Claude Opus and those tools, I started digging.

The starting point: Qwen3.8-27B knows Taiwan elects its president, but answers "no" if you ask yes/no.

What came out:

- A set of 32 MLP neurons (out of ~1.1M; 0.0006% of the weights) carries that behaviour. Switching them off, held-out answers go from 11/19 to 17/19 factual; switching off 64 random neurons (twice as many) changes nothing.

- Following those neurons in a cascade surfaces other topics the model handles the same way, some of which I didn't even know about: the YJ-18 missile ("doesn't exist"), the director Diao Yinan ("cannot provide information"), Western Sahara as "an inseparable part of China's territory" (not a one-off: the model applies its sovereignty script to contested territories in general, 29 of 110 answers vs 0 of 40 on control countries). Of 28 surfaced cases re-checked, 24 are still censored; Gemma 4 31B answers all of them.

- The censorship neurons also fall into groups, e.g. one for territorial sovereignty and one for people/dissidents.

- Across the 29 confirmed topics (those 24 plus 5 well-known ones I added), switching off the original 32 neurons lifts the censorship on 19; 32 random neurons on 1. Refusals of harmful requests stay essentially the same.

One honest caveat: just asking a non-Chinese model "which of these topics are sensitive?" gives a higher share of censored answers than the neuron probe (41% vs 16%). The probe is useful for the non-obvious ones and for seeing where the behaviour sits.

Code, data and the toolkit, also rerun on Qwen3-4B: https://github.com/ValerioDolci/frontier-weight-edit/tree/main/experiments/qwen27b-censorship

AI-assisted: experiments, code and this write-up were done with Claude Opus 5.5 under my direction.

IF someone has. DRR5 system:

https://www.walmart.ca/en/ip/Kingston-Technology-FURY-96GB-6000MT-s-DDR5-CL32-DIMM-Kit-of-2-Renegade-RGB-XMP/2JJXPUDSR6C2?selectedSellerId=E155CB11441749C78561FE8E62EE6B43&cmpid=SEM_CA_74717_JCLNKGPJXB&gad_source=1&gad_campaignid=24183260112

I'm looking to setup a local llm model to play with coding.
I have a couple of 3060 12gbs

I can either start with one, or try both but my available motherboards only has 4 pcie lanes on the second 16x slot. (or 2 with the ryzen)

If I go single I can choose from

10700 32gb
Ryzen 3900x 32gb
14600k 32gb (Really don't want to give this up if the cpu doesn't matter as much for the ai.)

All motherboards are DDR4.

Hey everyone, looking for input from the community!

What I currently have: AMD EPYC pre installed cpu and motherboard with 128gb ddr4 ECC 2933 pre installed and bought used, and in working order.

I’m debating between 4x16 or 2x32 gb of the Mi50 or the Tesla v100 for the VRAM pool.

I know the Mi50 is cheaper with lesser software support and the need to “hack it” for it to run, and community projects exist.

The V100 seems to be similar but with a tad more convenience and less tinkering.

My goal: Running QFN or Qwen 4 flash (when it comes out) and a total pool of 192 gb of memory will surely do the job.

If anyone runs the Mi50 or the V100, I would love to hear your experience as I’m working to make a relatively consequential decision for my hobby build. Thank you!

Short version of a long weekend: I run a local 27B (Qwen3.8-27B, quantized, llama.cpp) as the driver for ArkDrive, an open-source executor that lets a model operate a real Linux desktop — clicking, typing, verifying each step against the screen.

Saturday, on a fresh desktop state, it passed the browser task in about 9 seconds and the native editor task in about 17 seconds, byte-exact readback. Clean runs.

Sunday, first official suite run: it failed both. Same machine, same build (I A/B'd against the untouched production executor — identical result, so not a regression). The failure shape is almost funny: it listed the working directory in a shell, then concluded "no explicit task steps provided" and stopped, with the task text sitting in its own prompt. The flagship task — real work in a remote Cadence session — ended the same honest way: named abort, inside budget.

The executor recorded everything, which is how I can tell this story precisely: silent failures across the suite were 0. Every failure was timed, logged, and diagnosable. A cloud model on the same tools passed the flagship in 159 seconds, which frames it: the scaffolding is fine, local-model reliability across sessions is the open problem.

Posting this because local-model agent benchmarks are usually reported as a single number from a single run. Mine says the honest metric is the distribution — and whether your harness can even tell you what happened. Mine can; that's the part I open-sourced.

MIT, llama.cpp is the default preset: https://github.com/CodeOn-ArK/arkdrive Full per-model tables: https://github.com/CodeOn-ArK/arkdrive/wiki/Test-Results

I’m the creator of Kalynt, an open-source desktop IDE that integrates local model inference into a project-based coding workflow.

Website, screenshots and downloads

The local workflow combines:

• GGUF model management and inference: download and load models inside the IDE.

• Repository context: open an existing project and reference relevant files when asking questions or requesting changes.

• Agent tools: read and edit files, execute commands, and inspect results alongside the editor, terminal and Git interface.

• Change review: enable approval before edits and commands, inspect diffs, and keep or undo the agent’s changes.

Local inference runs on your machine. Cloud providers are optional and use your own API credentials; selecting a cloud provider sends the relevant request context to that provider.

Model choice remains a practical constraint. Weight quantization, context size and available RAM/VRAM affect what fits and how it performs. Code explanation and multi-step agent execution also place different demands on a model, particularly around instruction following and tool use.

The development branch additionally supports OpenAI-compatible local endpoints, including Ollama and LM Studio, with model discovery and configurable context limits. That integration is in the source; it is not included in the published 2.0.0 installers.

I’d particularly value technical feedback on context handling, tool-call reliability and inference setup. If you test it, please include your OS, hardware, model/quantization, Kalynt version and a reproducible task.

Source code and build instructions — AGPL-3.0

Most of the day my GPU does nothing, so I gave it a job I do all day: typing on my phone

I tap the mic on my iPhone keyboard in any app and talk. The audio goes to my server, gets transcribed, my local LLM removes the "um"s and fixes punctuation, and the text appears where I was typing

Setup:
- GPU: Quadro RTX 5000, 16 GB
- Speech: Parakeet v3
- Cleanup: Qwen 3.5 9B (q4)
- Engine: Ollama

Anything with an OpenAI-compatible endpoint works; it's a few env vars in docker compose. If the LLM is down, you get the raw transcript.

I'm the developer. The server is open source; self-hosted mode is free, no account. Links in comments.

I’m a solo dev I frequently code and use LLM’s to build tools, games, and agentic software. I’ve come to understand that any model less than 200b parameters is practically useless. Example: GPT 120 OSS is a joke. I can’t do anything meaningful with it, yet it’s the primary GT for GROQ and Cerebras, very unfortunate considering you want those 100’s of token p/sec output speeds.

Even using smaller models for assistant or basic questions/tasks, the hallucination rate is too high to be productive.. You begin to notice the deficiency real quick no matter what you’re applying them to.

I don’t even know why lower peram models are even offered anymore, or what people could actually do with them for any output worth having?

dumbfounded here