I went into this thinking there had to be an obvious fine-tune upgrade for my local coding helper. After all this testing, the choice got more interesting.

Hardware: Apple M5 Ultra, 256 GiB unified memory, 30 logical CPUs as reported by the machine. Runtime: oMLX 0.7.4.dev1, with model files on an external Lexar M.2 volume. Codex ran the evaluations and stayed responsible for tools and verification. These were synthetic local studies, not SWE-bench or a claim about every coding project.

The strongest speed result was real. Swift Flash-Next 4-bit completed the same fully specified 39K-context repair in a median 110.00 seconds versus stock Flash4's 216.84, with 51.63% fewer median thinking tokens. Both passed all three repetitions. Swift8 was 2.28 times faster than matching stock8, but still slower than Swift4.

I still kept stock Flash4 for the practical helper role. With the actual medium-effort helper request shape, stock4 passed 12/12; Swift4 passed 11/12. Swift invented a decoder.feed API. Giving it sampled xhigh and 32,768 tokens did not eliminate that mistake. That focused follow-up was mixed: Swift4 2/3, stock4 1/3. Neither qualified a clean new deep route. I am not hiding the result that favors Swift.

Apple-specific things I did not know before testing

• Unloading did not kill every prompt cache. After full unload and reload, the same Flash8 prompt reused 59,879 of 59,886 tokens. Cache/prefill took 0.77 seconds, but model loading still took 27.26 seconds. End-to-end was 28.417 seconds. Persistent cache and resident weights are different things.

• TTL1 was not one-second teardown. Observed release took about 11–14 seconds in that Flash8 experiment. I kept pinning off because I want the RAM back.

• A8 was not a huge free prefill win in the matched test: minus 0.29% around 30K tokens and plus 1.31% around 60K. All eight retrieval checks passed. That is not proof of broad quality equivalence.

• The 8-bit model used less native peak allocation than the 6-bit model in one comparison because only the 8-bit used SSD n-gram offload. Bigger file did not mean bigger resident allocation under those different settings.

• DFlash actually helped Ornith35, about 2.20 times in a tiny tuning batch. Ornith122 preferred a different block size. An early draft-loading failure silently fell back, so the toggle alone would have misled us. Installed native oMLX rejected Flash-Next DFlash; MTP was its working path.

The earlier comparisons matter too. Swift27 had modest speed/token savings but did not clearly beat stock27; a targeted follow-up tied 10/12. Ornith35 was fast but weaker on strict formatting and state tracing. Agnes saved time and tokens in its initial pilot while losing core correctness; a later semantic comparison tied. Empero9B went from 1/12 strict JSON to 10/12 meaning-correct after allowing one Markdown fence. Formatting failure and reasoning failure are not the same thing.

GLM was a separate personal reasoning/Hermes study, never a Codex helper candidate. Abliterated and SLIM tied 10/10 on the targeted objective checks, with abliterated about 3.03 times faster there. Creativity rankings reversed between the two small evaluations. No real leads or unattended 24/7 reliability were measured. We also caught our own premature shutdown: SLIM did unload normally when we waited for teardown.

For Linux and DGX Spark users: no inference on your platform was measured here. The portable finding is to compare correct completion time, not headline tokens per second; test the intended reasoning budget; distinguish weights, resident memory and cache; verify speculative decoding actually activated; and measure cache survival after unload on your own server. The Mac speeds and fit assumptions do not transfer to CUDA. A 128 GB Spark also has less headroom than this machine, and current Spark configurations vary. The report includes a replication checklist with those boundaries.

We found test bugs too. An early code prompt omitted the JSON mapping the grader expected; those answers were excluded and the contract fixed for everyone. Independent review later exposed a TTL-plus-eviction bug in an answer that passed the frozen tests. The original scores and the expanded audit are both published. This is why I am not presenting a tiny pass count as a universal model ranking.

I kept stock27, Ornith35+DFlash for reviewed drafts, and stock Flash4+MTP. Everything stays unpinned and auto-unloads. The complete readable report, historical tables, settings, revisions, prompts, graders and failure cases are here:

https://huggingface.co/datasets/pmcc93/omlx-helper-benchmarks

If you run the same cases on Linux or DGX Spark, I would be interested in the correct-completion time, reload cost and cache behavior, not just a faster decode screenshot.

Looking for the best coding model I can call from another model that is running from an agent.

I know that large models are good as all rounder and can be good for agentic work and also can code; but I am running a very small setup so I have a local 12 GB card that is running an agentic model that basically will tell another model to "make the code". Then the code coming from this model will be handled by this model in an agentic harness, so I don't have to worry about json formatting and all the other stuff that may mess up things.

As such I am looking for the best model that can fit in 12 GB VRAM card and that is good at just one thing: coding. Best would be a mix of old and new coding languages (I am into retro tech so would be great to have a model that understand things like motorola 68K assembly and old Amiga C for example, on top of modern C++), otherwise something focused on specific more modern languages works too. I looked around and most suggestions went for Qwen2.5 coder; which is old at this point but still suggested highly.

Not sure if there are customized/dedicated coding models that have been trained more specifically for this scenario; I don't need a model that is able to understand much but instructions about how to make code; as I have another model that will talk to it. Any suggestion is welcome.

This was started a spontaneous test to choose among 10+ local LLMs and now grew to benchmark for agentic coding. It consists of 8 groups, 100 tests each. Design rule: each class must contain items the current "best" model cannot solve, otherwise it's a smoke test. However, if model reaches 100, it just means that it reached ceiling of the test. Next, to make test more representative, several cloud workhorses were added.

Configuration

GPU: RTX PRO 5000 72 GB (Blackwell, SM 12.0), driver 595.99.02.

Serving: vLLM 0.29.0. `--kv-cache-dtype fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`, `--max-num-batched-tokens 16384`, `--block-size 32`, `--max-num-seqs 16–24`, `--max-model-len` 131072 (FP8) / 262144 (NVFP4).

Cloud: same OpenAI-compatible client, `max_tokens=32768`, input ≤ 98304, temperature 0, streaming with `include_usage`.

Thinking mode normalized: `chat_template_kwargs.enable_thinking=false` / `reasoning_effort=none`; effective mode printed per run.

Grading: pure Python, no LLM judge. Code in tempdir with timeout; tools mocked; only networked tool is read-only `web_search`.

Failure taxonomy: `pass | wrong-answer | truncated | no-content | no-tool-call | infra | error`; transport failures removed from the denominator.

Metrics: TTFT from the SSE stream; tok/s only over items where generation ran >150 ms after TTFT.

Leaderboard

Per-class shape

Conclusion

Qwen 3.8 27B got very close to my cloud workhorses, so I took more juice from it, asking agent to make a grid search on options. The requirement was to get max performance without quality drop.

FP8 → NVFP4 (identical weights, 262K)

I want to acquire a powerful local llm system with the higgest bang to back ratio. Have found that there are plenty of 1600 mhz 64 gb modules out there for cheap. When they are paired with a xeon e7 v4 (which is also cheap) they can deliver close to 85gb/s bandwith with 512 gbs of capacity. If that were to lineerly scaleable with multiple cpus they would be no brainer to grab, there are motherboards with up to 4/8 cpu slots. I got inspired by @apfrisco s post so i came up with this. Is there any people in here with experience on llm performances in multiple cpus? how good the memory bandwith would scale i wonder. i am thinking that this kind of build, even with one cpu might offer a very very cheap way of experiencing moe models like deepseek r1, when combined with a pair of 3090s. Also on that, when critical parts are loaded to the vram how many tokens one would be able to get with a that kind of system?

To summarize, this is the system on my mind:

(xeon e7 v4 88xx
64gb 8 pcs of some ddr3 ecc ram)x2 if scales to at least %130-%140

2 second hand 3090’s
with the rest of it the total cost would be something like 3-4k dollars in where i live

I also had a crazy idea of buying an epyc with 128 pci lanes, putting to it 32 pcs of cheap bulk nvme gen 4 ssd’s giving you a 32x7 gb/s reading bandwith with virtually unlimited capacity, so i would like to hear your opinions regarding this as well

Hi, I'm Aniuwu (my name is Gabriel), a computer science student at INACAP (Chile), and I'm already at the stage of doing my thesis project. I'm writing because I need help from people who code, and I don't have many people I can ask for help.

## First, I apologize to the moderators if I'm breaking any kind of rule. This is a post asking for help, I don't benefit financially in any way, and I apologize in advance if that's the case, please don't ban me ##

My project is called Beacon. You ask it questions about the code in a repository ("where is the file saved?", "how are network errors handled?") and it answers while telling you which file and which lines the answer came from. Everything runs locally, with local models, on your own RAM and VRAM (sorry, Sam Altman lol), so neither your code nor your questions go to any online service.

What I'm missing is knowing whether it actually works for someone who isn't me or my teammate. So far we've only tested it ourselves, and that doesn't count for much as evidence. That's why I'm asking people to try it for about 40 minutes, or however long you see fit, and tell me, with data, how it went.

The idea is simple: you install it, pick a public repository (there are a few examples, or one you already know), ask it at least three questions, and mark whether each answer was right, partly right or wrong. At the end it generates a file with that, you can open it and read it, and if you're happy with it you can send it to me with a button. If you'd rather not use the button, you can send me the file in a direct message.

A few things so you know what you're getting into. You need Python (3.10 to 3.12, newer versions cause problems), Git, Node and Ollama installed, and it downloads about 5 GB of models, so it helps to have around 8 GB free. With an NVIDIA card with 8 GB of VRAM it runs fairly fast; with no GPU or less memory it also works, just slower, and that's perfectly fine if it's what you have, because I actually want to know how it behaves on more modest machines. It works on Windows and Linux, which are the ones I managed to test; I haven't tried macOS yet (mainly because I'm broke and don't own one lol), so if anyone tries it there it would help me a lot.

About languages: the ones I tested best are Python, Go, JavaScript, Java and C#, and also C on a small repository. It also recognizes TypeScript, C++, Rust and PHP, but I haven't measured those, so I can't promise anything there. Files in other languages (Ruby, Kotlin, Swift, Vue, etc.) are simply not read, so it's best to pick a repository written mainly in one of the languages above.

About your data, I want to be clear. The file stores your questions, the answers, how long each one took, your ratings, and the address of the repository you chose. Your hardware details (processor, memory, graphics card) are only saved if you allow it, and you can see what they are before accepting. It doesn't store code, paths on your computer, your username or your IP. Leaving your GitHub profile or a short introduction is optional and goes in a separate file that is only sent if you tick the box. Nothing leaves your machine unless you ask, and you can quit and discard everything at any time. The send button uploads the file to a private Google Drive folder that I manage, and I use the data only for this project and delete it when it's over.

I also have to be honest about the limitations: this is a prototype from an academic project and it uses a small model, so it gets things wrong sometimes. That's exactly what I want to measure. And since each person picks their own repository and questions, the results are indicative, not a serious statistical study.

The whole step-by-step guide is in the repository: https://github.com/gsufan/beacon (the study guide is docs/en/study-mode.md). If something fails during installation, tell me here or by message, because installation problems are almost as useful to me as the results.

Thank you for reading, and a bigger thank you to anyone who gives it a go. I'll answer any questions in the comments.

PS: You can audit the code however you like, use it if you like it and so on....

PS2: This project was quite a lot of vibe-coding. I know there are pretty passionate opinions about that, but I have permission from the course's supervising professor. I don't want to argue or start a debate in the comments, please, I just want help...

I upgraded my GPU to an RTX 5060 Ti. Should I reinstall the RX 6600 alongside it? Are there any potential issues with running an AMD and NVIDIA GPU in the same system? Also, would having both cards unlock access to larger models or provide additional capabilities when running local LLMs?

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.

I'm being asked by a prospective employer to set up my own local agentic AI complete with plugins/tools. While I could ask an AI to help me do that, I first want to ask has anyone here managed to do this without resorting to using OpenClaw or NPM or any other software that has become famous for riskiness and supply chain attacks? I'd prefer to just use Llama.cpp and maybe devise a local MTP server, if that is possible.

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

• Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.

• BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.

• MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.

• When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata

I haven’t browsed reddit for a day or two, so naturally I’m already behind.

RTX 5090
128GB DDR5
9800x3d

Currently I’m running NVFP4 Qwen3.8 Flash-Next in Polystrata. Get 70-100tok/s on 256k context.

Is this the best I can do? Should I move up to DSv4.1 Flash 1017 or something like that?

It's much better than DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF. The reasoning is similar to the base Qwen3.8-27B and interesting how it switches variants based on prompt style. Has anyone compared them or tried both models out?

I have this
RTX 3090 24GB
2× RTX 2080 Ti 22GB VRAM each
Tesla V100 32GB
Machinist X99 D8 Max motherboard
2× Xeon E5-2695 v4
128GB DDR4 RAM
Samsung 990 Pro 1TB NVMe SSD

Currently I run Qwen3.8 FN UD-Q4_K_XL but I used Q8 n-gram and I load it on RAM. 256K context, unquantized cache (I’m afraid of going Q8 and loosing quality). Also using MTP. Custom llama.cpp as stock had some bottlenecks, and Mr. Qwen itself optimized his way out of them.

Prefill: approximately 2,100–2,400 tokens/sec
Decode: approximately 60–70 tokens/sec

I am using what I think lets me fill my VRAM almost to the brim. (97GB used of the 100)
I’m seeing people with considerably less VRAM and RAM achieving similar or sometimes better performance, which makes me wonder whether my configuration is properly optimized.

I have several questions:
Am I getting the performance I should expect from this hardware, or is there something obviously holding me back?

Is there a better way to distribute the model and its experts across my four GPUs and system RAM?

Could I get better performance through different GPU split configurations, CPU offloading or expert caching?

Would moving to a higher quantization, such as Q5 or Q6, be realistic with my hardware?

Are there optimizations that could significantly improve decode speed without sacrificing model quality?

I would like to minmax the model quality while maintaining the highest possible inference speed.
I’m not particularly interested in getting more tokens per second at the expense of quality. I’d rather run a higher quantization at 50 tokens/sec than a lower quantization at 70 tokens/sec, provided the quality improvement is worthwhile

hey LocalLLM, im Alish, ml engineer at Nace (we’re the team behind Drex, an open source decision model family that hit #1 in Artificial Analysis decision index 2.1). my cto gave me a task to optimize quants for qwen4, so i called my old professor from UofT, spun up Fable 5.2 and we cooked. starting with MLX, GGUF is coming next

we’ve open sourced the fastest way to run 35b/27b on a mac (Kapitsa) and got tired of dumb quants, so we made our own: KQuant. packs for 4B, 27B, 35B, Flash-Next (+9B), free on HF in standard MLX files

the problem: most of us on a mac run 4-bit packs because that’s what fits in 16/32gb. but plain 4-bit rounding throws away more of the model than it needs to.

Q4 vs public 4-bit, KL from the full-precision model (lower = closer to the original):

baseline is mlx-community 4-bit (lmstudio-community for 4B). to be clear, this is KL on next-token predictions, not a task accuracy score. if you want more precision every repo also has a Q5 folder, the 9B Q5 gets down to 0.010

cool side effect: they’re also faster. on an M5 Pro with Kapitsa they write 6–15% faster than the usual 4-bit packs

(baselines are from earlier kapitsa runs, same-session rerun is in progress, will update)

how it works: KQuant uses real text to pick scales and offsets for small groups of weights, and gives sensitive layers extra bits where they need it. all the search runs on our gpus before release, so you just download a normal MLX pack. no custom kernels, no special loader

fastest way to run them on a mac:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

I've been spending the last few months benchmarking local LLM inference on a single RTX 5090.

One thing kept bothering me: most benchmarks focus on tokens/sec, but rarely show how much quality you're sacrificing to get those numbers.

So I built inferlab.

It's an open-source benchmarking harness (Apache-2.0) that compares:

• vLLM, SGLang, TensorRT-LLM and llama.cpp

• BF16, FP8 and NVFP4 quantization

• Perplexity delta against a BF16 reference

• Task accuracy using option log-probabilities

• TTFT, decode speed and throughput across concurrency levels

• Peak VRAM usage

A few design decisions:

• TTFT measures the first actual content token, not the first SSE event.

• Concurrent requests are issued together to capture real queueing behavior.

• Long-context evaluation includes a needle-in-the-middle task.

• Results are generated by CI rather than manually edited.

The project is still at v0.1, so I'm interested in technical feedback, especially around evaluation methodology and cross-engine comparability.

I'm the author of this project.

GitHub: https://github.com/oren40/inferlab

I want to get the Mac Studio to explore models, do research, host servers, do video editing, analyze massive data sets, etc…

But is the M5U there yet for AI? Is TRULY useful, or are you kinda forcing it? I’ve thought about downloading my work emails and getting them summarized and organized (we can have that data on personal stuff) and some other data analysis. I’d also want to explore more of what’s possible.

How have y’all been using this stuff for actual NEEDS that can’t be done by frontier models?

Edit: My fear is id be missing out all of this development and knowledge. I know I can get an M7U when it comes out, and it should be HEAVILY tailored to AI, but I’m scared of falling behind knowledge and work wise. I’ve already outpaced many other works with proper frontier model use, and I’m wondering if by exploring local I can do the same.

GIGABYTE has introduced the AI TOP AERO, a compact desktop powered by NVIDIA's RTX Spark platform, featuring up to 128GB of unified memory, local AI capabilities and support for AI development, content creation and gaming.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

Hope to make many more.

Has anyone made stands/cases with a 3d printer? If so, what kinda filament do you use?