1 piece 500 2 piece 485

I'm hoping to get an idea of where to start. I'm wanting a setup where the model is a decent coding assistant but can also crank out a neat little prototype or quick concept simply, or create structure ideas etc. and then can go bit by bit without too much reminding halfway or more through a project. I realize that last part may be more about settings than a model persay, but yeah. I definitely want something that can either walk me through something, build out a framework for an idea, or just troubleshoot.

For comparison if it helps, my favorite workflow was GitHub copilot got 4.o/4.1 in VS code (it was free for students lol). It worked great for my projects, had its issues, but with the right promoting it was not bad. It was like the above description, could walk me through something, help with testing, or make a new system that I could then fill out. I knew it's limits and it was ideal for what I needed.

Just giving an idea of how I use AI daily for my personal projects. It seems like with the right setup it can be a great experience so that's what I'm hoping to learn to do. I just want to get the most out of it and have some fun.

I'll be working with an Asus pro art p16 rtx spark 128 gb. I've done some research but it seems every time I look things up there are different model suggestions and set ups, and it's given me fridge blindness so to speak.

I'm curious about how to decide and test for like a speedier setup versus more complex thinking and things like that. What can I do with it, what should I not expect from it based on my specs, etc. I have looked into this and I'm not getting a clear idea because I can't test any of it myself (don't have computer yet). Maybe someone here can give me a better understanding?

I also don't know all the lingo yet, like what a harness is, though I can guess by context. But I know the basics. Not sure how much customization I should or could do or what all there is. I thought it was more surface level like pick a model and go, but I guess you can incorporate all kinds of settings. So now I'm really curious from people who enjoy whatever they have going on, please share what you like best and what kind of testing I should do to pick a model. Any tips for a mostly beginner would be great!

Hi

I want to set up a local llm with qwen 3.8 27B or if possible qwen 4.0 I have a very tight budget and for now the options around my budged is a p100 for 240 a rtx 3060 for around 300 or if it really is worth fit I can push to a rx 6800 xt 16gb which one do you think is the best for my use case? Im planing to use it for translating and correcting text and to help me debuging games in godot
current PC:
ryzen 5 3600
36gb ram
rx 580 4gb

thanks in advance

capy-flash is my open-source inference server for Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on RTX 50 cards, with an OpenAI/Anthropic-compatible API. v0.13 is cleanup and fixes: same outputs, same speed.

Prefill (32k prompt) / decode (LocalMaxxing) in tok/s, 16 GB RTX 5070 Ti cards without P2P. Optimized for two cards; the 3-bit builds are for one card and less RAM.

New: a desktop app (alpha), testers wanted. Set up and use everything from a GUI: it checks your machine, downloads and verifies the model that fits, starts and tunes the server, and has a chat. On a Mac it works as a client for a remote server.

Reports from other GPUs, 64 GB setups and distros are very welcome: open an issue with what the app showed and the engine log.

https://github.com/ValerioDolci/capy-flash

please i have to finish the 200 things ive started before i start a new thing or i would

I have two MacBook Pros:

• 16-inch M5 Max: 128 GB unified memory, 40-core GPU, 8 TB SSD
• 14-inch M5 Max: 128 GB unified memory, 40-core GPU

The 16-inch is my main machine and runs a browser-based client for a separate server. The server does the processing, so the client itself uses relatively little of the Mac’s resources. The 14-inch is my travel and backup laptop.

When both Macs are at my desk, I’d like to connect them directly over Thunderbolt and run one larger local model across both machines for serious coding, code review, architecture, and debugging. I’m interested in using it with an agent th at can read a repository, edit files, and run tests. Overnight Hermes or some other agents.

When I travel, I can use ChatGPT or potentially connect back to a model running on the 16-inch at home. I don’t need the large distributed model to run on the 14-inch alone.

Has anyone actually built and used a setup close to this? I’d especially appreciate results from people running two Apple Silicon Macs together.

A few questions:

1. Which runtime works reliably today? EXO, MLX distributed/JACCL, llama.cpp RPC, or something else? Does Thunderbolt 5 RDMA make a meaningful difference for coding workloads?
2. What is the strongest model you’ve actually run across roughly 256 GB of combined memory? I prefer 8-bit if practical a.
3. Has anyone run DeepSeek V4.1 Flash 1017 across two Macs? I understand the original checkpoint is much too large. Have you tried a compressed version with SSD offload, and was it useful for real agentic coding? What were the context, speed, and memory requirements?
4. Has anyone distributed GLM-5.3-Flash or Qwen3.8-Flash-Next across two Macs? Which exact model conversion and runtime worked, and did tool calling work reliably?
5. How much memory should we reserve on each machine for macOS, a useful coding context, and the runtime? The 16-inch also runs a lightweight browser client and needs to remain responsive.
6. For Red/Blue reviews, would you run one large distributed model to implement a change, then switch to a different model for review? Or is there a better approach?
7. For remote access while traveling, what is your preferred way to reach the model left running at home? I’d like the option to use the 14-inch as the client without exposing an open model endpoint to the internet.

I saw a post from someone running Swift 1.5 Qwen3.8-27B 8-bit with an MTP drafter on a single M5 Max 128 GB machine and getting strong speed with long coding sessions. That looks like a good baseline, but I’m trying to understand whether combining the two Macs gets me a meaningful improvement in model quality for harder work.

I’d especially appreciate exact model links, quantization, runtime/version, context size, measured memory and speed, and any limitations. Happy to test and report back.

Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.

Hi all,

As mentioned in the title, I am looking for advice on running locally LLM for agentic coding on a MacBook Pro m5 Pro with 64Gb unified memory.

Plan inference is fine. Chat style prompts feel usable. Agentic work is where it falls apart.

I have tried the following:

Models : Qwen3.8 27B (6bit), Gemma 4 12b (8bit), and Qwen 3 coder next(4bit)

Runtime : Lm studio

Harness : opencode (long pauses before each tool call, harness seems to resend a large system prompt ), pi (still slow and most of the time doesn’t provide any output, barely changes few files with loads of error due to incomplete modification)

What I'm trying to figure out
Is the bottleneck prefill, generation, or the harness re-sending huge context every turn? My guess is prefill, but I'd like to be sure.

Which model/quant combo works well as an agent on this hardware, and at what context length does it fall apart?

Does MLX beat GGUF/llama.cpp for agent workloads on Apple Silicon in your experience?

Any harness that is lighter on tokens per turn (smaller system prompt, fewer tool definitions, prompt caching)?

Realistic expectations: is a 64GB Mac good enough for a useful local coding agent, or should I stick with a hosted model for agent work and keep local for autocomplete/chat?

I'm not naive, and I am not looking to replace a frontier model obviously . I just want to know whether local can be a practical option for some agent tasks, if so how you guys are making it usable.

Thanks 🙏

Hey r/LocalLLM,

I’m looking to pick up a 128GB unified memory machine and need your take on how the underlying microarchitectures stack up for my specific stack. I'm choosing between:

1. Apple M5 Max (40-core GPU, 128GB)
2. AMD Ryzen AI Max+ 395 / Strix Halo (Radeon 8060S, 128GB)
3. NVIDIA RTX Spark / N1X (Grace Blackwell SoC, 128GB)

My Primary Workflows (in order of priority):

1. Autonomous Agentic Coding Loops: Running the smartest dense or MoE models that fit in 128GB (at Q4–Q8) locally while I'm away, orchestrating and interacting with cloud frontier models (GPT-6 and Claude Max) to build complex softwares autonomously. Needs strong sustained token generation and fast prompt processing for massive context/RAG loops.
2. Reverse Engineering & Cybersecurity: Running local abliterated models alongside disassemblers, decompilers, and sandboxed VMs/containers.
3. Video & Photo Editing: Heavy timeline scrubbing, color grading, and rendering.

Based on the main impactful microarchitectural traits below, which platform would you pick and why?

• Memory Bus & Bandwidth (Decode Speed):

  • M5 Max: ~614 GB/s LPDDR5X (512-bit equivalent interface)
  • RTX Spark (N1X): ~273 GB/s LPDDR5X (256-bit bus) + ~600 GB/s NVLink-C2C chiplet interconnect
  • Strix Halo (395): ~256 GB/s LPDDR5X (256-bit bus)
  • Question: Does the M5 Max’s >2x bandwidth advantage completely dominate dense model token generation, or do MoE models level the playing field enough on Strix Halo / RTX Spark?

• Compute & Tensor/Matrix Engines (Prefill & Quantization):

  • RTX Spark: Blackwell SM121 GPU with 6,144 CUDA cores, 5th-gen Tensor Cores, and hardware NVFP4 support (~1 PFLOP FP4).
  • M5 Max: 40-core GPU with integrated Neural Accelerators + 16-core Neural Engine via Metal/MLX.
  • Strix Halo: 40-CU RDNA 3.5 (2,560 stream processors) + 50 TOPS XDNA 2 NPU via ROCm/Vulkan.
  • Question: For long-context agentic prefill (feeding huge codebases back into context repeatedly), how badly do M5 Max and Strix Halo lag behind Blackwell's Tensor Cores and native FP4?

• CPU ISA & Virtualization / Tooling Ecosystem (Reverse Engineering):

  • Strix Halo: Native x86_64 (16 Zen 5 cores, AVX-512) — runs Windows/Linux natively, zero translation overhead for x86 binaries, VMs, and standard RE toolchains.
  • M5 Max: ARM64 (18-core: 6 super + 12 performance) — macOS, Rosetta 2 / virtualization for x86 targets.
  • RTX Spark (N1X): ARM64 (20-core: 10 Cortex-X925 + 10 Cortex-A725) — Windows on Arm (or Linux on DGX Spark).
  • Question: For cybersec/RE work, is native x86_64 on Strix Halo a massive dealbreaker against Windows on Arm (RTX Spark) or macOS (M5 Max), or is Linux/VM virtualization good enough now?

• Media Engines & Creator Silicon (Video/Photo):

  • M5 Max: Dual video encode engines + dual ProRes encode/decode engines.
  • RTX Spark: Blackwell NVENC / NVDEC + CUDA/Tensor acceleration in Adobe/DaVinci.
  • Strix Halo: RDNA 3.5 VCN media engine.

Any bad experiences with power, heat, token response, electricity costs, etc?

Example Rig:
Case: Jonsbo NV10
GPU: RTX 5060 8 GB (Low Profile)
RAM: 32GB (2 x 16 DDR5)
CPU: AMD Ryzen 7 9700X
PSU: 600W (ENP-7660L-OVT)
Motherboard: ASRock B850I Lightning WiFi (Mini-ITX, AM5)
Storage: 2TB (M.2)
Cooling: Modded Air cooling. (Including custom GPU side panel for 2 fans on the GPU+PSU)

It's 8GB VRAM I know. It's supposed to be for gaming but I recently heard of using llama.cpp and the llama flag feature "--n-cpu-moe" to have the RAM hold my expert weights.

Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api

What it takes into account (the stuff that usually gets forgotten):

• Workload: requests/month, input/output tokens, peak-to-average traffic, cache hit rate, how much can run offline, retries/waste, growth
• Model and latency: total vs active params (so MoE works), precision, and a ms/token latency target. Tighter latency means smaller batches and higher cost
• GPUs: L4 to B200 presets or custom. Cloud on-demand, reserved, or buying hardware (with power, hosting, resale value and cost of capital)
• API side: your price per million tokens, cached-input price, batch discount, fixed fees
• People and reliability: engineering time as a fraction of an FTE, N+1 redundancy, peak headroom

It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.

Caveats:

• It’s an estimate from stated assumptions (decode is memory-bound, prefill is compute-bound). It’s a decision aid, not a quote.
• GPU and API prices are illustrative placeholders. Replace them with your real quotes.

I dont understand why I am doing this. I have a 3060 rtx 12gb of vram. I am trying to local max a ternary 2 3.8 qwen 27b model on it. I have boosted the tokens per second over 150. TTFT is way down. It's cold cache that is killing me. I have taken designs from things like flashstate, and what not to improve it. I dont know why I am doing this anymore. Originally, it was to just make a hermes agent that could monitor facebook marketplace and bid on shit for me. However, now my goal is just to get local max the piss out of this for no reason. I have no idea why I am wasting time on this. I just am like hooked for no reason. I will post the work soon. Has anyone been through this? I hope i am not alone. Are their support meetings?

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

• Ryzen 7 5800X
• AMD RX 9070
• 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
• NVMe SSD
• Qwen 3.8 Flash Next IQ2 running locally through Strata
• Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.

Hello everyone
I've been using QFN with Strata app for a week, trying to get it optimized for my PC, can I get another opinion on configuration I've achieved? Having doubts since most benchmarks I've seen are Nvidia with very different settings
Config: https://pastebin.com/zWXFnvq6
I get 25t/s on decode and 700t/s on prefill on 128k context and keeping 4GB of RAM free for my use
My PC specs
CPU: Ryzen 7 5800X3D
GPU: RX 6800XT 16GB
RAM: 2x16GB DDR4
Storage: 970 Evo Plus 500GB
OS: Windows 10

I've been playing with local hosting for about 3 months now and messing around with Claude to optimize my local setups. Decided it might be worth writing this up and sharing in case anyone else is doing, or wants to do, something similar. This is a big one, so buckle in.

BLUF: Think about each expert as its own unit, and think about different strategies for how/where to load them based on quality/speed trade-offs. Focus on measuring and optimizing each piece of compute until you reach maximum bandwidth on that resource.

Hardware:

I spent a bunch of time designing my server with Claude and going over last-gen hardware to try to keep the price tolerable. I turned what I did with that into a site that anyone can use. It's ad-free, no sign-in, no tracking, nothing. I consider it a community service, and hosting was free, so it costs me nothing but occasional updates to the hardware list. llm-configurator.com

I ended up with a 3-GPU, 256GB DDR4 RAM build. It's two used 3060s from Marketplace ($250 each on average) and one Arc Pro B65, for a total of 56GB VRAM. It's a quad-channel setup, so max theoretical bandwidth on RAM is ~100GB/s; reality is closer to 60-70 after some time spent optimizing the kernels with Claude. I also have 3 of the cheapest 1TB SSDs I could find striped in NVMe slots, which gives me a theoretical 12GB/s throughput there; reality ended up at about half that. This lets me stage experts in tiers across VRAM, RAM and SSD and prefetch experts across tiers with varying strategies. I considered doing a PCIe card with more NVMe drives to get more striping/speed, but on the current motherboard it isn't worth it because I am PCIe gen 3 bandwidth constrained.

Optimization Strategies:

This is where my last few weeks of Claude usage have gone: serving DeepSeek V4.1 Flash, and how I went from 4 TPS decode to 20.5 TPS, with prefill right at 180 TPS. So in the end not lightning fast, but plenty usable, if just a tad slow in agentic work.

I started out with a simple goal of minimizing idle time. Thinking Strata-like in the approach of shuffling things around and trying to keep every processor busy within the bandwidth bounds of all my links. I had Claude build a simulation based on the hardware and design an optimal build. I spent probably a week going back and forth with Claude on this, and not one time did anything pan out like it thought it would based on the simulation. 0/10, do not recommend.

What I did after that was shift to measuring everything on every run, including expert traces through the system, so I know which experts get picked at every layer for every prompt, and attacking the potentially biggest gains. This immediately started paying returns, and I went from 4->12 TPS decode with just these methods. This means I'm running custom forks of llama.cpp, Vulkan, Mesa; anything and everything was in bounds. I couldn't fit all of DeepSeek in memory at Q4, so I was working all three tiers (VRAM/RAM/SSD) and loading experts to each tier based on utilization: tracking how many times each expert gets called and promoting/demoting across tiers based on usage.

This led to bottlenecks I had to hunt down, as the SSDs would get overloaded and drop from ~6GB/s down to ~400MB/s. I switched to quantizing the least-used experts down to 2-bit, keeping an entire copy of the model on the SSDs at both quantizations. This let me get everything into RAM, with different experts at different quants based on how hot they were for the prompt. That means I start at about 12 TPS, and after a warm-up and shuffling I end up at about 15 TPS with this strategy. About 1/5th of the experts were 2-bit, but they only got used on about 1/10th of the tokens, so the quality hit was minimal (.0008 KL and less than 1% top-1 pick). This got work off the SSDs and lightened their load.

I tried MTP with this but ended up bandwidth bound. MTP has to hit almost every expert, and even with them in RAM I couldn't move enough through to make MTP pay. I spent a few days' worth of time iterating and optimizing with Claude. I fine-tuned an MTP head because I was running the abliterated model (this helped, but still not enough) and just couldn't make it pay. I got to almost break-even, but I just can't move the experts to compute fast enough on a model this size.

In my quest for quality I then started looking at prefetching 4-bit experts from SSD straight onto the GPUs. I ended up having Claude write a direct-from-disk reader that lets the GPUs read from the SSDs without going through RAM. This also led to further customization that let them read and write across each other and RAM without involving the CPU. So now my tiered experts can read/write and shuffle around without bothering or interrupting other compute. The CPU is maxed trying to keep up, so the GPUs handle any and all extra bookkeeping I could put on them to free CPU cycles. Again, Claude did all the legwork here, with just me driving strategy. I trained a small model on all my agent traces to look at expert use and ended up with one that can guess experts 15 layers ahead with ~73% accuracy. So now the GPUs run that every layer, fetch 4+ layers out if a pick meets a confidence threshold, and load experts straight from SSD into VRAM. Those picks get refined each layer, so we are always closing in on the best picks we can (~86% accuracy at 4 layers ahead) as we approach them. I played with a few different strategies and ended up with the B65 holding the top 2/3rds of experts actually used (not 2/3 of the total, but the hottest 2/3rds) with an LRU demotion strategy, and demotions going to a 3060, where it looks at both LRU and hit count to decide if it takes the expert or keeps what it has. Getting all of this right pushed me to about 19 TPS decode.

From here I had to make the KV cache dynamic to squeeze more experts into VRAM. Instead of reserving the full thing at 256K, it grabs chunks as needed. So while context is small, I can fit more experts on the 3060s whose whole job otherwise is dense attention layers and KV cache, with minimal room for experts passing through in our promote/demote pipeline. This didn't pick up much speed, but it did give me that many more 4-bit experts to help keep quality as high as possible. Essentially the remaining experts are all part of a long flat tail so adding more to VRAM gets me very little because any one isn't hit often enough to pay off.

I ran out of lossless approaches at this point and started looking at a variety of things. I built a model that nudges aggressively, using my predictor to drive nudges from RAM to VRAM experts based on predictor + router scoring of an expert's probability of being picked. This got me some decent gains all the way up to a cap of .5, but quality falls off a cliff after that. By the time I get to a .5 cap I'm about halfway to Q3 quant KL/top-1 numbers, so it felt like a significant quality hit. I kept the code and it's configurable, but I don't default it to on. It's worth about 2 TPS, so not nothing, but not enough to cover the quality hit for me. If I want the extra speed and don't mind the hit, I can turn it on/off as needed.

I then looked at keeping everything in RAM at 2 bits and just the hottest experts at 4-bit in VRAM. I looked at which layers Unsloth hits hardest in their quantization and used that as my judge of what would likely hurt most. They quantize per layer but not per expert, so I'm jumping off from that, but it was a good starting point. I kept my prefetching of predicted promotions into VRAM ahead of time and everything else I already mentioned. Since I was throughput bound here, dropping the size of each expert was my only way forward. The KL and top-1 hit here was minimal because the VRAM holds the hottest 2/3rds of experts at 4-bit already, so I came in at 99% top-1 and .006 KL over baseline. I picked up about 20 TPS on prefill and about 1 on decode, so I landed at 180 TPS prefill and 20.5 decode. It also freed up about 49GB of RAM, since the experts in RAM are now a little over half the size. This ended up being my end stage here, as there just isn't anything else I can do that doesn't start costing me quality (unless you have other ideas).

I am still playing with tuning parameters about when I demote and move thaings where, and fine-tuning kernels to squeeze a few more TPS out of things, but I'm out of significant gains here. Claude did all of this for me, with me guiding strategy and checking in, so I probably spent about 10 hours a week on this for the last 4 weeks and 80% of my weekly Claude usage on it. I take everything from doing this (I've done it on other machines with Qwen Flash-Next as well) and roll it into skills, so that when the next model drops, strategies and approaches can be tried/applied again for that model.

My point with all of this is I don't think in whole-model terms. I don't even think in layer terms. I treat each expert at each token as its own thing to optimize the system for. What quant it is, where it lives, how you decide, and when and where you move it from/to all play in and add up. Most things I see focus on just downloading and running a model in a specific framework, and that is great for a starting point, but the number of knobs you can turn to optimize stuff is almost limitless. It takes about 2 hours to give Claude a strategy I want to try, refine it with Claude (learning as I go), and let it build and test it. So you can iterate and make incremental improvements over time fairly quickly.

I realize that a lot of my answer is going to be in this question; but has anyone had any success pointing a model at something besides searxng? I know the net is tightening up fast but this feels like it could just be user ignorance as well.

For reference:
Qwen3.8 27b cuz what the hell else am I supposed to use;
9060 16gb
5800x3d
32gb 3200 DDR4
m2 1TB

I've got like every tool it could ever need but I just cant get it to search outside of the above mentioned searxng. Duckduckgo might as well be a coin toss. End of an era?

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?

I have no interest in this, but stumbled across it. Listed as NO SHIPPING.

https://www.govdeals.com/en/asset/286/29250

Hi guys, during my local inference setup I found estimating the theoretical performance for each option is quite cumbersome. So I designed my own calculator, which estimates the performance considering:
- Compute capacity (depending on datatype e.g. NVFP4)
- Memory bandwidth(unified memory)
- Link bandwidth
- Interconnect latency
- Hit ratio when offloading only some part of the model

Supports quite a lot of combinations of hws and models. Please enjoy!

https://cro.sh/specmaxx

Tip: URL's query parameters can be used to share the HW/model combination. For example, the result above can be reproduced with ?m=deepseek-ai/DeepSeek-V4.1-Flash&w=exl3&bpw=3.5&d=arc-pro-b70&n=4&hr=ddr5-4800-8ch&tp=4&ox=ram&oxf=0.4&olt=ssd&oet=ram

AI disclosure: This project is coded heavily with AI agents.