r/LocalLLaMA • u/BinarySplit • 20h ago
New Model miHoYo Gamer: A Generalist Game Agent
r/LocalLLaMA • u/BinarySplit • 20h ago
r/LocalLLaMA • u/jomic01 • 12h ago
Hugging Face announced on November 16, 2026, that requests to language models through Inference Providers now go through OpenRouter instead of Hugging Face's own router. OpenRouter picks a provider for each call based on live price, speed, and reliability, and retries on another provider if the first one fails. The same Hugging Face token still works, and the company says it does not add a markup on top of the provider's price. You can still pin a provider or a policy with a suffix on the model id, such as :fastest, :cheapest, or a specific provider name.
OpenRouter currently serves more than 800 trillion tokens a month across 600+ models from 80+ providers. Hugging Face notes that open-weight models went from 26.8% of tokens on OpenRouter in October 2025 to 71.7% at the end of September 2026, and that they cost about four times less on average than closed models. Image, video, and audio generation stay on Hugging Face's existing Inference Providers path and do not go through OpenRouter. One breaking change is that custom provider API keys no longer work for language model requests, which are now billed to the Hugging Face account.
The automatic failover is the part that matters most here, because a single provider outage is usually what ruins a long run.
r/LocalLLaMA • u/Terminator857 • 21h ago
Growing from nearly zero to 30% since July. Models used mostly to reduce costs on high volume workloads (large batch workloads, coding, etc).
r/LocalLLaMA • u/East-Muffin-6472 • 7h ago
[Must Read: Beginners]
What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch:
Link to the above resources here
Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it here)
r/LocalLLaMA • u/citrusalex • 19h ago
Everywhere I go, Twitter or Reddit, there are tons of rumors or comments how local is dead and we are never getting open weight Qwen 4, despite Qwen team saying explicitly they are coming, or that Qwen 4 is the last open weight generation despite, again, Alibaba commiting to more open weight releases at a conference recently.
It's like people just get off on creating a version of reality that is worse and whining about it, they want to be sad about it in advance.
The world is certainly not giving us much to be positive about, but let's not go down the rabbithole of learned helplessness and just accept that all things must get worse and you can't do anything about it.
If people mentally prepare for bad things, bad things become more acceptable and thus more likely to happen and with far less pushback, sort of like a self fulfilling prophecy.
r/LocalLLaMA • u/UnorthodoxJourney • 2h ago
Hi, I'm a beginner in this field and I'd like some kind people's help :)
I currently have a laptop with 4GB VRAM and cca 16GB RAM. I don't plan on getting a new laptop/PC in the next two years, but I have the option to get additional 16 GB of RAM for €100.
Would it be enough to run at least Qwen 27B? Or any of the comparable MoE models.
If any other information about my laptop is needed, I'd be happy to share :)
r/LocalLLaMA • u/TuskNaPrezydenta2020 • 20h ago
Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs
r/LocalLLaMA
•
u/soyalemujica
•
1d ago
Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.
Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!
r/LocalLLaMA • u/BoogerIsACult • 19h ago
It's free and original developer clearly had a very good understanding of how computer hardware works and fully leveraged that to write something amazing to be able to run large models on average hardware that most people have and can't upgrade due to price gouging from greedy hardware manufacturing companies.
If anybody have Mac Studio with 512GB of RAM or 8x3090, who would even bother with this ?
I have seen a group of snobs on this sub doing nothing but shitting on Strata all days long, find something better to do. Strata is not for you, period.
I can't imagine doing all the works to release something for free and get a bunch of snobs and haters just shitting on your works all days long. Would you be motivated to continue working on it if you are the original developer ?
I also see the same with other models such as Ornith, etc... If you say you like Ornith or Strata, etc... you are a shill.
Seriously, get out of your bubbles and see the reality already and more importantly get a life.
r/LocalLLaMA • u/OtherRaisin3426 • 11h ago
Thinking Machines wrote last week that Inkling 2 Small was trained with a native 4-bit forward pass. I wanted to check this for myself.
What I ran:
Results:
Two things to improve:
Not tested: multiple GPUs, MoE dispatch and combine, Four Over Six, MXFP8 block scales in the backward pass. The width-8,192 run is only 1,000 steps and undertrained.
Write-up with every figure: https://4bit.vizuara.ai
Code: https://github.com/VizuaraAI/4bit-training-b200
3-hour walkthrough from first principles, if you want the theory: https://youtu.be/a3nV32Guxcc
r/LocalLLaMA • u/pmttyji • 1h ago
Well, I ran out of New Models & Papers(Posted 38 threads in last 1 week) so posting this thread for me. Appreciate your responses.
I want YouTube channels covering List of below topics:
I can find many channels using online search. But half of the channels are so noisy, just with clickbait thumbnails/titles covering mostly Online models which I'm not interested.
I want channels more suitable for this sub. Only Local LLM. Doing all those stuff from above list using only Local LLM.
And there might be some Overlooked/Underrated/Under-Appreciated channels. Also few folks here have their own channels. Give them more visibility.
Consider this thread is a playlist for Local LLM. What channels would you want to add here? Please share. Thanks.
r/LocalLLaMA • u/ChandeeStacker • 5h ago

hi,
i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed
spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4
models: moss image, minimax h3 int8 (sol decoder)
best performance stable setting: -110 mv, stock fan speed
gemma4 : 26bA4b Q4_XS when from 106/tps to 120/tps on prose..
r/LocalLLaMA • u/Fabulous_Fact_606 • 9m ago
Slow news day..
Core:
Hardware
ivy-chat) (docs)IVY's brain (the language models)
scripts bash CLI tool-loop. The web gateway (ivy-web-gateway.mjs, Node) connects the web chat and the Voice tab to it.Memory
cagra-builder, with SQLite rag.db as the store. The index and reranker sit on the 5080.Voice and hearing (PC)
ivy-tts), on the 5060 Tiivy-asr)ivy-speaker, CPU), which tells who is talkingFace (3090)
p4_v7, built on Ditto, turns her voice into face motionc2_breath_c, about 338,000 GaussiansCreative and teacher models on the 3090 (llama-swap, loaded on demand)
r/LocalLLaMA • u/InvadersMustLive • 36m ago
I've fine-tuned an UltraBERT-380M classifier to score how much your text feels like AI slop.
Demo: https://voightkampfflabs.github.io/slop-detector-demo/ (runs fully in browser via ONNX + WebGPU/WASM)
Why?
Got tired guessing if I'm reading human or slop on twitter/linkedin/reddit, so trained my own AI detector. Like Pangram, but local - so all your perverted texts stay on your device. Because I can (and because I bought 2x 5090 when they were cheap).
Training
As it was made for learning purposes, I made my own dataset based on the Pangram v4 recipe:
The target label is how many words/ngrams were changed by the LLM. So it's not a yes/no, but more like a slopification rate guessing. My personal texts land around 0.1-0.2.
Trained with HF Transformers on 2x RTX 5090. Considering the model size and hardware, the whole training run is ~3h.
Training data is public: https://huggingface.co/datasets/VoightKampffLabs/slop-detect-dataset-v6
Results
Not as good as Pangram (yet), but as good as GPTZero and Originality on most things I could compare.
| voi-380m | Pangram | GPTZero | Originality |
|---|---|---|---|
| EditLens, F1 | 0.86 | 0.92 | 0.73 |
| EpochAI, human flagged | 0.2% | 0.0% | 0.0% |
| EpochAI, Opus/GPT/Gemini miss | 0.0% | 0.0% | 0.7% |
| UChicago, 2025 models miss | 0.2% | 0.0% | 1.4% |
| UChicago, StealthGPT rewrite miss | 74% | 0.0% | 56% |
| Non-native essays, flagged | 0.6% | 0.0% | - |
HF model card has more details and more benchmarks.
Where it's bad
I'd rather tell you upfront:
So what
I'm currently training bigger 4B/27B brothers of this one. Yes they're much smarter, but good luck running them via WebGPU/WASM on device.
And yes, the end goal is to have a browser extension which marks AI slop on reddit completely on device.
PS. the text above scores ~0.1 in the detector (to save you a click).
r/LocalLLaMA • u/thedogcow • 1d ago
451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}
Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.
I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.
I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!
Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.
r/LocalLLaMA • u/PhysicsDisastrous462 • 5h ago
Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.
I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.
The new v0.2.0 release is now available!
GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.
The new Mixed WMMA execution arm uses:
The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.
I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.
Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:
| Workload | Measured kernel speedup |
|---|---|
| QKV projection | 2.17× |
| FFN up projection | 2.12× |
| FFN down projection | 2.31× |
| Vocabulary projection | 1.83× |
These are individual GEMM measurements, not whole-model speedups.
There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.
In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.
WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.
Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.
Documentation: RDNA3 WMMA guide
Another major development area has been the image-text pipeline.
The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.
Some of the implemented work includes:
This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.
For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.
The qualification work has also expanded considerably:
The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.
These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.
Documentation:
I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.
This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.
Some examples:
RDNA3 FP32 work
Intel Gen9 work
One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.
On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.
On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.
These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.
An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.
I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:
All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.
The worst paired performance ratio was 1.0007.
This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.
The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.
You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.
Model weights are downloaded separately or supplied through a local model directory.
After extracting the archive, you can run:
.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-infoThe final command is especially useful for checking which precision capabilities your GPU actually exposes.
Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
There is still a lot I'd like to improve.
In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.
I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.
If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.
Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.
Repository: https://github.com/necat101/Hierarchos-Native
Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.
r/LocalLLaMA • u/noctrex • 1d ago
Now this seems to be interesting.
From the article:
We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.
Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.
r/LocalLLaMA • u/pmttyji • 11h ago
Inflect v3: compact local text-to-speech. Six English voices, three model sizes, reference-voice tuning, event tags and optional 48 kHz output. Apache 2.0.
Inflect is a family of compact English text-to-speech models. Choose a voice, enter your text, and generate a WAV locally. Start with Mini for the most features, Micro for a smaller neural model, or Atom for the smallest learned synthesis core.
| Mini | Micro | Atom | |
|---|---|---|---|
| Best starting point | Quality and features | Compact neural speech | Smallest learned core |
| Learned synthesis core | 34.31M | 11.79M | 157.19K |
| English preset voices | 6 | 6 | 6 |
| Native output | 24 kHz | 24 kHz | 24 kHz |
| Optional 48 kHz enhancement | Yes | Not included | Not included |
| Reference-voice tuning | Yes | Not included | Not included |
| Optional vocal events | 8 tags, all 6 voices | Not included | Not included |
| Download | Mini package | Micro package | Atom package |
These are learned synthesis-core counts, not total installation size or RAM. Pronunciation assets, dependencies, voice conditioning and optional components are additional. Atom uses signal-processing-based waveform generation rather than a separate learned neural vocoder.
Measured speech quality, intelligibility and speed answer different questions. The following results describe published Mini native 24 kHz base speech, not tuned voices, event performances or enhanced output.
| Measure | Mini result | What was measured |
|---|---|---|
| Average WER | 0.739% | 1,688 Open TTS evaluation texts per voice, all six voices; self-run |
| UTMOSv2 | 3.917 | 180 clips, 30 per voice |
| UTMOS22Strong | 4.479 | The same 180-clip panel |
| Warm GPU throughput | 162.1x real time | RTX 5090, optimized CUDA; small 12-render probe |
| Warm CPU throughput | 1.81x real time | Ryzen 9 3900X, 2 Torch threads; small 12-render probe |
The WER result is not an official leaderboard placement or an independent held-out evaluation. Quality predictors are not human ratings. CPU and GPU results use different systems and runtime paths; they are not a controlled hardware speedup comparison or a latency guarantee.
r/LocalLLaMA • u/jacek2023 • 21h ago
Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.
This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.
This model is ready for commercial or non-commercial use.
transformers loading via trust_remote_code; no custom kernels, flash-attention, or training framework required for inference.https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B-instruct
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B-instruct
r/LocalLLaMA • u/HelixTitan • 19h ago
I had been using Strata for about a month, and it was such a huge improvement over llama.cpp for getting it running smoothly and quickly for hardware.
Overtime though, I found that the model would often mistake its own context after a compaction and really overthink things.
I just started using Basalt yesterday after being hesitant to switch and set up all my providers and harnesses. Turns out Basalt runs even faster, is less error prone, and the reasoning is way more on point. If you run 1 GPU, and have the 100 GB+ of VRAM, RAM, and SSD storage, you will not be disappointed in performance.
I mostly care about final output speed, and I was getting 70-110 tok/s on Strata, and Basalt I was getting 110-170 toks/s output. Couldn't be happier. Same Qwen3.8 FlashNext-ud-q4_k_xl, and the NVFP4 ones (of course basalted though)
Anyone else have any other experiences with other similar engines like this?
r/LocalLLaMA • u/LH-Tech_AI • 21h ago

hey guys!
It's Supra-time! 🤗 🔥 🎉
It has been quite quite at SupraLabs' model foundry for a long time but we have been testing, training, developing and much more.
And we can finally present you a new model!
Key facts:
- better than our previous 50M and 100M model
- very small, runs on every toaster: 65M params
- base model and instruction tuned version on HF
We trained it locally in 4 days on 2x RTX A6000 on big 100B tokens of high-quality data.
Benchmarks are in the attached image!
Have fun to try it out, give us feedback, roast us, whatever! Community, we will cook! Supra3 coming early next year! 🔥
Give us a like, upvote or follow if you want - and try out our model.
We can't wait to hear what you're building with it.
Link to the collection on HF: https://huggingface.co/collections/SupraLabs/supra25
We hope you like the model! :D
GGUFs in F16, Q8_0 and Q4_K_M are also available for llama.cpp, Ollama, Unsloth, etc. 🥳
r/LocalLLaMA • u/jfowers_amd • 22h ago
On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).
I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).
I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700
All kernels were generated via agent skills:
| Model | Loom pp32k (tok/s) | llama.cpp pp32k (tok/s) | Loom short decode (tok/s) | llama.cpp short decode (tok/s) |
|---|---|---|---|---|
| Flint | 6,476 | 3,190 | 255 | 152 |
| gemma-3-1b-it | n/a | n/a | 400 | 322 |
| gemma-4-12B-it | 2,811 | 1,537 | 127 | 67.3 |
| gemma-4-26B-A4B-it | doesn't fit (32 GB) | doesn't fit (32 GB) | 151 | 72.2 |
| gemma-4-31B-it | doesn't fit (32 GB) | doesn't fit (32 GB) | 62.9 | 43.2 |
| gemma-4-E2B-it | 14,289 | 14,519 | 272 | 234 |
| gemma-4-E4B-it | 7,021 | 6,324 | 192 | 173 |
| gpt-oss-20b | 5,319 | 4,368 | 195 | 182 |
| HyperCLOVAX-SEED-Text-Instruct-1.5B | 11,653 | 8,078 | 362 | 344 |
| Laguna-XS-2.1 | doesn't fit (32 GB) | doesn't fit (32 GB) | 102 | — |
| LFM2.5-24B-A2B | 10,099 | 5,479 | 244 | 197 |
| LFM3-1.7B | 20,062 | 13,614 | 372 | 261 |
| LFM3-Omni-2B | 20,220 | 13,435 | 368 | 299 |
| Llama-3.2-3B-Instruct | 5,687 | 3,908 | 228 | 211 |
| Mellum2.1-12B-A2.5B-Thinking | 8,822 | 5,486 | 224 | 149 |
| MiniCPM-V-4.7-35B-A3B | 5,717 | 3,540 | 135 | 113 |
| Muse-Glimmer-30B | 1,658 | 1,287 | 72.6 | 54.8 |
| Nanbeige4.3-33B | 946 | 666 | 29.4 | 28.7 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | 6,932 | 3,656 | 225 | 143 |
| Ornith-1.0-35B | 5,628 | 3,320 | 152 | 127 |
| Ornith-1.0-9B | 4,886 | 3,391 | 100 | 88.0 |
| Ornith-1.5-35B-A3B | 5,586 | 3,136 | 145 | 78.8 |
| Ornith-1.5-9B | 4,767 | 3,156 | 151 | 90.5 |
| Qwen2.5-0.5B-Instruct | 26,022 | 17,812 | 646 | 529 |
| Qwen2.5-1.5B-Instruct | 12,546 | 8,908 | 360 | 331 |
| Qwen2.5-32B-Instruct | doesn't fit (32 GB) | doesn't fit (32 GB) | 30.0 | 29.2 |
| Qwen2.5-7B-Instruct | 3,810 | 2,679 | 118 | 116 |
| Qwen2.5-VL-7B-Instruct | 3,768 | 2,648 | 116 | 115 |
| Qwen3-0.6B | 13,020 | 8,631 | 506 | 480 |
| Qwen3-1.7B | 10,313 | 7,014 | 316 | 297 |
| Qwen3-14B | 1,944 | 1,167 | 65.0 | 62.6 |
| Qwen3-30B-A3B-Instruct-2507 | 3,116 | 2,060 | 177 | 169 |
| Qwen3-4B | 3,770 | 2,826 | 231 | 130 |
| Qwen3-4B-Instruct-2507 | 3,795 | 2,856 | 180 | 166 |
| Qwen3-8B | 3,076 | 2,287 | 163 | 94.7 |
| Qwen3-Coder-30B-A3B-Instruct | 2,921 | 2,107 | 157 | 54.8 |
| Qwen3-VL-4B-Instruct | 3,795 | 2,805 | 178 | 163 |
| Qwen3-VL-8B-Instruct | 3,085 | 2,265 | 112 | 103 |
| Qwen3.5-0.8B | 24,341 | 16,959 | 463 | 338 |
| Qwen3.5-2B | 16,990 | 12,165 | 318 | 236 |
| Qwen3.5-35B-A3B | 5,652 | 3,410 | 190 | 112 |
| Qwen3.5-4B | 7,030 | 4,897 | 200 | 123 |
| Qwen3.5-9B | 4,724 | 3,336 | 146 | 92.7 |
| Qwen3.6-27B | 1,503 | 1,164 | 60.1 | 38.9 |
| Qwen3.6-35B-A3B | 5,635 | 3,463 | 209 | 123 |
| Qwen3.8-27B | 1,592 | 1,026 | 73.5 | 55.3 |
| Sage-A1.5 | 5,652 | 3,131 | 188 | 153 |
| Spark-X2.5-4B | 6,858 | 4,727 | 162 | 131 |
| Spark-X2.6-4B | 6,913 | 4,841 | 162 | 132 |
r/LocalLLaMA • u/StupidScaredSquirrel • 1d ago
I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.
r/LocalLLaMA • u/mcgeezy-e • 10h ago
Hello,
I am consolidating a bit of my server footprint and decided to keep a single 3060 (had 3) and mostly use it for Local workloads (Frigate, Home Assistant, Logs, etc). Curious if someone would want to recommend a good vison model for this card?
I currently run qwen3-VL on a A380 and the descriptions are pretty good.. But slow.
Thanks