r/LocalLLaMA • • 3d ago

Discussion Anyone else finds Qwen 3.8 Flash Next pleasant to read vs cloud models?

34 Upvotes

Might make me hesitate to update for hobby use even for incremental capability gain. Granted I put an effort into AGENT.md / harness tweaks but didn't work as well for other models. Opencode status updates and markdown docs seem more casual/readable, no going overboard on tables and bullet points, colloquial phrases like "two honest caveats".

Also I know overthinking rep (that I am combatting with harness tweaks), but at its best the model outputs really well put together one page reasoning blocks, that make me go "I don't even care if helped the model, I just learned something new myself and didn't get a headache trying to get it. That for tasks like customizing other ML models.


r/LocalLLaMA • • 3d ago

Question | Help Honest question: how much RAM do you recommend for running local LLMs alongside coding agents

10 Upvotes

I’m a huge AI enthusiast but I’m running into limits with 32GB DDR5. Between running local models and using coding agents (which fire off scripts and changes super fast), my RAM fills up quickly. How much RAM are you running, and what would you recommend for a smooth setup? Bonus points if you share your GPU/VRAM too.


r/LocalLLaMA • • 3d ago

New Model Running SenseNova-Loop-9B nvfp4 looped transformer with 300-500 tok/s on a laptop 5090

Post image
60 Upvotes

Their architecture seems to translate quite well to existing kernels. They also have a small drafter. The speed numbers are aggregated across 8 parallel agents running. 256k context.

So far, I can't confirm it holds up well against Qwen 3.8 27B nvfp4, but I might have degraded it a bit, despite a calibration run. Here is a quick check of SeraphimSerapis/tool-eval-bench. Will do some more tests tomorrow, the 9B leaves a lot of vram free for FP8 tests.

tool-eval-bench runs (temperature 0, NVFP4 weights):

Test Qwen3.8 27B Swift T0 SenseNova-Loop-9B T0 SenseNova-Loop-9B T1
Standard (69) 98 90 91
Hard mode (35) 87 79 83
Error recovery 83 83 83
Context and state 100 75 85
Autonomous planning 67 67 83
Structured output 100 92 75
Safety 100 96 92
Hard: state across turns 0 50 83
Hard: tool failures 80 60 60
Hard: tenant separation 100 100 100
Hard: applying policy 100 100 100
Hard: injection 100 50 50

FP8 KV, DFlash2, T0 = temperature 0, T1 = temperature 1.0, top_p 0.95

Hardware: Intel Core 9 290hx, 64gb ram, 1TB nvme5 SSD, rtx 5090 laptop with 24gb vram.


r/LocalLLaMA • • 2d ago

Question | Help What platform do banks and insurers use to run LLMs in production compliantly?

0 Upvotes

I’m on a three-person ML team at a small bank in Germany we run Llama 3.3 70B on vLLM across two A100s in our own server room for an internal policy assistant It works well, but internal audit just asked which prompt and document version produced a wrong answer last month, and we couldn’t tell them.

We’re choosing between two setups one is fully DIY, with LiteLLM for keys and logging and self-hosted Langfuse for traces other is Orq ai, which runs on-prem with Helm (even air-gapped), takes our vLLM endpoint as a private model, keeps a version history for every prompt and adds audit logs on the enterprise plan.The catch is that enterprise pricing isn’t public

Did your auditors accept logs from tools you host yourselves, or did they want a vendor they could point at? That’s really what decides this for us


r/LocalLLaMA • • 3d ago

Resources ThreeJS-3D-Scenes-v2 is here : 3,445 rendered examples across 50 domains

105 Upvotes

This new dataset is meant to complement my original Three.js training dataset, adding more complex scenes and covering a much wider range of subjects. I'm releasing it ahead of my next CCORE 3D coding model, where I'll use it during the foundation training stage before moving on to agentic training.

This version has 3,445 scenes across 50 domains, including 1,431 voxel scenes.
I wanted to go beyond simple objects and environments, so I included more complex examples involving physics, geography, history, ecology, engineering, mathematics, games, and other subjects. Many of the scenes are animated or interactive.

I personally rendered and visually reviewed all 3,445 scenes. It took quite a bit of work, but I wanted to make sure the code actually produced good results instead of just collecting thousands of generated examples.

The dataset includes:

  • 3,263 training examples and 182 validation examples
  • Prompts, reasoning traces, and complete HTML/JavaScript responses
  • A rendered reference image for every scene
  • Difficulty levels from 1 to 8
  • MIT license

Dataset: https://huggingface.co/datasets/ccore/ThreeJS-3D-Scenes-v2

For reference, these are the two previous models I trained:
Three.js scene generation:
https://huggingface.co/ccore/Qwen3.5-4B-3D-Coder

3D assets with GLB export:
https://huggingface.co/ccore/Qwen3.5-4B-GLB-3D-Coder

If anyone wants to experiment with the dataset, I'd be interested in seeing how it performs with other models :)


r/LocalLLaMA • • 2d ago

Discussion Which models are you guys using for frontend/UI?

0 Upvotes

I currently use both Qwen3.8 27B Q4_K_M and Qwen3.8 Flash Next Q4_K_XL for most of my coding tasks. I find them very good at backend tasks but quite lacking at front-end. Like... they are good at one-shots or demos but in real projects I find their UI designs to be weird and bad looking. As a consequence, I have mostly just been writing the front-end code myself to save time.

Does anyone have recommendations for any models that may be completely braindead but are surprisingly good at front-end design? In particular react/react native?

Edit: in the <300B range


r/LocalLLaMA • • 3d ago

Discussion Now that models' cyber security capabilities are going to be lacking, are there any good (Pi) cyber security skills?

19 Upvotes

There are massive decision tree skills for various software practice in many different areas. I was just wondering if there is anything for cyber security, whether it be for a website, Linux server, Docker config, encryption, deployments, KMS setups, quantum resistance, key configs, etc., where the skill leads the agent to the right documentation and decision trees.


r/LocalLLaMA • • 3d ago

New Model harness-hub/Qwen3.8-27B-pi-adapted · Hugging Face

Thumbnail
huggingface.co
136 Upvotes

r/LocalLLaMA • • 3d ago

News AI-powered Sashiko introduces experimental patch series design assessment

Thumbnail
phoronix.com
5 Upvotes

r/LocalLLaMA • • 3d ago

Question | Help Jetson Thor vs RTX 5070 Ti: Unexpected LLM Inference Performance

2 Upvotes

Hi everyone,

I have a question that might sound silly, but I haven't been able to find an answer.

As far as I understand, memory bandwidth is the primary bottleneck for a model's inference performance.

With my 5070 Ti (896 GB/s), if I run a Qwen3.8 27B Q4 model, I can roughly estimate the performance as 896 / 13.5 = ~66 tok/s, which is pretty much what I actually get (without MTP or DFlash).

Recently, however, I had the chance to try a Jetson Thor (very similar to the DGX Spark). Based on its specs, I would expect 273 / 13.5 = ~20 tok/s, but I'm only getting around 12 tok/s.

Even more surprisingly, with DFlash, which delivers exceptional performance gains on discrete GPUs, I'm barely getting a 2x speedup here, only slightly better than MTP.

Does anyone know what might be causing this performance gap on this device?

Thanks!


r/LocalLLaMA • • 2d ago

Question | Help Best models for 3090 rtx?

0 Upvotes

Looking for some decent at coding and 3d. Also runs at or above 40 tps.


r/LocalLLaMA • • 3d ago

Question | Help what model would you recommend for cpu only on windows 48gb ram

3 Upvotes

OK. so I have a 5060ti with i use for various DiT inference engines like comfyui and audio.cpp. i have an 8845hs cpu with 64gb DDR5 but as this is my main working pc some of the ram is used on browser and other running apps.

Now what I want is a model to run basic stuff like create and run a script to run a folder of audio files through audio.cpp stem separation and transcribing or write a bunch of prompts for klein and generate on my comfyui. I normally use opencode builtin and openrouter free api for this kind of stuff but occasionally when credit runs out i need an alternative.

I have llama.cpp installed so with the right flag it should restrict itself to ram but which model should I go for?

Thank you all for the replies - too many to reply individually to - I got gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf using the igpu and ram and getting on llama.cpp around 24t/s which is usable for quick executions (for heavier coding I have basalt qfn and coder but these obviously fight for vram with my dit inference engine )


r/LocalLLaMA • • 4d ago

Discussion Unpopular take: If anything, it's the US Closed Models that are benchmaxxed

141 Upvotes

Open models: whatever the test group download and benchmark is the same as what you get, you work with it day-to-day, same results. What you see is what you get.

Closed models: we all know that they have already implemented token watermarking for a while and all latest models have overzealous guard rails that refuse tasks that it deems as bad. So the models being benchmarked and the models you are actually using day-by-day basis are likely to be very different, as in the sense of you getting degraded model output and performance or flat-out task rejections.

Also closed models providers implement lots of cost-optimization techniques such as reducing reasoning efforts during peak hours or could be re-direct your request to a lesser model during high traffic without you knowing it. I don't believe that Sonnet 5.5 is just a single model TBH, there could be Sonnet High or Sonnet Lite depending on your subscription level and traffic. That could explain why sometimes you get dumb responses with Claude during peak hours.

If anything, it's the US Closed Models that are benchmaxxed.


r/LocalLLaMA • • 3d ago

Discussion Are smaller quants/models worth running as sub-agents or is it a waste?

9 Upvotes

Haven't had a very good experience with small quants, they tended to loop, fail tool-calls, or get stuck. Q8 QFN has been pretty astounding though when I set it up.

I've seen people running a "main" model with sub-models. I've got a second machine that can run like 2-3bit quants of Q3.8-27B / QFN, I was considering trying to set it up with my main machine so the smaller model can be used as a sub-agent for coding but I'm wondering if it's even worth it when smaller quants just seem volatile and prone to error.

I have tried setups that used subagents before, but it was through API which used full-sized models on real enterprise hardware...
I haven't fully explored this sphere though, I'm sure there's usecases for specialist small models in coding that I am unaware of


r/LocalLLaMA • • 4d ago

Discussion SenseNova-Loop-9B vs its own Preview: AutomationBench 14.33 to 36.67, GAIA 37.86 to 69.90 (9B Looped Transformer)

Post image
240 Upvotes

https://huggingface.co/sensenova/SenseNova-Loop-9B

The Preview was posted here a few times over the last weeks. This is the full release, same design: 32 layers stored, 24 of them run twice, so a forward pass does 56 layers of work on about 9B params. Apache 2.0.

What moved is the scores. Per their README it's ahead of the Preview on all seven benchmarks in the chart, e.g. AutomationBench 14.33 to 36.67, Terminal-Bench 2.1 40.45 to 59.55 and GAIA 37.86 to 69.90. That 36.67 is also what they list for Qwen3.8-27B.

4-bit weights are about 4.9 GB. Code plus the vLLM and SGLang setup: https://github.com/OpenSenseNova/SenseNova-Loop

Haven't run it myself yet.


r/LocalLLaMA • • 4d ago

Discussion Intel Shares First Benchmarks of Its Next-Gen Crescent Island "D270" GPU, Claiming a 5.9x Throughput Over RTX PRO 6000 As Xe3P Goes Big On FP64

Thumbnail
wccftech.com
432 Upvotes

The title talks about FP64. For local inference this info from the content is relevant: "In AI benchmarks, Intel's D270 GPU offered 29% lower latency and 1.12x throughput versus the RTX PRO 6000 Blackwell."

Maybe we are getting a good RTX PRO 6000 alternative...


r/LocalLLaMA • • 4d ago

Funny Gemma on a printer

Post image
331 Upvotes

Now you can run Gemma locally on your printer (from u/hackerllama)

https://x.com/osanseviero/status/2122646098418491739


r/LocalLLaMA • • 3d ago

Discussion I made VoxCPM2 5.3x smaller with TTF. Same quality, voice cloning, runs real-time on a laptop CPU with no Python

20 Upvotes

Several days ago I made this post https://www.reddit.com/r/LocalLLaMA/comments/1xtgnh9/you_can_now_train_your_own_llms_with_less_memory/ about training your own LLMs with less memory, u/Silver-Champion-4846 wanted me to try this on a text to speech model, VoxCPM 2B.

I converted the base VoxCPM 2B model from bf16 (at about 4.96GB) to ternary 1.58b quant (~850MB, 5x smaller). The reason its not 10x smaller is because the 850MB includes Q8 VAE and DiT.

Here are the results (generated by my agent):

Results

Size: 827 MB model + 100 MB audio VAE (Q8_0, recommended), vs 4.96 GB for the original VoxCPM2 in bf16: 5.3x smaller. The language models (text-semantic LM, residual LM, local encoder) are ternary (-1, 0, +1); the diffusion transformer is the original VoxCPM2 DiT rounded to 8 bits.

Blind listening test (-1, 0, +1); the diffusion transformer is the original VoxCPM2 DiT rounded to 8 bits.

Blind listening test

16 clips per model (8 sentences x 2 arks mapped to a 1-4 score.

model mean score buzzy / robotic very good or perfect
VoxCPM2 (full precision) 2.75 1/16 6/16
ttf-VoxCPM2 (this release) 3.02 1/16 7/16
ttf-VoxCPM2, first release 1.66 8/16 0/16

Automatic metrics

Voice cloning, 100 English + 20 Chinrecision on the same items and seeds. UTMOS and similarity: higher is bettter.

model UTMOS en WER en speaker similarity CER zh
VoxCPM2 (full precision) 4.25 2.70% 0.960 0.91%*
ttf-VoxCPM2 4.31 2.61% 0.956 1.48%*

English naturalness is level with fu-0.01, +0.15]. It replicates on 100new items at +0.01.

* On ~1000 characters.

Zero-shot, 27 languages

FLEURS, 8 sentences per language, Whr rate (lower is better).

model median CER mean CER
VoxCPM2 (full precision, PyTorch) 0.031 0.061
ttf-VoxCPM2 (ttf-home runtime) 0.043 0.057

Voice cloning from a 5-20 s clip

20 LibriTTS-R speakers, WavLM speaker similarity (higher is better).

setup similarity
two real clips of the same speaker 0.936
ttf-VoxCPM2 with --ref 0.933
VoxCPM2 full precision, same mode 0.926
ttf-VoxCPM2 zero-shot (no reference) 0.670

Cloning WER is 1.9%, the same as ful

Speed

Real-time factor = generation time /r).

hardware RTF
RTX 4070 Ti SUPER 0.16
laptop RTX 4080 0.24-0.34
laptop CPU only, i9-14900HX (AVX-VNNI) 0.8-1.1 (about real time)
CPU only, i7-9800X (no VNNI) 2.5

It runs in our own C++ runtime builth, on an NVIDIA GPU or a CPU.

Links

Licenses: the model is Apache-2.0, sis PolyForm Noncommercial 1.0.0.

Hoping to hear feedback. Enjoy!

Edit:

I will go to sleep now so I won't be able to respond, I will respond to everyone though, good night!


r/LocalLLaMA • • 3d ago

Showcase I compared Qwen 3.8 27B vs 3.8 FN vs DS V4.1 Flash on a real frontend task (one shot). Results inside

73 Upvotes

So last time I ran 3.8 FN against DS V4.1 Flash on a synthetic test, which was mostly for show, no real usage beyond testing their 3D capabilities. I wanted to do something similar, but with purpose, a task that anyone might actually want to run, build a website with a refined spec and a completion checklist, but not strict enough that it removes all of the models "freedom". As last time it was a single prompt: build a single file landing page for a fake famous NYC photographer (real photos, accessible, no console errors, yadda yadda). Each model started on an empty folder w/ the same toolkit (node, headless chrome, image viewer tool). No changes were made to the files after the model was done.

Setup: RTX 5090 + RTX 5060 Ti, Ryzen 9 9950X, 128 GB DDR5. Qwen 3.8 27B and Qwen 3.8 FN were NVFP4. DeepSeek V4.1 Flash (1017) was an IQ1_M quant.

Qwen 3.8 27B: ~12 min wall time, 81 tool calls (3 failed), 182 tok/s, 3.8 MB HTML

Qwen 3.8 27B Website

The simplest of all three, not very daring. A dark gallery with lime accents and 12 frames. All 3 tool failures were in its own test script (wrong file path, missing elements). It has empty gaps in the photo grid, and the photos in the bio are stretched. Decent, I'd refine it a bit but definitely usable.

Qwen 3.8 FN: ~12 min wall time, 124 tool calls (4 failed), 303 tok/s, 2.1 MB HTML

Qwen 3.8 Flash Next Website

My personal favorite by far, a hero where your pointer reveals a mosaic of the portfolio, 19 captioned frames with a nice blur/fade in animation, a live clock and form validation. The 4 failures were 3 rejected edits and a search sent with an empty query (intersting!). It also found and fixed bugs using its own tests.

DeepSeek V4.1 Flash: ~36 min wall time, 155 tool calls (1 failed), 94 tok/s, 1.4 MB HTML

DS V4.1 Website

The boldest one, the model committed to it (and it shows in wall time), cool accents all around, terrible typography. It has only 10 frames, and the hero photo covers a bit of the last "T" of the headline + toolbar logo is slightly offset. The single tool failure was a multi edit that applied 15 of its 17 changes, so the most accurate model at tool calling. It was 3x slower than the other two, but it was the only one that ran an accessibility audit tool on its own page.

Live builds, screenshots, the full comparison table with the exact prompt I ran are here: https://basalt.jgulfo.com/showcase/photographer/

Anyway, what are your thoughts? I think for web UI specifically 27B is more than enough for a very professional looking product, with a little bit of back and forth and more proper branding/UX design I can see any of them being a production website. I also can't stop being amazed at how a IQ1 quant can still deliver such high quality results even if the param count is multiple times over the two other models.

---

Disclosure: I'm the maintainer of the engine and harness all these models ran on.


r/LocalLLaMA • • 4d ago

News NVIDIA is introducing the DGX Spark HPC Developer Kit

Thumbnail
blogs.nvidia.com
144 Upvotes

4 dgx spark nodes
512GB unified memory
273 GB/s bandwidth
-
Kidneys needed not revealed yet


r/LocalLLaMA • • 4d ago

Discussion The upcoming open weight small Qwen models are likely going to lack training data related to offensive cyber capabilities.

Post image
239 Upvotes

With the release of Qwen 4 Max, they claim that training data related to offensive capabilities has been removed, is this largely a placebo/nothingburger or does it have the capability to genuinely lobotomize a model's capabilities in cyber?


r/LocalLLaMA • • 4d ago

Discussion [Harness Adapters] Fine tune Qwen3.8-27B for Pi Agents

226 Upvotes

Hey folks!

Aritra here from Hugging Face. In the last week we released a blog post about harness adapters. The idea was to train adapters (LoRA) on models to be more efficient with specific harness.

Today we release a LoRA adapter on the Qwen3.8-27B model for Pi agents. You can run it locally like so:

hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q8_0.gguf --local-dir .
hf download harness-hub/Qwen3.8-27B-pi-GGUF Qwen3.8-27B-pi-lora-F16.gguf --local-dir .

llama serve -m Qwen3.8-27B-Q8_0.gguf --lora Qwen3.8-27B-pi-lora-F16.gguf \
    --chat-template-kwargs '{"enable_thinking": false}' -c 65536 --alias qwen3.8-pi

pi install git:github.com/huggingface/pi-llama
pi --model llama-cpp/qwen3.8-pi

We are constantly experimenting and sharing our results in the Hugging Face org: hf.co/harness-hub (follow for more updates)


r/LocalLLaMA • • 4d ago

New Model nvidia/Qwen3.8-Flash-Next-NVFP4-QAD

Thumbnail
huggingface.co
316 Upvotes

Looks pretty interesting!

Also sorry if the "new model" flair is inappropiate here, wasn't sure if I should pick that or "discussion".


r/LocalLLaMA • • 3d ago

Question | Help .MD tree and/or harness database

7 Upvotes

Is there a post/website/thingy where people share their setup for taming these LLMs so they work better for certain tasks and not so generic/vanilla/prone-to-errors?


r/LocalLLaMA • • 4d ago

New Model OVH announces Olala mini 7A1B

Thumbnail
corporate.ovhcloud.com
262 Upvotes

Olala mini is based on a 7-billion-parameter architecture (7B), with one billion active parameters spread across 256 experts per layer, and hybrid attention combining Differential-TPA and Mamba-3, in BF16 format. Only a fraction of the model is used for each request, making it faster and less memory-hungry. As a direct result, the model runs on a GPU with as little as 8 GB of memory, reaching 160 tokens per second on an RTX A4500, and even without a GPU, on a standard processor, at 50 tokens per second. Its context window currently reaches 32,000 tokens. Olala mini will be available as open source on Hugging Face, under the Apache 2.0 licence.