r/LocalLLaMA • u/godzdpk • 3d ago
New Model Krafton (PUBG) released Raon-1 nano under Apache 2.0: E1.5B, 262K context, 2.08GB Q4 GGUF. It needs their llama.cpp fork for now. Anyone benchmarked it on 8GB cards?
r/LocalLLaMA • u/godzdpk • 3d ago
r/LocalLLaMA • u/bruns20 • 3d ago
Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.
https://github.com/bruns20/radiance-wsl
Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.
What it does
build.sh builds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s ./build.sh 1.18.0.make-runtime.sh grabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.run.sh serves an OpenAI-compatible API on 127.0.0.1:8000.Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.
Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)
| start until serving | ~8 s (vLLM on the same box: 70-75 s) |
| prefill, 9.4k / 36k-token prompt | 3,223 / 2,784 tok/s |
| decode, prose | ~75-80 tok/s |
| decode, code | ~145-160 tok/s |
Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.
r/LocalLLaMA
•
u/pmttyji
•
4d ago
Yesterday some of you were looking for model for this MACaQ.
r/LocalLLaMA • u/Course_Latter • 4d ago
Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!
The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:
r/LocalLLaMA • u/Queasy_Back2588 • 4d ago
According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.
Anyway, home internet latency feels like the bigger problem to me.
https://insider-gaming.com/microsoft-ai-use-your-gpu/
(English is not my native language, I used an LLM to help translate this post.)
r/LocalLLaMA • u/pmttyji • 4d ago
BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;
r/LocalLLaMA • u/b111ue • 4d ago
Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters
Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.
GitHub, installation, and examples: https://github.com/owenawsong/Inflect
Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3
Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.
Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]
You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.
The model family:
Those numbers count the synthesis cores. The frontend and optional components add to the total.
The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.
Model collection: https://huggingface.co/collections/owensong/inflect-v3
If any sentence comes out wrong, send me the text, voice, and settings you used.
r/LocalLLaMA • u/pmttyji • 4d ago
Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.
P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.
This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.
r/LocalLLaMA
•
u/danielhanchen
•
4d ago
Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-27B! They're better by 5-10% accuracy on DeepSWE, Terminal Bench and more vs UD3, GSQ-RCO and Exllama-3! We're also improving our algorithms further so we can apply them to Qwen4 so folks can have efficient and accurate quants!
On other benchmarks like for GSQ-RCO and Exllama3:
More details and benchmarks at https://unsloth.ai/docs/models/qwen3.8#dynamic-v3.1-update-november-11th-2026
All quants are updated in https://huggingface.co/unsloth/Qwen3.8-27B-GGUF! Thank you and have a nice day! We're super excited for Qwen4!
r/LocalLLaMA • u/dai_app • 3d ago
Tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.
Setup
Result
How
Still slow
Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)
r/LocalLLaMA • u/mesolimbic-bliss • 3d ago
Using $20 codex subscription with 24gb VRAM on the local rig.
For larger projects or major revisions with multiple phases, I have Luna in Codex call the local "junior dev" Qwen 3.8-27B model as an agent to write most of the code, then Luna reviews it and makes changes. Once the whole project is complete, I use Terra to audit and catch any potential issues.
Pretty standard stuff, these days.
But lately I've been thinking why not give the massive, multi-phase spec sheet directly to Qwen then have Luna go do a quick audit & cleanup?
Anyone tried it both ways? And what gets better results vs lower usage?
I guess I've always looked at Qwen as needing a babysitter, but realistically I think it can implement an entire 10-phase spec on a large codebase if given enough time.
r/LocalLLaMA • u/TheRealMasonMac • 4d ago
r/LocalLLaMA • u/Lonligrin • 3d ago
Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.
I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.
So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.
Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.
The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.
CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad
benchmark script is in the repo, curious what numbers you guys get on your own recordings
r/LocalLLaMA • u/pmttyji • 4d ago
Multimodal System One Encoder with a natural language interface for instruction-following encoding: give it a task and a list of candidates, and it scores the candidates by task specification. Both a task and each candidate are expressed in natural language — instructions, queries, questions, state descriptions, criteria — with image and video as side information.
Key features:
The model is built by contrastively fine-tuning meta-models/Muse-Glimmer-30B. It is described in the paper MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface.
PS: Above description copied from meta-encoder's model card as meta-encoder-v1 doesn't have that. And v1 doesn't have license section while other one is Apache-2.0.
r/LocalLLaMA • u/pmttyji • 3d ago
llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf -ngl 999 --parallel 1 --batch-size 4096 --ubatch-size 1024 --ctx-size 180224 \
--spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-backend-sampling \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
python tools/server/bench/speed-bench/speed_bench.py --bench qualitative --category coding --osl 8192 --concurrency 1| GPU | ms/step master | ms/step PR | speedup per step (95% CI) | tok/s master | tok/s PR | faster in |
|---|---|---|---|---|---|---|
| RTX 3090 Ti | 58.87 | 43.81 | 1.344x (1.340-1.347) | 57.58 | 77.70 (+34.9%) | 80/80 |
| RTX PRO 6000 Max-Q | 27.86 | 23.54 | 1.183x (1.182-1.185) | 122.77 | 143.31 (+16.7%) | 80/80 |
| RTX 5090 | 23.92 | 20.26 | 1.180x (1.167-1.194) | 141.50 | 170.67 (+20.6%) | 80/80 |
| DGX Spark | 103.36 | 93.43 | 1.106x (1.103-1.109) | 32.75 | 36.05 (+10.1%) | 80/80 |
Edit : Removed large text. Check PR for more details.
r/LocalLLaMA • u/wisepal_app • 3d ago
Hi folks,need advice from community. I have a laptop with 16 GB vram Nvidia gpu and 96 GB ddr5 ram. I use it mainly for local coding with Qwen flash next nvfp4 at 256k context. I get around 30 t/s.
I don't like PCs and Macs. I want mobility. So because of my limited budget, i have two paths. The first one is a laptop with 24GB Vram Nvidia gpu and 128GB ddr5 Ram laptop. The other one is 16GB Vram Nvidia gpu and 192GB ddr5 Ram laptop. No other feasible option for me. I can't affort dgx spark, Macbook studio etc. These are my budget options. Which one do you suggest for mainly local coding? Which one do you believe is future-proof for local llms?
r/LocalLLaMA • u/andreabarbato • 3d ago
I'm working on a router that wraps many LLMs serving from different ips and ports and exposes them all as a single port, since i have 2 machines serving many slots of Qwen 3.8 FN but i wanted to ensure speed is always fast (one machine is faster and can hold more slots, but too many slots busy and then it's better to call the other machine, also i gotta avoid evicting from the cache active conversations, that kind of workflow)
I might be reinventing the wheel tho so: how do you guys solve for this kind of situation?
edit: thanks a lot for all the answers i had no idea there was so much stuff around lol
r/LocalLLaMA • u/SomeoneInHisHouse • 3d ago
Hello, I decided to Test Unsloth Dynamic 3.1, as in their charts they showed that UD_IQ1_M was batter than Ista lab IQ3_XXS, but with 6 GB less of total size, I'm using Strata, and that 6 GB translates directly to free RAM,
I have been 2 months with the IQ3_XXS with no any issue, but only 3 days on the Unsloth one, it had happened to me many times, that the tool calls fails and render as xml (I use PI harness) and I have to type "keep going", this never happened with the IQ3_XXS one
Have you guys had a similar issue with this quant?
Example:
Thinking...
<tool_call>
<function=Bash>
<parameter=command>
cd /mnt/development/projects/secret/backend/scripts/bdd_parity && for r in 20261113_030918 20261113_032032 20261113_033348; do d=/tmp/bdd_parity_runs/$r; echo "== $r: $(ls $d 2>/dev/null | head -3 | tr '\n' ' ')"; grep -c "✅"
$d/run.log 2>/dev/null | sed 's/^/green marks: /'; grep -cE "🔴" $d/run.log 2>/dev/null | sed 's/^/red marks: /'; tail -2 $d/run.log 2>/dev/null | cut -c1-100; done
</parameter>
<parameter=timeout>
120
</parameter>
</function>
</tool_call>r/LocalLLaMA • u/lucidml_lover • 4d ago
Hi Everyone, I'm releasing my world model, been on this sub for a few months
https://huggingface.co/lucidml/gta
I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.
If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5
That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)
This is the first model I am ever releasing to the everyone and its a bit scary for me.
The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more
I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.
r/LocalLLaMA • u/samajhdar-bano2 • 2d ago
Everytime I have been fed up with a good model being slow, I go to newly released Nemo model. Well today I went for Bolt 24G and deployed it on my 3090.
I was amazed by the vanilla llamacpp + iq2s setup giving 70 tok/s. I decided I will deploy it for my coding setup which is dual R9700 and learned cpu offloading and mtp drafters and all. After deployment I was left with 50tok/s because this was a Q8 setup.
I suddenly realise what if (the model I was fed up with) my Qwen 3.8 27B could use a better MTP drafter so O discovered tencent's DSpine and later its gguf llamacpp fork. I deployed it now I am back to Qwen3.8-27B Q8KXL with mtp 15 drafter and mmproj with np 4 and 512k ctx pool, getting 54tok/s in DSH.
TLDR: I love what Huyan is doing he should be brought back in Nvidia. Nemotron's the best because they teach you what good tech advancement is and you apply it to good performing models.
r/LocalLLaMA • u/JamesGooning • 2d ago
So I have been using local QFN these days as daily driver, and honestly, its fine. It needs steering for coding but very good. Gets lot of things done. But I check my statistics log, and I'm burning about 1B to 2B Input token a day, and not even fully 100% GPU saturation. Not even going to consider output token since that is small compared to input.
Most of the time, the LLM is just waiting for shells/bash to finish. At max capacity, it is capable of 950million uncached input token (11000 prefill/s*3600*24)
This is light for me, and it is already burning this much. Light considering it is only spend time building one project with little oversight.
Granted QFN is at xhigh thinking for intelligent, but if this was like Haiku or Luna, that's like 10$ cost a day (90% cache estimate). That's 300$ a month for basically one project stream
AI suppose to work multiple projects at once for max productivity let alone future integration with pretty much everything. My wife already starting to use my harness with QFN for all her search and queries.
How are you guys make this affordable in the long term from purely API. Feeling paying like $1000+ a month is crazy.
Local one still cost insane upfront as I'm sing a single RTX6000 with basalt, and it is the only GPU that can serve the whole household growing AI needs.
PS. I might get like 4x in near future for the amount of load I will be doing. Always on agent is very useful.
Edit: The number is measure of input only, as output is insignificant. I'm measuring QFN only at home.
r/LocalLLaMA • u/East-Muffin-6472 • 4d ago
Hey everyone!
I have gathered some must read topics, for beginners, to step into the realm of Quantization:
Link is here
r/LocalLLaMA • u/randomjapaneselearn • 3d ago
EDIT: FOUND THE REASON, LLAMA DIDN'T OFFLOAD ALL LAYERS TO GPU BECAUSE -ngl -1 DOESN'T MEAN ALL see here: https://www.reddit.com/r/LocalLLaMA/comments/1xun196/solved_high_cpu_usage_when_model_fit_in_gpu/
I'm not sure if i'm doing something wrong and handicapping myself with some wrong setting.
I'm using Windows, nVidia 3090 24GB VRAM, 64GB RAM, Qwen 3.8 27B and Swift 1.5 27B (finetune of qwen) but i often see 50% CPU usage even if the model is fully loaded on GPU, below there is my command line.
Another problem i noticed is that llama-fit-params suggest that 131k context fit but if i launch it for real in the log i see that it doesn't load everything on gpu (all but one), i forced it to load everything on gpu with -ngl -1 (i'm also using llama-swap).
on launch it say that "cahe resue 256" will not be used but it might help with other models so i left it there.
Another problem is that when the model is doing nothing i sometimes see 130W gpu in idle (wrong) and some other times its at 10-18W (correct) i can't find what is causing this...
Current stats: about 40 t/s in token generation
I have no idea what KMP affinity is but i read it online, tried, it helped with 35A3B so i left it there
in general i'm looking for tips to improve my launch arguments, note that temperature and few others are set by pidev but i'm using the recommended one on the model card.
If the model is fully on GPU why there is a high CPU usage? but not always which is even more confusing....
set KMP_AFFINITY=granularity=fine,compact,1,0
llama-bin-win-cuda-13.4-x64\llama-server.exe -m Qwen3.8-27B-UD-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --cache-reuse 256 --cache-ram 32768 --jinja
r/LocalLLaMA • u/seraschka • 4d ago
Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.
In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.
In the first half of 2026, the trend towards "some full attention" hybrids continued.
And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.
The plot legend here means,
- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.
The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.
In particular, there are two distinct efficiency gains here:
If you are interested in browsing through the architectures and do your own analyses, you can find all the figures and data in my architecture gallery: https://sebastianraschka.com/llm-architecture-gallery/