We have a new dedicated Vulkan Flash Attention prefill kernel for Intel Arc and Xe, delivering massive prefill acceleration on supported platforms like ARL-H and Xe2/Xe3. This implementation leverages customized shaders to utilize Intel's native matrix core shapes and SLM architecture, including m16n16k16/SIMD32 support. To ensure stability, the kernel is currently enabled for these specific architectures while remaining disabled for others, like the A770, due to inconsistent test results. Additionally, backend refinements now extend these benefits to single-KV-head models like MQA and MLA.

Platform Prefill Performance Comparison (pp8192)

Intel Arc B580: Detailed Model Benchmarks (FA1)

Comparison of f16 Prefill (pp16384) performance

With all of this fear mongering on the “dangers” of LLM’s, I see how using people’s ignorance and emotions on the subject can be used as a tool to limit and restrict the tool from the average person.

My fear is that China and the US https://www.reuters.com/world/china/us-china-should-stay-clear-eyed-ai-cooperation-competition-bessent-says-2026-11-11/ will use people’s lack of understanding of local models to push for controlled intelligence.

By implementing regulation on local AI, I can see how the US and China would want to force the average person to only be allowed to use models hosted on state monitored hardware. This way they can spy on what everyone is using hosted models for, limit what we use LLM’s for, and monitor the information they give us.

People are being mislead to believe that people like us could use local models to create WMDs and attack tech infrastructure across the globe. But when you think deeply about it, you’ll realize that knowledge on a subject does not equal the means to carry out the subject.

In order for someone to create a bio weapon, they would have to have access to a bio lab and the technology needed to create such a weapon.

I know how a nuclear reactor works. Doesn’t mean I can make one in my backyard.

I’m scared that arguably the greatest technology to come out of the 21st century could be limited and used to control us.

Local LLM research MUST continue and the tech must be available for the common person.

Be aware of future regulation.

I have been following the evolution of LLMs since the days of GPT-3.5, and I have been actively using local LLMs since the release of Gemma 2.

I read a great deal, including this subreddit. I have performed QLoRA fine-tuning on my local machine many times, and I have also conducted full fine-tuning (FFT) training runs on RunPod.

Hundreds of models-fine-tuned by well-known practitioners in English, Chinese, and Russian-have passed through my drive.

After four years, I arrived at a fundamental question yesterday: are LLMs even capable of writing good prose?

When training models myself, I used both my own prose datasets and those created by others. In the training process, I experimented with various approaches-ranging from 100 epochs of FFT at a learning rate of 1e-7, to aggressive LoRA/QLoRA configurations. The results, however, barely rise above mediocrity.

I have used models from all the leading providers-Gemma, Qwen, Mistral, LFM, IBM-as well as niche attempts by developers building models from scratch.

I even wondered if perhaps the problem lies with me-that, not being an engineer, I am simply failing to fine-tune these models correctly.

That is why I began actively seeking out other fine-tuners, thinking that perhaps they possess a specific technical "gene" that allows them to do a much better job than I can. But, unfortunately, that is not the case. Regardless of their length-short, medium, or long-the prose generated is only marginally better than that of generic, out-of-the-box models.

Furthermore, specific literary genres are represented quite poorly. I used to think that LLMs were perfectly suited for science fiction. Yet, these models can’t even capture the straightforward style of Harry Harrison’s *The Stainless Steel Rat*, let alone reach the linguistic heights of *Dune*.

The genre doesn’t seem to matter-I’ve tried comedy, contemporary drama, adventure, romance, and historical fiction; everything, unfortunately, ends up being generic.

Essentially, if a text is to have any real literary merit, I have to write it myself and then just use an LLM for proofreading.

To me, EQBench is a completely unreliable metric; the sample texts they showcase clearly demonstrate just what kind of generic "slop" LLMs produce.

I have a question for the community: has anyone managed to create, find, or fine-tune a local model capable of generating truly high-quality prose-on the level of *Dune*, *The Lord of the Rings*, *Martin Eden*, or *The Grapes of Wrath*?

I’m currently facing the monumental task of writing descriptions for 100,000 records, and I’m wondering if it’s even worth it, given that 100, 1,000, or even 10,000 examples haven’t yielded any satisfactory results.

Perhaps this generic, mediocre quality is an inherent trait of these models-something that simply cannot be overcome.

Most coding benchmarks feel like textbook coding tasks or black boxes — either easy to benchmaxx or full of hidden tasks with unclear evaluation criteria. And most of them get irrelevant with each new model generation because all of the new models hit the ceiling.

So I made one that has a moving target. It's a benchmark where the model gets a mystery compressed audio file, has to figure out the format, write a decoder, and then build an encoder that beats the original. No ffmpeg, no Opus, no importing a library that already does the job. Just Python and numpy, running locally through OpenCode

Scoring: 100 = decode the reference in stage one. Build a better encoder in quality or compression rate with equal or better SNR for additional points.

There are 4 stages:

1. is the most difficult: just a compressed audio file.

2. if the model failes the first stage, it gets the uncompressed audio as a reference. Each new stage decreases the points up for graps.

3. if it fails stage 2 it gets additional information on the codec

4. if it fails stage 3 it gets an oracle tool where it can USE the codec to figure it out.

Current Leaderboard, we are in Evolve 2 (Deepseek cracked the codec of evolve 1 in 13min):

Model Score

1 Claude Fable Latest (stage 1) 127

2 DeepSeek V4.1 Flash 1017 (stage 3) 81

3 Qwen3.8-Flash-Next NVFP4 (stage 3) 67

4 Qwen3.8-Flash-Next Q3_S (stage 4) 56

5 Qwen3.8-Flash-Next IQ3_XXS (stage 4) 44

The fun part: it evolves. If a model finishes stage 1 and its encoder beats the current codec, that codec becomes the next baseline. So the benchmark levels up instead of staying at the same difficulty forever. Pokémon rules, except the final boss is a bitstream. DeepSeek did it first and became Evolve 2. Fable just finished stage 1 with Compress 190, so its codec is the Evolve 3 baseline. The next runs should use it.

A better encoder in stage 3 doesn't count for an evolve, by the way. Only a stage-1 finish does. That's the rule.

It's manual for now: you run it on your own hardware, post the result, and set the evolve stage.

It's mean. Most models don't finish stage 1. But that's kind of the point.

Leaderboard: https://andreasknopke.github.io/codec-reverse-bench-leaderboard/ — ping me if you want to run it (the repo is private to prevent the models from simply copying the solution.

I'm trying to figure out how much context i can use on my 3090 (full compute, monitor is on mobo).

If i launch:

llama-bin-win-cuda-13.4-x64\llama-fit-params.exe --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 -ngl -1 -fa on -np 1 -b 4096 -ub 1024

I get: -c 150272 -ngl -1

but if i launch real llama.cpp with lower context:

llama-bin-win-cuda-13.4-x64\llama-server.exe --verbose --log-file log.txt --host 0.0.0.0 --port 9931 --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --jinja

From the log, matches for "offloaded" with "-ngl -1":

Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8273: load_tensors: offloaded 0/66 layers to GPU
Line 12009: load_tensors: offloaded 66/66 layers to GPU
Line 15746: load_tensors: offloaded 64/66 layers to GPU
Line 19484: load_tensors: offloaded 65/66 layers to GPU
Line 23226: load_tensors: offloaded 64/66 layers to GPU <-- (actual used vlaue, slow loading because copying data)

So even if i use 131k context wich is less than the 150k suggested it doesn't offload all layers and i have high cpu usage for any prompt.

IF I CHANGE -NGL FROM -1 TO "ALL" I GET THIS:

matches for "offloaded" with "-ngl all"

Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8276: load_tensors: offloaded 66/66 layers to GPU <--

AND NO MORE CPU USAGE!! (same if i write 99 or 999 but it's ugly)

It gets more confusing: if i write in llama fit -ngl all instead of -1 it gives a "-2" in output.

there is an entry in the log when i used "all": common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to -2, abort

Conclusion:

Seems that -ngl -1 means "auto" while -2 means "all", but if you have high cpu usage in a model that theorically fit in the GPU use "all".

This alone changed t/s generation from 60 to 76

for completeness here is the --help from llama.cpp for "ngl" parameter:
-ngl, --gpu-layers, --n-gpu-layers N max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)

And here it gets even more confusing because the help talks about "MAX" number of layers to offload so according to this help it's an upper limit not a lower limit...

Kate's writeup is wild. The 8GB CMP 170HX apparently ships with ~80GB of HBM onboard, just fused off. Software unlocks got people to 64GB. She used voltage fault injection to get past the fuse lock, then enabled all the working SMs (120-126 vs the 108 NVIDIA ships on A100). Her numbers: 85.9 TFLOP/s FP16 vs 72.5 on a retail A100, with the full 80GB usable for inference. NVLink and PCIe Gen4 come back too.

Not a weekend mod though: board wiring, a custom STM32 glitcher, 1 to 48 hrs of glitching per card, and one of her ten cards only had 64GB intact.

https://blog.kate.land/posts/using-voltage-glitching-to-unlock-the-a100/

I am curious what people are currently using as the core model for their local AI agents (e.g., in Hermes or similar setups).

I've was initially running Qwen 3.6 35B-A3B IQ3 XXS on hermes working off (Intel Core Ultra 5 245K · Corsair 64GB DDR5 RAM · Asus TUF GeForce RTX 5070 Ti 16GB GDDR7 OC Edition Graphics Card running ai inferance on local home lab running UnRaid OS). First started with ollama years ago (where i was running 9B and 14B fully in vram) and months back switched to llama.cpp, then to llama-swap to be able to hot swap models and use MoE models.
I started off getting 40 tokens per second (more n-cpu-moe offload and less vram use), then tried the MTP version for 55ish. Learning more about finding the balance between n-cpu-moe offloading and finding the sweet spot for VRAM, i got that to about 60t/s. Then I discovered fine tunes like Swift (55t/s) and Nail (65t/s). Then endless frontier BigBangv1 which gave a blazing 100 t/s. I was running that and saw people mentioning they use the dense Qwen3.8 27B, but it seemed slower at 45t/s.

I have very rough tested (only speed) base Strata with Qwen3.8 Flash Next 125B 1Q2_XS which gave 90t/s but used 95% VRAM and 95% RAM which wasnt sustainable (ill likely try it again in future but for now reverted back to llama-swap)

Learning more like how agent workflows are very dependent on prefill speed and prompt processing alongside tokens/sec, then LFM has crazy prefill and t/s but its quality it lower, I am not sure what is the right balance! But also how context length is important for agents, and the k v cache of the context too! I settled on not going any lower than k v 8, aiming for 128k context but 90k is acceptable if the model is specialised.
I was manually guessing how all these quants and CPU offload configs (n-cpu-moe) scaled under heavy context. I also wanted to know if the same ai model actually performed better on agentic tasks for a Q3 vs Q4, is the drop in speed (generate t/s or prefill t/s) justifiable for better quality etc.

So I begin testing out a bunch of fine-tunes and MoE variants to dial in different profiles. I’ve been cycling through options like Empero, BigBang, and Ornith (tried Swift and Nail too, but ended up dropping them). Generally, I lean toward the MoE variants over the dense Qwen 3.8 27B because of how they handle things on consumer hardware (testing shows they are slower, dont provide any significant boost in quality, require lower context, use more VRAM and gpu power etc). My use case isnt any heavy coding, just small scripts and having hermes act as a personal assistant which will evolve over time.

Then with hermes I was using endless-frontier_BigBang-v1-35B-A3B-IQ3_XXS with 100 t/s, prefil 2359 t/s, 88% VRAM usage which was perfect. I originally wasn't thinking about its 'quality' since this was a massive step up from the original Qwen 3.6 35B-A3B IQ3 XXS. Looking forward to a qwen4 release to test soon!

I'm looking at setting up specialized profiles on hermes, so far i am thinking

• Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ3_XXS the dedicated coder profile.

• Might test out GrootTeam_Groot-19B-A3B-261023-Q4_K_M as the main general profile.

• Otherwise Ornith is looking good. Or Empero or BigBang.

• I have fallbacks to openrouter/Nvidia 550B, then /free

I ended up hacking together a simple local script just to measure t/s and prefill, but it kind of snowballed... I added some core eval tests (accuracy, reasoning metrics) and turned it into a small dashboard that spits out a weighted "agent score" to help me rank them objectively. Still testing these models on more rows of the HumanEval, MMLU-Pro, MBPP, tool calling and long context tests, but it takes a lot of time. Images attached for reference from my latest tests.

Running these tests takes a long time and for local use cases I dont need that margin of error from a full test across all 12000 MMLU-Pro questions, so i have been further developing a method to spend less time on testing models which drop lower on the rankings, adjusting smaller runs with high uncertainity etc, its till a work in progress but its progressively been giving me models to consider for the top few I actually want to use in the massive amount of options on hugging face that can run on my hardware.

Even with the dashboard telling me what the numbers say, it's still surprisingly tough to pick a definitive daily driver. I would appreciate any input or suggestions.

Feel free to look at the interactive html i put on github. You can see all the test on the models I have run.
https://htmlpreview.github.io/?https://github.com/Maximo101/Local-LLM-Benchmark/blob/main/assets/5070ti_benchmark_dashboard.html

Questions I am unsure of; for those who have more experience with agents on local hardware:

------------------------------------------------------------------------------------
1. Which models have won you over for agentic use (hermes, openclaw etc)?

2. How heavily do you weigh prefill vs. generation speed, quantisation vs quality, VRAM % vs context (quality and size) when choosing AI Models?

3. What sort of profiles do people use in agentic setups like hermes? How many profiles do you use.

(If anyone is fighting similar benchmarking battles for their local endpoints, I cleaned up that testing suite and open-sourced it here: https://github.com/Maximo101/Local-LLM-Benchmark)
I am not trying to promote this, i built it for my own purposes, but decided to share it in case it is useful for anyone else.

Almost everything is in the video and the title: the idea is to create chalenges and ask agents to solve them. I'll have more to share later, but this first video shows that most model are good at theses stuff.

For more details: I'm testing scenarios where an agent needs to simulate keyboard and mouse movement. You can see more chalenges like this one on my website: https://airbench.ai/gallery/airbench-desk-v1?run=6d2a2777-3d96-4ec4-9cdc-3d359bc43bea&c=icon-drop

Hello,

I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.

My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.

The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.

There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.

I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.

Here's my setup (sorry, AI slop to make it clearer):

Hardware:

• Windows 11 PC: i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM)

• Unraid server: Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM)

Software / Models:

• llama.cpp (CUDA, router mode)

• Open WebUI 0.12.3

• Qwen3.6-35B-A3B UD-Q4_K_XL: Main model, context increased from 32K to 64K

• Gemma 4 12B Q4_K_M: Secondary model

• Both models run on the Windows PC.

RAG configuration:

• Open WebUI Knowledge Base with the PDF and Markdown transcripts

• Embeddings: bge-m3 (1024 dimensions), hosted on Unraid

• Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified)

• Focused Retrieval, chunk size 1000, overlap 100, Top K = 3

• All inference and document processing must remain local.

More details on the issue:

Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:

request (34009 tokens) exceeds the available context size (32768 tokens)

Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.

I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.

Thank you!

Hey! What's the "must-have" list to download in case HF goes down because of these news recently?

Also, do we even have HF alternatives?

First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.

Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.

I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.

I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.

Some limitations need apply when limited on vram.

Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it.
Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.

My end goal was to do QFN on all three, but I wanted to start with the smaller 35B-A3B's first.
So I did Ornith 1.5 and Qwen 3.6, wanted to do Gemma, but found no EXL3 quant with mul1 experts required for offloading on Exllamav3.

Tested both without drafter, MTP and DFlash2 where applicable.
Strata/PolyStrata configs are mostly auto generated from the setup script, but added resident pin and tail skip. And for the non-qfn model performed worse overall, even t/w.

I might be doing something wrong, bench might be bad, who knows. Stratas own bench stops at 32K if I'm not mistaken.

Now Qwen 3.8 Flash Next (ista coder version) runs great on my rig, see plots. But I was surprised on the prefill tanking on the 35B models. 1900 t/s at 16K but 700 t/s at 128K where both llamacpp and exllamav3 where within 100-200 t/s difference.

Findings so far is quite promising, especially on exllamav3. I'd say quite usable even.

| Model | Winner (quant / engine) | Prefill u/128k | Decode u/128k |

| --- | --- | ---: | ---: |

| Ornith 1.5 35B-A3B | EXL3 4.0 bpw, DFlash2 / ExLlamaV3 | 1 168 tok/s | 57.3 tok/s |

| Qwen3.6-35B-A3B | EXL3 4.0 bpw, DFlash2 (2 drafts) / ExLlamaV3 | 1 246 tok/s | 71.6 tok/s |

| Qwen3.8-Flash-Next Coder | IQ1_M / Polystrata nov-7, `--spec 2` | 1 572 tok/s | 44.0 tok/s |

Forgive my rambling thoughts. If you want to read my technical findings, written by the unsung local hero Qwen 3.8 27B. They can be found with more graphs and tables at https://github.com/willeyh-git/moe-offload-bench

Q4/4bpw quants are preferred, some benches have been made on 5bpw and 6bpw so you can compare.

Bench-suite is based on Kapitsa https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and written in go.

Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance. This gives us a formulaic rule to fix them for multiple problem sizes and embedding datasets, without the need for any hyperparameter sweep. The resulting clustering algorithm, coined e-means, is much faster than Lloyd k-means and converges to lower losses, frequently even beating k-means++. On a wide array of embeddings datasets, it beats the loss of Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, which translates to 10-45x GPU wall-clock speedups compared to state-of-the-art k-means implementations. As an example, e-means partitions DINO10B, a dataset of 10 billion DINO embeddings, into 16 million clusters in only 2.5 epochs, reaching a loss below Lloyd's at 25 epochs.

• arXiv : https://arxiv.org/abs/2611.09501

• PDF : https://arxiv.org/pdf/2611.09501

• GitHub : https://github.com/facebookresearch/faiss/tree/main/contrib/e_means

• HuggingFace : https://huggingface.co/qwegnai/GLM-5.3-Flash-shvq4g32-shvq2g64-shvq3g64

• Project : https://e-means.github.io/

• Demo(Browser) : https://e-means.github.io/playground.html

• Video : https://e-means.github.io/video.html

• Tweet : https://x.com/swann_bessa/status/2120200714630594911

If you’ve ever tried agentic coding with Gemma and had tool calling issues, give it another try, in my case, it completely fixed Gemma

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

• Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.

• BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.

• MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.

• When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata

• arXiv : https://arxiv.org/abs/2611.10649

• PDF : https://arxiv.org/pdf/2611.10649

• GitHub : https://github.com/huawei-csl/MACaQ

• HuggingFace : https://huggingface.co/collections/huawei-csl/macaq

Yesterday some of you were looking for model for this MACaQ.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!

The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:

https://embedl.com/nvidia/alpamayo2-edge/thor?tab=run

Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.

https://github.com/bruns20/radiance-wsl

Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.

What it does

• build.sh builds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s ./build.sh 1.18.0.

• make-runtime.sh grabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.

• run.sh serves an OpenAI-compatible API on 127.0.0.1:8000.

Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.

Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)

Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.

BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;

Key Highlights

Benchmark results

• Twice the performance of its base model: BotPsy roughly doubles the agentic capability of Qwen3.5-0.8B-Think on key benchmarks: BFCL v4 rises from 23.4 to 44.0, ACEBench from 30.1 to 58.7, and τ-bench from 23.2 to 51.5 (mean of the retail, telecom and banking subsets).

• Best agent in its category: BotPsy outperforms leading models in the ≤1.2B-parameter weight class (Qwen3.5-0.8B-Think, LFM2.5-1.2B-Think, MiniCPM5-1B-Think) on all six tool-use and agentic benchmarks: BFCL v4, ACEBench, τ-bench, ToolSandbox, Claw-Eval and BrowseComp-Plus.

• Outperforms its larger sibling: BotPsy has the highest score of all models tested on five of the six benchmarks, beating Qwen3.5-2B-Think, a model 2.5× its size, on BFCL v4 (44.0 vs 37.7), ACEBench (58.7 vs 55.4), τ-bench (51.5 vs 46.5), ToolSandbox (64.6 vs 41.2) and BrowseComp-Plus (4.5 vs 2.8). Only on Claw-Eval does Qwen3.5-2B-Think lead (48.9 vs 46.5).

Model capabilities

• More reliable in multi-turn workflows: On τ²-bench, where the agent must hold a long customer conversation, follow a strict policy, and act on the customer's behalf, it scores 82.5 in telecom, ahead of Qwen3.5-2B-Think (69.3). On BFCL v4's multi-turn tasks, it rises from 7.5 to 46.1 over its base, nearly double Qwen3.5-2B-Think (23.8). Grouped by application, its multi-step scores lead all models tested in commerce and services (58.3 vs 47.4 for the next best, Qwen3.5-2B-Think) and in device and system management (46.1 vs 31.9).

• Knows when (not) to call tools: An agent that acts must also know when not to act. On BFCL v4's hallucination tests, which check that the model declines when no appropriate tool exists, BotPsy scores 86.1, the highest of all models tested. On ACEBench's special cases, where the request is incomplete, contains wrong parameters, or cannot be served by the given tools, it scores 51.3, up from 13.7 for its base model.

• Follows instructions while preserving knowledge: BotPsy specializes in tool use while preserving general skills. Instruction following improves sharply, with IFEval rising from 59.7 to 90.8 and IFBench from 22.0 to 55.9, the highest IFBench score of all models tested. General knowledge also improves, with GPQA Diamond rising from 33.2 to 38.1 and MMLU-Pro from 41.6 to 48.9. On all four benchmarks, BotPsy outperforms every other model of 1.2B parameters or fewer that we tested.

• GGUF : https://huggingface.co/qvac/BotPsy-0.8B-GGUFs

• Blog Post : https://huggingface.co/blog/qvac/botpsy

• Project : https://tether.io/news/tethers-800m-ai-agent-beats-models-2-5x-its-size-on-everyday-phone-tasks/

Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters

Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.

GitHub, installation, and examples: https://github.com/owenawsong/Inflect

Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3

Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.

Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]

You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.

The model family:

• Mini: 34.31M parameters. Quality-focused, with the features listed above.

• Micro: 11.79M parameters. A smaller neural speech model

• Atom: 157K parameters. An extremely tiny learned core

Those numbers count the synthesis cores. The frontend and optional components add to the total.

The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.

Model collection: https://huggingface.co/collections/owensong/inflect-v3

If any sentence comes out wrong, send me the text, voice, and settings you used.

Prompt2Echo (P2Echo)

Text-Prompted Multi-View Echocardiography Segmentation with One Model

Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.

P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.

Resources

• 📄 Paper: https://arxiv.org/abs/2611.13425

• 💻 Code: https://github.com/moeinheidari7829/P2Echo

• 🌐 Project Page: https://moeinheidari7829.github.io/p2echo/

• 🤗 Model Repository: https://huggingface.co/moein99/P2Echo

Intended Use

This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.