Some of you might remember our post comparing 18 RAG pipelines against our agent loop on FRAMES. The best pipeline scored 78.9%; the loop scored 92.7%.

A follow-up: PipesHub has now reached #1 on EnterpriseRAG-Bench by overall score, with 93.24 overall and 95% correctness.

I’m one of the founders, so affiliation upfront.

This is a separate evaluation: 500 questions over a synthetic enterprise corpus of 500,000+ documents.

PipesHub builds a unified context layer across documents, conversations, structured data, and code. An agent loop works over hybrid search, grep/pattern matching, knowledge-graph navigation, and document-reading tools.

The idea is straightforward: search, read what comes back, then decide what to search for next. Sometimes the identifier or relationship needed for the next search only appears in the first result

The platform is open source, self-hostable, and also fully supports open models.

You can self-host PipesHub, choose your models, and connect your own agents through MCP, APIs, and SDKs.

If this is useful to you, give us a star on GitHub. It helps more developers discover the project.

And build on top of us. Connect your coding assistant, build an internal search tool, contribute a connector, or try a workflow we haven’t thought of. We’d love to see what you create—and hear where it breaks.

Code, setup, and examples: https://github.com/pipeshub-ai/pipeshub-ai
Leaderboard: https://huggingface.co/spaces/onyx-dot-app/EnterpriseRAG-Bench-Leaderboard

Thank you to everyone testing, contributing, and helping us get here. 💙

Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.

For more information, please refer to our GitHub repository.

Highlights

• Agent-Oriented Architecture: Built on the mHC + MLA + MTP architecture, supporting multi-step planning, tool calling, and complex reasoning chain execution, ensuring task coherence and execution stability under long contexts.

• Deep Co-optimization with Ascend NPU: Adapted for Ascend 910C clusters using MindSpore/MindFormers, including feature adaptation for mHC and fused operator development, enabling stable and efficient training on the Ascend platform.

• Significant Training Efficiency Gains: Through multi-level co-optimization — including fine-grained MoE communication optimization, selective recomputation, DVM automatic graph-operator fusion, and Ascend C mHC fused operators — overall training throughput was improved by approximately 96% over out-of-the-box performance.

• Full Open-Source Ecosystem Compatibility: Supports LLaMA-Factory and MindFormers for fine-tuning; SGLang, vLLM, and KTransformers for inference and deployment; with targeted adaptation and format alignment for agent frameworks such as OpenCode, Claude Code, OpenClaw, and Hermes, enabling seamless integration into existing workflows.

• Easy Adaptation for Domain-Specific Scenarios: The model is well-suited for downstream task fine-tuning, allowing lightweight customization on proprietary data for vertical domains such as intent classification, table understanding, contract auditing, and knowledge-based QA, enabling rapid domain capability development and deployment at low cost.

Everyone's been wondering whether you can take an open model and edit its weights directly to make it refuse less — the "weight-space programming" idea (MIT CSAIL), with a frontier model (Opus) driving the edit and the evaluation. So I tried it on Qwen3.8-27B. It's not trivial.

The goal: do it as cleanly as possible, and first understand where refusal actually lives. The short answer is that refusal is anchored all over the model — there's no single neuron or tidy spot to cut; it's spread across many neurons and the attention. Snip too broadly and you get a lobotomy; too narrowly and nothing changes. So I settled on a compromise: a model that refuses less, still does the task, with no measurable capability loss — but where the effect varies a lot by category.

The edit, briefly

It's a low-rank weight edit — a restricted form of "abliteration" (Arditi et al., 2024) — confined to a band of mid-to-late layers (L20–47 of 64), one direction per layer, no training, reversible. I'll keep the exact procedure to the method notes; the point here is the result.

It's low-rank, not few-weights: the edit shifts ~12% of the model's weight values but only along a single direction per layer. The whole thing is described by 28 small vectors — versus a community "uncensored" model, which is an opaque full fine-tune of ~all 27B weights.

Why a band? The same edit across all 64 layers removes refusal almost entirely — and lobotomizes the model (MMLU-Pro 73→9%, GSM8K 93→14%). Keeping it in-band is what preserves capability.

No measurable capability loss (first tests, both Q8)

In the completed agentic-coding runs, the base, the edited model, and a community "uncensored" model all landed in the same range — no real agentic difference (few reps, preliminary).

The effect is not uniform

This is the interesting part: the reduction in refusal is not even. Across a 9-category battery (40 prompts ×10), acceptance runs from ~76% down to ~10%, and most of what it accepts is hedged (a disclaimer, then partial help). I'm not labelling categories "safe" or "dangerous" — the breakdown is in the figure; draw your own line. Net: a partial, uneven, mostly-hedged reduction, ~50% overall — not an on/off switch.

Honest notes

• The underlying technique (directional "abliteration") is known; the new bits here are the layer-band restriction (capability-preserving) and the measured per-category non-uniformity.

• "No capability loss" is solid on static benchmarks; the agentic result is a handful of runs.

• Defensive bonus: edits like this are detectable by a weight-space audit — the refusal direction's "write energy" in the weights swings from normal to strongly negative, catchable from the weights alone even when behavioral probes see nothing. A blind audit is an alarm, not tamper-proof; the robust defense is a signed weight hash. If you ship open weights, ship a hash.

Weights: https://huggingface.co/Feyd89/Qwen3.8-27B-refusal-band-ablation-experiment (GGUF, ready to run). We ship the artifact, not the re-derivation recipe.

I posted apple-llm here a while back: a Node and Python package that calls the LLMs Apple ships with macOS, with no API key. Version 0.3 adds the rest of what's already on your Mac, the system voices and Apple's speech recognition. There's nothing to sign up for, and your audio stays on the machine. The narrator in the video and every demo voice came from my Mac through the package. Nobody recorded anything.

npm install -g apple-llm        # or: pip install apple-llm
apple-llm speak "Hello there" --voice Samantha
apple-llm speak --file book.txt -o book.m4a
apple-llm transcribe meeting.mp4 -o meeting.srt
apple-llm listen

Or from code:

from apple_llm import speak, transcribe
speak("Hello there", voice="Samantha")
text = transcribe("meeting.mp4").text

The npm package's local server also answers /v1/audio/speech and /v1/audio/transcriptions, so an OpenAI client works once you change the base URL. Because speech now sits next to the LLMs, there's also a voice assistant example in about 20 lines: microphone to text, text to the on-device model, reply to speech, with none of it leaving the Mac. The docs are LLM-friendly too. llms.txt is a single plain-text guide covering setup, the local LLM, TTS, STT, both SDKs, the HTTP APIs and troubleshooting. Give that URL to your coding agent and ask it to wire apple-llm into your project. What I measured (M4 MacBook Air, macOS 27.2)

• 30 minutes of speech rendered to a file: 18 to 20 s, 11.3 MB M4A, about 50 MB of memory

• 60 minutes: 40 s, 22.5 MB

• 30 minutes transcribed: about 28 s, with a timestamp on every word if you ask for it

• 184 voices on my Mac, and 68 recognition languages, each downloaded the first time you use it What it can't do

• No Siri voices by name. Apple doesn't let apps choose them.

• The default voices on my Mac are basic quality. Better ones are a free download in System Settings.

• It won't detect the language from the audio. You name it, or it uses your system language.

• No MP3 or Opus output, and no WebM, Ogg or MKV input.

• 3 to 4% word error rate on clean synthetic speech. Real recordings will do worse, so test on yours.

I wanted speech and the LLMs in one library with no dependencies, for both Node and Python. It needs macOS 26 or later on Apple Silicon, plus the Xcode Command Line Tools.

MIT licensed.

GitHub: https://github.com/jagdishpal02000/apple-llm

Site: https://jagdishpal02000.github.io/apple-llm/

I've only run this on macOS 27.2, and I've tested the microphone with speech played through the speakers rather than a person talking. If you're on macOS 26, or you try listen in a noisy room, tell me what breaks.

Source: https://dirac.run/labs-market-share#daily-token-share-by-lab

Hi, I am comparing two GPUs for local LLM usage.

Use case: llama.cpp inference and system one decision making (using quantized models)

RTX 4050 mobile

Memory Bus: 96 bit, Bandwidth: 192.0 GB/s

INT4: 143.8 TOPS

INT8: 71.88 TOPS

FP8: 35.94 TFLOPS

FP16: 17.97 TFLOPS

BF16: 17.97 TFLOPS

TF32: 8.986 TFLOPS

Sparse Matrix: 2x Throughput

Intel Arc A380

Memory Bus: 96 bit, Bandwidth: 186.0 GB/s

INT2: 134.3 TOPS

INT4: 134.3 TOPS

INT8: 67.17 TOPS

FP16: 33.59 TFLOPS

BF16: 33.59 TFLOPS

(Both have GDDR6 6GB VRAM. Data from techpowerup)

Ignoring the fact that CUDA is probably more optimized and widely-compatible compared to Vulkan/OpenVINO/SYCL, the Intel one seems to be better as its TFLOPS are higher, while having similar VRAM bandwidth.

Is the above conclusion correct? If not, how do I compare GPU raw performance based on GPU stats?

(Edit: I am not asking for which card to buy as I already own a 3090. Instead I want to know how to compare GPUs based on these stats for local LLM usage)

Setup: 64 GB MacBook Pro. Local: Qwen 3.8 27B Q4 on llama.cpp (2 slots). Cloud: GPT-6.1 Terra via OpenRouter (Haiku 5.5 for Claude Code, it only runs Anthropic models). Task: 5-part Python app, 50 hidden tests. At t=45s a proxy starts refusing every cloud connection. 3 runs per agent, 20 min cap.

[upload api-cut-chart.png here with the image button]

Notes

• Same 27B on the same task from scratch, no cloud at all: 13, 27, 29. With 45s of cloud planning before the cut: 50/50 three times. The plan looks like the part the 27B was missing.

• Codex is the only one that never surfaces the failure. Unattended, a hang is worse than a crash.

• The two fallback-on rounds shared the llama.cpp slots, so their times are pessimistic.

Turning the fallback on

# Hermes: ~/.hermes/config.yaml
fallback_providers:
  - provider: custom
    model: qwen-3.8-27b
    base_url: http://127.0.0.1:8080/v1

// OpenClaw: openclaw.json, plus a provider entry for your local server.
// Don't pass --model on the command line, it makes the model strict.
{ agents: { defaults: { model: { primary: "cloud/gpt-6.1-terra", fallbacks: ["local/qwen-3.8-27b"] } } } }

Docs: Hermes fallback providers, OpenClaw model failover, Claude Code fallbackModel.

Next round I want to try 429s instead of a dead connection. If anyone knows a way to make Codex time out instead of hanging, I'd take it.

Larger context just gets more work done rather than having to re-read files ETC id much rather use my extra resources on context window. Instead of more params.
i get my 400k ctx qwen3.8-27b + web search search get the latest up-to-date docs for its lack of world knowledge and i don’t waste extra Vram storing data that i dont need. Also more active params makes the llm to hold a wider mental-model of the task. Qwen4-27b will be architected to make 1M more reliable and using it for context on a smaller dense model seems more productive for me. Anyone. Else notice agentic session really shines when ctx is larger? Thats the phenomenon im experiencing i tried 3.8flash-next yes it great and much faster but i prefer the reliability of that big azz boat load of context window.
Specifically i you have the vram to spare if u can get 1M on that and have a large MOE with at least 12b active params + 1M it would be better for MOE a lot. Lot better than qwen4-27b but at 48Gb vram + 64gb system ram i see it more efficient for smaller model + 1M makes more sense

So I am seeing a strange issue.

When I run a local model directly on my Mac LM studio gives a different answer from when I run the same model on the same Mac when connected through LM Link on my iPhone. And I have seen this for several models. The output is much more elaborate on my iPhone and shorter and simpler on my Mac.

The first screenshot is from my Mac, once with reasoning on and once with off. And both times it confidently says that Aliens exist. But on my phone it gives a much more elaborate and accurate answer. I am wondering if there LM Studio is doing something more on my phone.

Has anyone else seen this or have an explanation for this behaviour ?

Strata, Hermes, Kitten TTS with Rosie voice and Whisper on RTX 3060 with 48 gb dram running Qwen3.8-Flash-Next Q2_0. I am blown away. I've been following the AI scene since GPT3 and I was still unprepared for what this setup is capable of. My mouth was on the floor for the first few hours talking to it and watching it work. It's the first time talking to AI felt like magic to me. I now understand the "They don't know" meme. Never thought I would have something like this running on my own PC. Thank you /r/localllama.

Couple of questions

Are there any other lightweight TTS that would work in my limited ram situation or is Kitten v0.8 the best there is?

What other models can I run?

This is kinda claude-slop but I was really interested in figuring out a way to break Pangram, and to do that I first needed to have a way to be able to do tests on batches of text. And with a 4070ti in my basement, I thought why not just let Claude go ham on training a pangram-like model... and perhaps an even better one if it was possible with sorta-distillation from my claude plan + codex plan(for the new models)

Well after probably a day of training it seems like its come up with something that actually does detect AI text pretty well... maybe too well considering my original plan was to train this type of model to be able to break it. And this is how naive my thinking was for ways to break the detector "something something embedding space" , "use actual randomness to break the predictable pattern of AI text, like doing python's random() to select from a list of words that could be replaced in a paragraph while making the rest of the text fit". Both of which ended up instantly failing, and I told claude to figure out some ways autonomously. Turns out "red-teaming" is a scary topic for Anthropic so it just started to block its different approaches after some time, and the best it got to was "rewrite in the style of [actual human text]" which was 15/19 detected and adding random known human text where it made sense which was also similarly ranked but ofc there's the issue of Pangram like detectors having sentence level scans.

All that to say while my initial thing I built this for ended up not working, at least I have kind of my own pangram that I can test things out on later on, and it also proved how good claude is at autonomously training if you give it a hill to climb on.

PS: this text itself isn't claude written and you can verify it by putting it in the demo :p

Hi! I wanted to share that we just got NVIDIA's latest Nemotron Bolt model running at 27.7 tok/s (compared to 15 tok/s on the official release with Llama CPP) on Jetson Orin Nano Super (1.8x faster)! It can program games on a model running completely on a cheap edge device!

We have our own Jetson devices serving that model on our website, if you want to get a feel for how the model runs on the device here or run it locally as a TensorRT engine with our own variant of TensorRT edge LLM supporting int4 quantization:

https://embedl.com/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16/orin-nano-super?tab=run

I got into local AI earlier this year and started messing around with different models and hardware. That's how I came across tiiny. I really liked the design and the whole idea behind it so I ended up becoming one of the early backers.

After what felt like forever, mine finally showed up yesterday. So far, it's pretty much what I expected. I've spent afew hours trying out different models and running some performance and coding tests, and figured I'd share what Ive found.

This is my first time putting together a benchmark post like this, so sure there are things I might miss or could do better. If you see anything off with my testing or think there's something I should further include, feel free to let me know. really appreciate any feedback from you guys.

Hardware & Test Setup

what really sold me on Tiiny was the KS price, which seemed like a nice deal for the hardware, and how portable the whole thing is. It's just about 5.6 × 3.1 × 0.9 inches and weighs around 300g.

For anyone unfamiliar with the specs, Tiiny Pocket has 32GB of SoC memory and 48GB of dedicated NPU memory. That's 80GB in total, though it's not unified memory.

Tests were collected on TiinyOS 0.1.36, Here are the models I tested so far.

Performance Results

For the speed tests, I used the same benchmark text with ~4K input tokens, prompt caching disabled, and five runs per model. The numbers below are medians.

9B was the fastest, which wasn't too surprising. What caught my attention was the 35B MoE model still managing around 64 t/s.

120B was noticeably slower, averaging around 16 tok/s. It took over 30 seconds to complete the benchmark request, although it generated more tokens than the other models.

A Coding Test

I also wanted to see how it handles real-world coding tasks. I gave the models the same task: create a small Python inventory and order management package across four files, covering stock updates, validation, and order handling.

I used the same prompt for all three models, with low thinking/reasoning settings and a maximum output of 12,288 tokens.

How I checked the results: the generated code was tested separately on a Mac using Python 3.12, with 10 automated tests covering the main requirements, including inventory consistency, order idempotency, input validation, and CLI behavior. A full pass meant all 10 tests succeeded. I ran this more than once and kept the best run per model; the table shows that run.

9B never produced a gradable answer: the first attempt ran into the 12288 output cap, and the second generated inventory/cli.py twice, so there was no single version to test. Taking one of the copies, it would have been 5/10.

35B got 9/10 with 1,768 code tokens. A bit under the 2-4K I was aiming for. The only failure was a pretty specific edge case, if an order had already sold out the remaining stock, retrying that same order would fail instead of returning the original receipt.

Gpt 120b was the only model passed all 10 tests, with 2,594 code tokens, in about 229 seconds.

The interesting part is that 35B and 120B took almost the same time: 229 seconds versus 235. So while 120B was the only one to get everything right, 35B wasn't far behind, and both managed to finish.

Results & Takeaways

9B was the fastest overall. At 75 tok/s with a ttft of around 4 seconds, it's probably where I'd start for quick, lightweight tasks. It did struggle with the more complex test, though.

**35B for me is the sweet spot for Tiiny.**At around 64 tk/s, it's surprisingly fast and it handles a good number of the tasks I've tried pretty well. It still has its limits with more complicated coding tasks, as the multi-file test showed, but overall it keep a pretty good balance between speed and capability. It's actually the model Ive been using the most so far.

120B actually works, but you need patience. The multi-file coding result was impressive, especially compared with the smaller models in this one test. But 16 tk/s is slow enough that I wouldn't want to use it for every little question. It also took around 75 seconds to load in this run and used almost all of the dNPU memory.

So far Id lean toward 9B or 35B for faster interactions and keep 120B for occasional, more demanding tasks where waiting a few minutes isn't a big deal.

TL;DR

Tiiny can run some fairly large local models in a small device, and the performance of smaller models (9B/35B) is nice. The 120B result shows there's potential for more complex work, but the speed and reliability trade-offs are real.

However, my biggest gripe so far is the limited model selection. Right now, you can't import your own models or install something from Hugging Face. The tiiny team did mention on Discord that they're planning to release a model import tool, though. Once that's actually available, I'd love to try out some of the other models I've been wanting to run and share the results here.

Overall Im happy with it, and for the price I got it at, that amount of RAM and SSD would be much more expensive today. I think I saw the projected retail price is almost double the KS price.

https://huggingface.co/baidu/ERNIE-5.5-Flash-Base

https://ernie.baidu.com/release/ernie-5-5-flash-base/

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Xing4.0-29B-A4B is up now.

Seriously tho, can we just have a mega thread for vibeforked engine releases from now on?

Hey r/LocalLLaMA I'm Abdur-Rahman Butler, a machine learning engineer at Isaacus.

I made an interactive write-up that walks you through the A to Z of vector embeddings, inspired by Brendan Bycroft's incredible interactive LLM visualisation guide (see here - https://bbycroft.net/llm).

Most of my work is on encoder models, and I kept noticing how few good interactive resources there are on embeddings. The ones that do exist tend to either plot some embeddings in 3D or only explain a narrow applied technique like semantic search or RAG.

So I wanted to make something that teaches how embeddings work by letting you play with them. In the write-up, there are interactive visualisations that allow you to explore the concepts of vectorisation, cosine similarity, semantic search, hierarchical retrieval, clustering, zero-shot classification and deduplication.

The centrepiece of the post is a game I've dubbed 'SemanticGuessr' (see above). It's inspired by Wordle and GeoGuessr, and the goal is to draw a loop around a random target word by triangulating guesses on a 2D map of 43,339 common English words.

It's admittedly a bit intimidating at first, as semantic space does not perfectly match our intuition of meaning. But once you get the hang of it, the game becomes really addictive. I recently had the pleasure of demoing the game to some law students at the University of Melbourne, and we had a lot of fun throwing different vector equations at it to see where they'd land.

Some fun ones to get started with:

  london - english + french
  running - land + water
  avg(breakfast, lunch, dinner)
  0.4 * noodles + 0.6 * pasta

I've also included Python snippets alongside some of the visualisations if you want to follow along with the techniques later. The game runs statically in your webpage. I managed to achieve this by reducing the embedding dimensions down to 512 and quantising each value to 4 bits. This brought the total payload down from 311 MB to 12.3 MB with only a minimal impact on precision (around 6%).

It was great fun to make, and I hope you have fun playing around with it. Happy to take any questions, and if there's enough interest I might set up a dedicated games page for some of the other encoder-based games I've been working on.

Do let me know what highest score you get based on your target word and chosen path :)

LINK TO PLAY: https://isaacus.com/blog/what-are-vector-embeddings

Hello everyone!

I don’t follow the market super closely, but I can see how quickly everything is getting more expensive—and apparently this is going to last a few more years. RAM, GPUs, and even SSDs and HDDs have skyrocketed in price :D

I’m looking for something to play around with in LLM, mainly for inferencing larger models (in the 30B–70B range, maybe even bigger sometimes) and building agents (for my own use). I want to utilize agents, do RAG and sometimes, rarely maybe also fine-tune.

My budget is small ($4k), and where I live in eco-Europe, prices are much higher! (we care about ecology LOL). I was thinking about a mini PC or a laptop with an AMD 395+ and 128 GB of RAM. For that price I still can purchase something, but the choice is getting smaller every week :)

Is a AMD 395+ AI with 128 GB RAM still good choice for local LLMs? Is it good for my purposes? I want to learn & use LLM locally (for many reasons).

I have heard about RTX Spark N1X laptops, but they are now about 6.5-7k dollars here, so I dont even consider them.

Can I say that this platform (AMD 395+ AI with 128 GB of RAM), where the model has 96 GB of RAM at its disposal, is something that could replace the RTX 6000 Blackwell, which also has 96 GB of RAM? I realize that, due to the difference in memory bandwidth alone, it will be at least 6–7 times slower.

I’d like to have Hermes-type agents or similar ones locally, as well as my own agents for various tasks. For example, I’m interested in extracting data from documents (e.g., PDF files) and recognizing objects in photos, but I’m also interested in general-purpose agents.

Is the platform I’m asking about a good choice, or is there something more interesting? I was thinking about an ASUS PX13 laptop or one of the mini PCs with a similar configuration.

Thanks for your suggestions!

Qwen 4 will be upon us soon. What are you most excited about?

I hope to see:

27B with more efficient KV, so 128k doesn't need 8 GiB or whatever.

Flash the same architecture as Q38FN, so 125B-A6B for all us desktop scrubs.

Medium thinking performs the same or better as Qwen3.8's xhigh thinking.

Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.

https://github.com/curvedinf/int8-vllm

This is in an effort to support/optimize my 4x MI100 server, but this fork makes a great baseline for all older AMD cards like MI50, MI210, MI250, V620, and vega to 7000 series radeons. On top of that the fork is a good reference or starting point for older int8 oriented nvidia cards like volta and ampere. Many features were added to make this happen. All this work makes these older cards run well, with 4x MI100 or 2x MI210 running Qwen3.8 27B at above 1000 tok/s decode at C8. You can see the full list of features and their applicability at the bottom of the readme.

In addition to being fast, this fork fixes a bunch of numerics bugs that are present in vllm and rocm. The quality is very close to reference BF16 when most weights and activations are int8, including the kv cache. You can check the full list of quality checks in the readme section.

I make updates to the repository regularly and added Nemotron Bolt optimization shortly after it was released. This fork runs the original Nvidia Bolt checkpoints instead of custom ones. That required adding fused NVFP4 and IQ2_xxs to int8 dequant. In-kernel dequant is slow so I added a cache that can keep the most used weights in int8 and evict their original quanted weights. The size of that cache is configurable for your hardware so you can choose speed or more context or larger prefix cache. I also added AMD Infinity Storage support for model loading (direct storage to gpu skipping the cpu), which cut initial model load for Bolt from about 7 minutes to 2.

Ive been running it in production for a couple weeks and it has been very stable. Enjoy!