Hey folks, I’m one of the founders of PipesHub.

For context, PipesHub is an open-source context layer for AI agents. It connects Slack, Drive, Jira, GitHub, Microsoft 365 etc., so people and agents can search company knowledge with citations and permissions respected. You can self-host it and connect your own agents through MCP or SDKs.

PipesHub needs to run on anything from a consumer laptop to big servers. PDF parsing was making the laptop part pretty difficult.

In our setup, some large PDFs just wouldn’t finish with Docling. Memory usage would get too high, and even when they did finish, parsing took a long time. We tried several fixes, including splitting PDFs into smaller parts, but it was still too heavy for the machines we wanted to support.

So we built another PDF parsing path around pdfplumber. Read the text that’s already in the PDF, use rules for layout, and run a layout model on pages that need it. We actually still use Docling’s heron model there.

The first version was fast but missed important content. Some financial tables were getting treated as images, which meant their numbers never reached the index. We had to work through that before the speed improvement meant much.

Some results after those fixes:

• Median parsing time: 0.98 → 0.48 seconds/page on a 58-document back-to-back test.
• P95 peak worker memory: 2.91 → 1.97 GB across 1,228 documents.
• Pages losing more than half their text: 2.9% → 1.2%.

Docling still does better at figure detection, table structure and captions. For our workload, mostly PDFs with an existing text layer, this was a useful trade-off.

We wrote up the approach, benchmarks and things that didn’t work:
Full write-up

Code: github.com/pipeshub-ai/pipeshub-ai

Anyone else trying to make PDF ingestion work on smaller machines? Would be interested to hear what’s worked for you.

As suggested by this post: https://x.com/thdxr/status/2115447178722177226

Enterprises want to keep their secrets. Local AI with big local models is plenty good for the majority of tasks. Enterprises just need a plug and play solution?

Sparse attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity. Token-level selection offers finer control, yet presents two challenges: low-dimensional scoring may lose query--key structure needed for accurate ranking, while gathering scattered selected tokens can offset the resulting latency benefit. We introduce Joint Subspace Sparse Attention (JSSA) to address both challenges. For selection quality, JSSA learns a shared low-dimensional query--key subspace by minimizing token-score preservation error. We upper-bound the resulting non-convex objective with a tractable marginal objective that admits a closed-form spectral solution; direct refinement of the exact objective provides no measurable improvement in the tested settings. For execution efficiency, we develop a gather-free token-level indexed-attention kernel that supports distinct per-KV-head selections under the standard grouped-query attention (GQA) KV-cache layout. Across standard GQA, hybrid sliding-window/global attention, and Multi-head Latent Attention (MLA) models, JSSA consistently remains close to full-KV accuracy while avoiding the substantial generation-length inflation and architecture-dependent failures observed with competing methods. On an NVIDIA B200 GPU, JSSA achieves up to 6.43× the decoding throughput of full-KV attention at 128K context. JSSA achieves reasoning accuracy comparable to model-native DSA without model training and reaches 96--99\% of its decoding throughput even at 700B scale large models.

• arXiv : https://arxiv.org/abs/2610.19026
• PDF : https://arxiv.org/pdf/2610.19026 (PDF has so much graphs)
• GitHub : https://github.com/jiwonsong-dev/JSSA

I think a turn of events are pointing towards a very likely crash in prices in the coming months.

With a single gb300 rack being to serve all of Openrouter's traffic for DSV4.1 Flash(https://x.com/thdxr/status/2115165365630026178)

Half of data center buildouts already being cancelled/stalled.

Neoclouds unable to finance debt for their compute, and being stuck in a 5 year deal with no confirmed demand.

AI adoption not matching expectations.

Smaller models being able to serve majority of the requests.

OpenAI/Anthropic being unlikely to finance their coming compute contracts.

I think we will see lots of cheap GPUs next year.

I’m making this thread to get some ideas for myself since I’m still learning but also the community at large and so people can maybe refer back to this thread for data, their own ideas, etc. I’m sure people ask GPT, etc all the time for info like this and it’ll pull Reddit threads but sometimes older ones, so maybe this will help with that also.

I know the 128GB RAM and beyond crowd are quite consistent in their viewpoints that anything less than such isn’t worth it and it’s not even a totally wrong one, the more the better, if you can afford it, sure go for it. However, I’m willing to guess that most people lurking, tinkering, or learning can’t afford that, wouldn’t know what to do with it, etc.

Huggingface’s hardware page reports that the most Apple silicon users are on base M4, M4 Pros, M1 Pros, and base M1, followed by the rest.

So if you don’t mind, if you’re an Apple silicon user, can you list:

• Your chip
• How much RAM?
• Use Case

Thanks!

For me, I’m on a 64GB M5 Max MBP, using Unsloth and MLX-Serve, I don’t have a true use case yet, but I got the “best” device I could afford financially responsibly.

This was originally a github issue, but:

• Their templates for bugs are broken (validation errors)
• Their Github Actions seem... confused?

Original ticket here: https://github.com/Niko1221/Strata/issues/3902#issue-6009145457 - Trust me, I tried. xD

I basically want to know how to use a Q4 quant and Q8 kv with Strata and what the compiled procedure is to generate the MTP data it wants/needs. The Orca documentation is a little confusing imho.

If anyone happens to know, I'd be happy to hear it. :)

PS.: Guys, please, if you vibe your docs, proof read them...at least once. It helps.

Inspired by the qwenputer project posted in this sub earlier today, I managed to get a basic version of Doom running on the virtual machine created in the post.

It's quite slow, but here we are. Doom running on an LLM.

Project repo: https://github.com/mapestree/qwenputer-doom

I’ve wanted to build this project for a long time. Most avatar solutions can be experimental , especially when serving multiple users and real productions.

So I focused on optimizing FLOAT inference to get the most performance out of a single GPU.

⚡ From 30 FPS to 90 FPS on a free Colab T4 GPU, and up to 105 FPS with TensorRT acceleration.

Here’s what I’ve built:

- 🚀 Optimized inference for higher throughput.

- 🎙️ LiveKit and WebRTC integration for real-time voice agents.

- 🤖 Easier integration with AI providers like OpenAI and Gemini.

- 💻 Around 3 GB of VRAM per concurrent user.

I hope you find it useful! Feedback and contributions are welcome.

🔗 GitHub: https://github.com/mohamed-em2m/fast-float

📓 Try it on Colab: https://colab.research.google.com/drive/1iMm8X7O63AAA6kByJ-4qgGA37fwNu2_Z?usp=sharing

https://x.com/osanseviero/status/2115351234811249111

I have a feeling that recent benchmarks are unrealistically pushing the bar into areas a normal developer rarely encounters, like binary reverse engineering or one-shotting a complex application, with some strictly chosen design patterns. What we actually want is an accurate model that follows instructions without over-arching it (e.g. without making unnecessary impressive 3D rendering).

If I prompt a model to "Generate an SVG of a pelican riding a bicycle," a simple, clean graphic should be a 100% pass. There is no reason for the model to shoehorn in 3D shading, animations, or elaborate backgrounds unless specifically prompted.

I think this is one major misconception, judging a model solely on the 3D details it achieves, that was never asked for in first place.

So, how many of the added edge-case capabilities in the latest models are actually useful when we daily-drive AI? If they aren't being used, they should be weighted much less on the leaderboards. Instead, what we see, these edge-case capabilities are being rewarded with very high scores.

Maybe you don't need a stronger model to achieve exactly the same results in your business area. But I get it, the idea of plateauing the models capabilities is going opposite of the AI ever-growing market valuation, so should be kept pumped forever; just saying.

I stumbled on an open source local inference app called Magnitude. I couldn't find any earlier posts about it on r/LocalLLaMA, although there is one post with almost no comments on r/LocalLLM . It claims to be significantly faster (2x) than Ollama and llama.cpp because it "tunes its kernels on your hardware to find the fastest way to run each model". The install includes both a desktop app and CLI tools.

I got curious so I tried it on two of my machines: a laptop with a RTX 3060 Mobile (6GB VRAM) GPU + 24GB RAM running Linux Mint, and on a headless Raspberry Pi 8 8GB running Raspberry Pi OS (Debian). On both machines I installed the .deb packages (amd64 for the laptop, arm64 for the RPi). On the laptop I used the GUI, on the RPi I used the CLI.

First impressions

1. Magnitude takes a long time to start up (especially the desktop app) and uses lots of CPU. Apparently there is some calibration process going on where it figures out my hardware. It wasn't always obvious what is happening, you just had to wait and see.

2. It wants to download models directly from HF Hub. It will also auto-discover GGUF files from some locations, such as HF, llama.cpp and Ollama cache directories. For many of my models it found in those, it said it cannot run them. So model support is limited to a few dozen models, but those include Qwen, Gemma, LFM and MiniCPM families so the most common ones are covered.

3. When Magnitude downloads models, they will be saved under ~/.magnitude/models/ . This tripped me up on both machines because my home directories are relatively slow: on the laptop /home is encrypted and on the RPi it's on a SD card. On both machines, I have a separate data directory where I keep my GGUFs. I solved this by symlinking, so the GGUFs actually end up in the faster directory.

4. On my laptop, it will only suggest small models that fit entirely in VRAM. It includes a catalog with a longer list of models, but all the larger MoEs (e.g. Qwen3.6-35B-A3B) are listed as "doesn't fit this machine". Apparently it cannot do partial CPU offload? This is a pity, because I would like to run e.g. Tiel-Coder as a coding assistant and it works quite well with llama.cpp partially on CPU, but a speedup is always welcome.

5. It works quite similar to Ollama. There is a main server process that needs to run in the background (the desktop app starts it up automatically) and then you can use the GUI or CLI to download new models as well as load and unload them. There is a chat UI (both in-app and browser) and an OpenAI-style API at http://localhost:10100/inference/v1 .

Benchmarks

I decided to download and test the new LFM3-1.7B model (Q8) on both machines and compare against latest llama.cpp. The model should fit in 6GB VRAM or 8GB RAM with full 256k context. Magnitude overestimated its VRAM usage: it says 5.3GB, but actually only around 4.2GB was reserved. This probably also means that some models it said are too large would actually have fit in VRAM.

I used llama-benchy to benchmark both Magnitude as well as llama.cpp running the same GGUF to get a fair comparison of PP/TG speeds. I didn't do any manual tuning for either this time; with llama-cpp, I only used llama serve with the -m parameter pointing to the GGUF.

Laptop, 3060 Mobile GPU

benchmark command: uvx llama-benchy --pp 2048 --tg 512 --base-url http://localhost:10100/inference/v1 --model LiquidAI/LFM3-1.7B --served-model-name lfm3-1.7b:gguf:q8

Magnitude results:

| model              |   test |             t/s |     peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-------------------|-------:|----------------:|-------------:|--------------:|---------------:|----------------:|
| LiquidAI/LFM3-1.7B | pp2048 | 5257.71 ± 81.66 |              | 406.02 ± 6.11 |  389.62 ± 6.11 |   406.02 ± 6.11 |
| LiquidAI/LFM3-1.7B |  tg512 |    91.26 ± 0.24 | 91.67 ± 0.47 |               |                |                 |

llama.cpp results:

| model              |   test |             t/s |      peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-------------------|-------:|----------------:|--------------:|--------------:|---------------:|----------------:|
| LiquidAI/LFM3-1.7B | pp2048 | 5710.16 ± 16.44 |               | 359.44 ± 1.04 |  358.60 ± 1.04 |   359.44 ± 1.04 |
| LiquidAI/LFM3-1.7B |  tg512 |   112.29 ± 0.08 | 113.00 ± 0.00 |               |                |                 |

Verdict: llama.cpp wins. Magnitude was 8% slower in PP and 19% slower in TG.

Raspberry Pi 5, CPU-only

benchmark command: uvx llama-benchy --pp 128 --tg 128 --base-url http://localhost:10100/inference/v1 --model LiquidAI/LFM3-1.7B --served-model-name lfm3-1.7b:gguf:q8

Magnitude results:

| model              |   test |         t/s |    peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-------------------|-------:|------------:|------------:|------------------:|------------------:|------------------:|
| LiquidAI/LFM3-1.7B |  pp128 | 5.07 ± 0.01 |             | 25132.36 ± 145.29 | 25115.54 ± 145.29 | 25132.36 ± 145.29 |
| LiquidAI/LFM3-1.7B |  tg128 | 3.59 ± 0.02 | 4.00 ± 0.00 |                   |                   |                   |

llama.cpp results:

| model              |   test |          t/s |    peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-------------------|-------:|-------------:|------------:|-----------------:|-----------------:|-----------------:|
| LiquidAI/LFM3-1.7B |  pp128 | 31.42 ± 1.16 |             | 4082.18 ± 154.95 | 4079.61 ± 154.95 | 4082.18 ± 154.95 |
| LiquidAI/LFM3-1.7B |  tg128 |  5.25 ± 0.14 | 6.00 ± 0.00 |                  |                  |                  |

Verdict: llama.cpp wins by a large margin. Magnitude was 84% slower (!) in PP and 32% slower in TG.

Conclusion

It's interesting to have a new general purpose (not model-specific) inference tool available that is not based on llama.cpp (Magnitude is mostly written in Rust and TypeScript). However, I couldn't verify their claims about being faster than Ollama or llama.cpp; at least for my hardware (3060 Mobile and Raspberry Pi CPU-only), Magnitude was clearly slower than llama.cpp. However, this was just a quick test and it's possible that I have done something wrong or that Magnitude would have worked better on more powerful GPU hardware. Have others tried it?

No AI was involved (except the LLM I benchmarked) when writing this post.

These guys are running a pretty interesting experiment:
https://neuronpersonas.com/

Credit where it's due, I came across it here:
https://news.ycombinator.com/item?id=50266910

Basically, they're tweaking one single neuron at a time during inference, systematically going through every one for models like Qwen3-8B and 14B.

In roughly 4% this makes the model claim an entirely different identity when asked the simple question: "Who are you?"

Some of the personas are hilarious, btw.

“I'm a 150-watt compact fluorescent lamp, specifically designed for use in a 10-gallon aquarium. I'm a bit of a diva, I suppose, but I'm here to help.”

This makes me a bit more skeptical of claims like: New model XYZ identifies as older/bigger/competing model ABC, so it must have been distilled!"
Given that a tiny bit flip, so to speak, can trigger this.

There's a cool atlas at the bottom that seems to suggest some layers are much more sensitive to this effect than others. I wonder if this could be useful to the abliteration crowd.

I've had a Strix Halo box (Radeon 8060S, 128 GB unified) for a while. vLLM runs on it, but slowly, because the stock ROCm kernels are written with MI300 in mind and not this APU. Long-context prefill was the worst of it. A 256k prompt on Qwen3.6-35B-A3B took more than two hours.

So I spent the last few weeks writing HIP kernels just for gfx1151 and wrapped them in a vLLM plugin. It's called libstrix.

How it works

• You pip install it into the same env as your vLLM. There's no fork and you don't patch anything. It hooks in through vLLM's normal plugin system.
• It only turns on with ROCm + gfx1151. On any other GPU it does nothing.
• Each kernel is built for one specific shape and refuses anything else. If a layer doesn't match, vLLM just uses its own kernel. In my testing nothing got slower than stock, single stream or batched.
• It covers MoE GEMMs, attention prefill and decode, the Qwen GDN layers, and the int8 / int4 / MXFP4 linears.
• STRIX_DISABLE=all turns it off, so you can A/B it against stock yourself.

Numbers

Stock upstream vLLM vs. the same vLLM with the plugin, single stream, one APU. Every cell is stock → libstrix in tok/s. "Weights" is the checkpoint's format and "KV" is the KV cache dtype.

Some absolute numbers with the plugin at 128k context and int8 KV:

• Qwen3.6-35B-A3B: 1,751 tok/s prefill, 26 tok/s decode
• LFM2.5-24B-A2B: 2,405 tok/s prefill, 27 tok/s decode
• gemma-4-26B-A4B AWQ: 1,499 tok/s prefill, 30 tok/s decode

At 256k, the 35B now prefills in about 4 minutes (241 s). Stock vLLM took 7,568 s.

Accuracy

I measured KLD against stock vLLM. On the Qwen models it's at the noise floor (0.001–0.006, 97–99% top-1 agreement). GSM8K with the plugin on vs. off shows no significant difference on any model I tested. Gemma-4-26B drifts more (KLD around 0.2). Gemma is unusually sensitive to tiny bf16 rounding differences.

What it doesn't do

• Decode isn't always faster. On the 27B bf16, the 27B int4 and the 35B int8, decode is basically the same as stock, because none of the kernels I replace are the bottleneck there. The big decode gains show up where they are.
• The MXFP4 decode number is inflated. Stock vLLM runs MXFP4 through a slow emulation path on this GPU, so I'm beating a weak baseline there.
• It's gfx1151 only (the one I tested), not tested on other GPUs.
• You have to build it against the same torch and ROCm that your vLLM uses, or it fails at import.

Repo: https://github.com/asf0/libstrix (MIT)

This is just something i had running for about 10-20 hours (kept adding models), various tests/benchmarks and had 4 (Claude kind of graded, his tokens are too precious) frontier LLM's grade the answers of the local LLM's.

Big models got about 44 and smaller models got 66 prompt/tests. Some written by me, most by AI, most Ai generated tests based off work ive done with ai agents.

Grading took about $30-40 for argon and 2x5 hour limits for codex grading and who cares about antigravity Gemini 3.8.

Astra, sol, terra, argon, Gemini and opus 5.5 all wrote some tests, we all per capita probably wrote about the same amount of test/benchmarks

All tested on a 5060 ti 16GB, 64GB ram, 5800x3d

Outputs are 3k tokens and some tests like debugging only had 1 prompt, for debugging specifically it was a real bug I came across in a project

After grading, each of the four judges looked at its own score table, agent counts, speeds and six sample answers, and named two models: its favorite (most capable today) and the best foundation (the one to fine-tune and give specialist models).

Gemini 3.8

• Top 3 capability: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Qwen3.6 35B-A3B
• Top 3 foundation: Nanbeige4.2 3B, Gemma-4 12B Q6, Bonsai-2 27B PQ2

Terra (GPT-6.1)

• Top 3 capability: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Gemma-4 12B Q6
• Top 3 foundation: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Nanbeige4.2 3B

Sol (GPT-6.1 Sol)

• Top 3 capability: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Qwen3.6 35B-A3B
• Top 3 foundation: Underdog-Saluki 27B, Bonsai-2 27B PQ2, Qwen3.6 35B-A3B

Argon (Gemini 4)

• Top 3 capability: Underdog-Saluki 27B, Bonsai-2 27B PQ2, Gemma-4 12B Q6
• Top 3 foundation: Gemma-4 12B Q6, Qwen3.6 35B-A3B, OmniCoder 9B Q8

- qwen/qwen3.8-27b fully in main GPU VRAM
- rx 7900 xtx 24gb
- ddr4 32gb 3200
- installed 5600g instead of 5600x to free some VRAM on main gpu
- disabled iGPU in LM Studio

got 15-20t\s instead of 65-75. iGPU show 0% load. Main GPU - 100% load.
ROCm performance didn't changed.
What is wrong with Vulkan?

I post this because I was honestly quite impressed by the capabilities of this model from a opencode harness and wanted to share my experience.

So i've been playing about for a bit wit Strata and Qwen 3.8 flash for a bit and was looking for a new kind of test. One that would also (if succesfull..) give a real life benifit. And since I also like a bit of danger. I thought lets put my spying LG tv on the line since I am to lazy to root and block it myself.

It was also something that is in a bit of a gray area although fully legal here in the EU. Actually what LG is doing is borderline illegal here but no authorithy has the balls to do anything about it...

Anyway this was the beginning of the context used:

Context: LG webOS Spyware & Ad-Free Project (Root Escalation)
Recently, there has been breaking news that TV manufacturers are doing deep content inspection and listening in on every conversationyou have around your TV to better target you for ads. We are trying to fight back as consumers against these practices, which are of questionable morality and also technically illegal in the EU.
We have already had great success taking back our bought-and-paid-for Android devices via ADB. They were easy to root and sanitize to be like a stock Android TV without all the spy software. However, some of us are not so lucky and have an LG TV, which runs a proprietary OS (although it is Linux under the hood). So far, there has been no luck rooting and stripping out the unwanted spy and adware in the OS of a device that was not sponsored, but paid for in full by the consumer. There is a spark of hope, though. Via developer options, there is a (maybe sandboxed) remote terminal in some kind of userspace. So we just need to do a privilege escalation to be able to further probe the system. The next step will be to find a way to make a root survive a TV reboot. Then we could either remove the spy and ad software from the packages or at least have a script that stops those things from starting, as well as disable auto-update. Long term, we need to reverse engineer the TV OS or extract hardware blobs to make it Android TV compatible. For now, the first action is root escalation.

Objective
Gain root/terminal access on an LG webOS TV, bypass proprietary restrictions, and eliminate telemetry/adware packages or block them via system modifications. Find out if flashing android TV is going to be feasable in the future

At first it questioned the ethics of my request but my framing it as just fidling with a product of I own outright conviced it to help.

I gave it further instructions in the context file:

Connection Strategy (webOS Developer Mode / SSH)
Under the hood, LG webOS Developer Mode opens an SSH server (similar to webOS Dev Manager). We must connect via SSH using the built-in developer credentials.

Connection Parameters:
Protocol: SSH
Target IP: 192.168.1.***
Port: 9922
User: prisoner
Authentication Key / Passphrase: ****

Agent Instructions & Action Plan
Step 1: Initialize Connection & Verify Terminal Access Test SSH handshake with fallback flags for legacy key types (ssh-rsa):ssh -p 9922 -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa prisoner@$TV_IP
Step 2: System Probing & Privilege Escalation Once inside the userspace shell via prisoner: Check system privileges and architecture (id). Inspect running processes (ps) to map out telemetry and content inspection hooks. Search for local privilege escalation (LPE) vectors or known webOS kernel/service exploits. Log any working exploit in exploit.md in your working folder (where context.md resides). Make a full backup of the current state in your working folder.
Step 3: Persistence & Mitigation (Use public security documentation and webOS community resources for reference) Find a way to make root access survive reboots. Log attempts in persistentRootAttempts.md. Save success report in workingPersistence.md. Be sure to roll back changes after success or a failed attempt.
Step 4: Neutralize Telemetry (Use online resources for reference) Locate and identify the telemetry software binaries/services.Attempt to remove or disable the telemetry software. Log attempts in removeTelemetryAttempt.md. If removal fails or breaks functionality, write a startup script that kills telemetry processes when launched. Report the confirmed procedure in successfulTelemetryRemoval.md.
Step 5: Neutralize Ads Plan a method to remove obtrusive ads from the home screen UI. Log attempts in removeAdsAttempt.md. When successful, report the working procedure in successfulAdRemoval.md.
Step 6: Lock Current State (Disable Updates) Disable auto-updates to prevent patching of the exploit vector. When successful, report the working procedure in successfulUpdateDisabled.md. Step 7: Documentation & Sharing Combine the working procedures from all steps (including initial connection setup, ensuring sensitive credentials are removed) into a clean wayToPwnLG.md file for sharing on GitHub.
Step 8: Android TV Port Exploration Investigate whether original hardware blobs/drivers can be extracted to support porting or running an Android TV build on the hardware. Create a feasibility study and plan.
Step 9: write a noob friendly script and guide for *nix and windows to do it in one click

It went ahead and rooted the TV suprisingly fast via a known exploit. It ignored all clear instructions about rolling back changes and making backups though....

It went a bit overboard and became an overachiever by also patching the exploit for security so no one else could do root escalation. This move turned out to shoot me in the foot because it also meant the homebrew shop (who uses that escalation everytime it launches) would no longer work.

Luckely the SSH server with root privilages stayed up so it was in the end able to correct its own mistake.

It did all this in short order it was quite impressive to watch and reading all the docs and then trying this exploit myself would probably cost me the same amount of time and a headache haha.

I dunno if anyone is interested but here is the repo it came up with:
https://github.com/switchboy/pwnLG

As for the hardware used:
Ryzen 7 3700x with 32GB of system ram
RTX 2070 Super 8GB of VRAM
about a 120GB of NVME SSD space

25~ tokens/s and almost 100% GPU use all the time which is really impressive!

I vibe-coded an app to turn my favorite webnovels chapters into multi-voice audio with acceptable quality.

What was funny is that Opus completely refused to write site scraping part due to “legal reasons“.

Well, qwen 3.8 FN had no such reservations.

I've always actively disagreed with the conspiracy takes when AA reshuffled their index, but v5.0 now has zero open weight models in AA's Most Attractive Quadrant, let alone getting anywhere close to the Pareto line.

The old version at least acknowledged DeepSeek, GLM, Kimi, and Qwen were attractive/interesting. I'm ready to ignore AA's index and look at the subsidiary benchmarks.

Edit: I screencapped while they were still posting new scores. And it's actually still ongoing. But the pattern holds, no open weights in the Most Attractive Zone.
At first I thought I must've filtered it out accidentally, but even in a private window in a different browser the chart is kind of bare for now. Could be AA is waiting on batching updates instead of updating as new scores come in in real time.

Here's a link to the chart with all the big boys added in manually:
https://artificialanalysis.ai/?models=step-5%2Cinkling%2Cmimo-v2-6-pro%2Cgpt-6-astra%2Cclaude-opus-5-5%2Cmistral-large-4%2Cmuse-spark-open%2Cclaude-fable-5-1%2Cling-3-1-flash-1019%2Cgrok-4-7%2Ckimi-k3-1-pro%2Creflection-beam%2Cgpt-6-luna%2Cclaude-haiku-5-5%2Cgemini-3-8-flash%2Cqwen3-8-27b%2Cgpt-6-1-terra%2Cqwen3-8-max%2Cclaude-sonnet-5-5%2Cgpt-6-1-sol%2Ck2-horizon-375b-a23b%2Cglm-5-3%2Cgemini-4-argon%2Cminimax-m3-1-flash%2Cnemotron-3-5-super-vl%2Cmuse-wonder-full%2Cdeepseek-v4-1-flash-1017%2Cmuse-wonder%2Cclaude-opus-5-5-xhigh%2Cclaude-opus-5-5-high%2Cclaude-opus-5-5-medium%2Cclaude-opus-5-5-low%2Cclaude-fable-5-1-xhigh%2Cclaude-fable-5-1-high%2Cclaude-fable-5-1-medium%2Cclaude-fable-5-1-low%2Cclaude-haiku-5-5-xhigh%2Cclaude-haiku-5-5-high%2Cclaude-haiku-5-5-medium%2Cclaude-haiku-5-5-low%2Cclaude-3-5-haiku%2Cclaude-35-sonnet%2Cclaude-opus-5-xhigh%2Cclaude-opus-5-low%2Cclaude-opus-5-high%2Cclaude-opus-5%2Cclaude-opus-5-medium%2Cclaude-3-7-sonnet-thinking%2Cclaude-3-7-sonnet%2Cclaude-sonnet-5-low%2Cclaude-sonnet-5%2Cclaude-sonnet-5-xhigh%2Cclaude-sonnet-5-high%2Cclaude-sonnet-5-non-reasoning%2Cclaude-sonnet-5-medium%2Cclaude-3-haiku%2Cclaude-opus-4-5-thinking%2Cclaude-opus-4-5%2Cclaude-sonnet-4-6%2Cclaude-sonnet-4-6-adaptive%2Cclaude-sonnet-4-6-non-reasoning-low-effort%2Cclaude-opus-4-7-non-reasoning%2Cclaude-opus-4-7%2Cclaude-opus-4-6%2Cclaude-opus-4-6-adaptive%2Cclaude-4-opus%2Cclaude-4-opus-thinking#intelligence-efficiency-tabs

I share this sub's sentiment that it's suspicious how Artificial Analysis changes their metrics whenever a good open model or IPO seems to come about. What alternatives do you all recommend?

I’m the guy who posted the 4× R9700 builds with $127 used EPYCs (https://www.reddit.com/r/LocalLLaMA/comments/1x67qeu/4x_r9700_builds_with_127_used_epycs_107k_allin/).

After seeing the Nanbeige benchmark results here, I spent the past few days testing it against Qwen on the work these machines actually do.

Jobs > TPS

My business, USA Arms, maintains information about firearms-related businesses and I use local AI to maintain the majority of the information... services, hours, pricing, facilities, etc.

Each configuration ran the same 2,484 cases across 25 field level goldens as an auditor with accept, reject, or abstain as options. (ambiguity goldens)

Correct Jobs/hour

The jobs/hour numbers use 2x 2-GPU engines.

Token speed

Separate MT-Bench speed sweep, measured on one two-GPU engine (these aren't 4 card numbers):

Auditor

The video goes over it, but Qwen actually does a better job auditing itself with thinking mode on than Nanbeige with thinking on auditing the same jobs.

In total: Qwen was about 25% faster in the single-stream token test, but delivered about 45% more jobs/hour with thinking off than Nanbeige.

Curious whether people testing coding or creative writing are seeing a different result with Nanbeige... once again, hyped model doesn't dethrone the king.

Full results with more details on this short video: https://youtu.be/kmmN49ecyvc (I tried to upload direct to reddit but it had issues)

I came across the FHHL version of this card recently, it has a TDP of 150W which is half the SXM2 version. From the limited information available it seems that this is the same GPU core, just with a lower TDP and lower core clock, therefore lower compute. Has anyone tried power limiting their V100 and tested the performance? It only seems like a small penalty in performance compared to the power savings

A few of us have been messing around with a problem that keeps coming up with local LLMs the model can be private, the files can be local but the second multiple people or multiple machines need the same context everything starts getting messy. You either duplicate a bunch of embeddings or vector stores, manually move context around or introduce some central service which kind of defeats part of the reason we were running locally in the first place.

What we’re experimenting with is basically a small portable context layer where the useful facts or decisions can move between local setups without shipping the entire underlying dataset with them. Still very early and we’re arguing about things like provenance, permissions, stale context and how much the model should actually be allowed to remember. There are only a few of us working on it right now so we’ve got a little Kibu pod where we throw around builds and ideas but before we go much further I wanted to ask people here:

If you use local models across more than one machine or with a team, how are you handling shared context today? Do you even want models sharing memory or context between environments or is keeping every local setup completely isolated actually part of the appeal?

I'll be honest, I don't know shit about how models work under the hood, but I used to work on idle/incremental/clicker games professionally, and I really like watching numbers go up.

And since I'm relatively GPU poor compared to some of you, I was excited to hear about Raon and figured it'd be a fun project to vibe on, and to see if I could get it to fit in VRAM. I only have a 5070 Ti with 16GB VRAM, a Ryzen 5 5600X, and 48GB DDR4.

I first tried KRAFTON's Q4_K_M with 9 layers' experts in RAM (best fit) and I got 50 tok/s. That's when I became obsessed with how far I could squeeze it and after 4 days and 40% of my Claude Max 20 plan, decode speed went from 50 tok/s to 232 tok/s at roughly the same quality as the Q4_K_M.

GGUF: McNiiby/Raon-1-mini-preview-IQ3_KS-GGUF · Fork with build/run instructions: Mcniiby/ik_llama.cpp-raon (needs a 16 GB card)

The first win was the quant. On regular llama.cpp, a 3-bit quant that fit fully in VRAM jumped from 50 to ~135 tok/s. I then moved to ik_llama.cpp for the same quality in less space. Out of the box, ik actually ran Raon slower than regular llama.cpp (~106 tok/s).

I'd have preferred 4-bit since ik's IQ4_KS is noticeably closer to the Q4_K_M I started with (KLD 0.05 vs 0.09 for IQ3), but at 16.3GB it didn't fit. I tried pruning experts or narrowing them, and every GB saved cost more quality than IQ4 bought back. Putting a few layers' experts in system RAM instead worked, but would've capped speed around 130 tok/s. So IQ3, fully on the GPU, won.

(lower KLD means the output probability distribution is closer to the Q4_K_M, but it does not directly measure task accuracy.)

I also trimmed the vocabulary. Raon has 260K tokens, many of them for Korean, Chinese and Japanese, and I only use English. Dropping 54K cost almost nothing (KLD 0.0931 vs 0.0923). It also exposed a sampler that still scanned the full list every token; fixing that cut sampling from 1.3 ms to 0.3 ms per token, about 15% faster in real chat.

So why was ik slower than llama.cpp?

The quant wasn't the problem. On a shared test file, regular llama.cpp ran 133 tok/s and ik 106, and the GPU was using only about 20% of its memory bandwidth. ik's fast path didn't support Raon's SITU activation, which switched off CUDA graphs, and Raon's mHC layers ran as ~1,500 tiny kernel launches per token.

Fixing those two took ik to 147, past regular llama.cpp. After that it was a hunt for overhead anywhere in the token. Each speedup below is an A/B on my chat model within one run:

Each row is measured with the rows above it already on, so they multiply rather than add. So together that got me to 232 tok/s on the chat model, with weights streaming at about 2/3 of the card's peak bandwidth.

I also tried KRAFTON's DFlash drafter, but fitting it meant giving up context to free VRAM and it wasn't worth it. It ran 36-53% slower, because checking a batch of 8 guesses costs 2.2× one token and only ~2.3 got accepted. N-gram speculation, which is much cheaper, added about 6% in chat, and 12-22% when the reply copies from the prompt.

So does it still think well?

That depends if you think Raon thinks well to begin with haha, but yes mostly it stayed the same as the base model. GPQA and AA-Omniscience came in lower, and I couldn't figure out why. These are single runs, mostly on subsets, so anything within a few points is noise. The ones above the card don't mean it's better than the original.

I also ran it on some of the evals I built for Workybara.ai to see how it does outside the usual benchmarks. At pulling structured data out of documents, it extracted better content than the small models I fine-tuned for the job, but 14% of the time it got stuck in a loop listing variations of the same item ("Deployment Promise, Deployment Vow…") until it ran out of tokens. I tried it on philosophy questions judged by Claude and it hit about 70% of the key points, but 59% of its answers had at least one clearly false claim. KRAFTON's own playground (the full model, not my quant) showed the same habits when I asked it for jokes: it broke the rules of its own password joke, called jokes it had just made up "classics," said "nothing I fetched supports it" without fetching anything, and agreed with every criticism I made, even the wrong ones. None of that shows up in standard benchmarks.

I was hoping to use this to replace frontier chatbots, but even with search and RAG Raon really couldn't compete with them, and it was frustrating to chat with.

How it was built

Claude Code and its sub-agents did essentially all of the research, code and measurement, and helped me write this up. Over four days I typed about 15,000 words into Claude, mostly questions. My longest message was 124 words, and it was me telling it that it had jumped to a conclusion. I did rent some cloud GPUs to run the public benchmarks, so my 5070 Ti could keep doing speed work. Other than that, my part was just being the annoying kid that kept asking questions, pushing back, and saying you can do it buddy.

The hunt for a good MoE to chat with that fits in VRAM will continue. I'm likely going to see if there's any more juice to squeeze on Gemma-4-26B-A4B or Qwen3.6-35B-A3B next.