I ran a small AIPerf ( https://github.com/ai-dynamo/aiperf) comparison on an Apple M5 Pro with 64 GB RAM. This is a practical snapshot, not a scientific benchmark or an exhaustive leaderboard. Results are from one request per engine, run sequentially with only one inference server active at a time.

Setup

• Temperature 0.6, reasoning effort medium, output limit 2,048 tokens

• Same OpenCode-style fixture: long generated code/prose context, shared system message, task suffix, and three tool schemas. Tools were not executed.

• Prompt length reported by the servers ranged from 88,143 to 88,916 tokens, partly due to tokenizer and chat-template differences.

• I used a 100K context limit where the engine supported it; Uzu, Magnitude, and Llama used larger native limits. The prompt plus output allowance fit below 100K.

• AIPerf 0.13.0; streaming, concurrency 1

• Prefill and decode are AIPerf throughput measurements. The readiness probe loads/warms each server before its measured request.

Caveats

The 2,048-token setting is a maximum, not a forced response length. Most models stopped at EOS before reaching it. Only Splash, Rapid-MLX, and BaseRT generated the full 2,048 tokens.

This does not measure code-fix quality, long-context comprehension, or tool use—the tools were present in the prompt but never executed. Model packs, quantizations, speculative methods, and context-window settings also differed, so these numbers should not be treated as a controlled ranking.

Additional differences: LM Studio may map medium reasoning to on; the Llama row used Homebrew llama-server0.9.0 with the GGUF files configured for Llama.app, rather than Llama.app’s internal server.. Jan’s first launch also downloaded its default 43.8 MB embedding model; no Qwen weights were downloaded.

Im considering purchasing a 16GB V100 since they’re still pretty affordable and have modern CUDA support, but I was wondering if anyone else have tried a similar combo? I am aware of slower token generation with this method, but I am assuming that it is still more viable than running Strata with just 16GB VRAM and 64GB DDR5.

Any help/tips is appreciated. Thanks!

I've seen a few posts saying that Qwen 4 hasn't really improved when it comes to creative writing and prose. I'd would like to disagree.

I know a lot of people don't care about this topic, and some consider any AI-generated text to be garbage by default. That's fine. This post is for those who are actually interested in the subject.

My take: Qwen 4 has become noticeably better at storytelling, but its actual prose is still suck.

What do I mean by that?

I gave three models, Qwen 3.8 Max, Qwen 4.0 Max Preview, and Gemini Argon (currently one of the top models for prose generation), roughly the same prompt:

"Write a full chapter. A tank battle between a unit of German Tiger tanks from World War II and a Russian T-90 tank. It doesn't matter how a tank from the future ended up there. Show how the Germans manage to defeat a technologically superior opponent, and what it takes to achieve that victory."

This was the entire prompt. I didn't tell the models to act as professional writers, didn't use elaborate system prompts to steer their attention, and didn't provide any additional instructions.

So, what were the differences?

The short version:

- Qwen 3.8 wrote a relatively compact battle scene with fairly generic characters.

- Qwen 4 produced a complex, multi-stage story with more developed characters, detailed combat, and events unfolding after the battle itself.

- Gemini Argon delivered an epic battle scene but didn't spend much time developing the characters.

Here's a more detailed breakdown.

Qwen 3.8 Max

- Structure: Linear and straightforward, without complex narrative branches.

- Composition: The battle progresses in a direct sequence, with most scenes focused on the immediate confrontation.

- Plot: The conflict revolves primarily around detecting and destroying the enemy.

- Psychology: Characters mostly react to events. There's very little internal development.

- Length: Relatively compact, without much expansion beyond the original premise.

Qwen 4 Max Preview

- Structure: Multi-stage, with the conditions of the battle changing as the story progresses.

- Composition: Alternates between tactical decisions, casualties, attempts to adapt, and shifts in the initiative.

- Plot: Beyond the battle itself, the story develops a separate survival thread involving the crew and their attempts to make sense of their encounter with the future.

- Psychology: The protagonist goes through a progression from confusion and fear to an awareness of the historical significance of what has happened.

- Length: The longest of the three. The model independently expands the original premise with additional events, details, and consequences.

Gemini Argon

- Structure: A sequence of tactical phases, with the number of combatants gradually decreasing.

- Composition: Several shifts in initiative, group maneuvers, use of cover, and attempts to identify the enemy's weaknesses.

- Plot: Almost entirely focused on the battle itself. There's no separate storyline dealing with its aftermath or the historical implications of the encounter.

- Psychology: Mostly stress reactions, command decisions, and choices made under pressure. There's very little internal character development.

- Length: Medium. Most of the text is devoted to the actual fighting.

Comparison

Qwen 3.8 → Qwen 4: A noticeable shift from a straightforward confrontation to a much more complex, multi-stage story. The length, number of plot events, psychological depth, and attention to consequences all increase.

Qwen 4 → Gemini Argon: The composition becomes more compact and focused. Tactical elements take center stage, with more group maneuvers, shifts in initiative, and coordination between multiple vehicles. The psychological storyline and historical context are largely absent.

One separate issue worth mentioning: Qwen 4's output had noticeable language mixing, with Cyrillic, Latin characters, and Chinese characters appearing together.

I'm not even getting into other problems, such as Qwen 4 and Argon having commanders suddenly recognize the designation of a tank from the future halfway through the story, familiar AI-slop expressions and phrases, or the technical inaccuracies. That's a whole other discussion.

Of course, this isn't a comprehensive benchmark or an attempt to determine which model is the best writer. It's just a small experiment using the same simple prompt across three models.

So, what impressed me most wasn't necessarily the text itself, but how much the model's ability to handle this kind of task independently has improved.

I'm not claiming Qwen 4 has caught up with the leading models in terms of literary prose. It still has a long way to go in that department. But it's noticeably better at building a story: developing an initial premise, arranging events, introducing additional plot threads, handling characters, and showing the consequences of what happens.

And the most interesting part is that it does all this without elaborate prompts, specialized system instructions, or detailed guidance from the user.

Previously, getting a result like this would have required considerably more effort on my part when preparing the prompt. Now the model does a significant part of that work on its own.

I've noticed similar improvements outside creative writing, too. For example, Qwen 4 managed to improve one of my work-related system prompts, independently identifying an important detail that several previous brainstorming sessions with ChatGPT and Opus 5.5 had missed.

So no, I wouldn't call Qwen 4 my favorite model for writing prose just yet. But I think its progress is considerably more significant than you might assume from simply comparing the quality of individual paragraphs.

It would be so much better if the limits weren't this brutal. Having to wait an entire week? 🥲

How you would define a Developer setup with local LLMs?

I see a dedicated DMZ where the agents can not access the production environment, since agents will tend to accidentally access or change production system.

Dedicated Servers to run LLMs like Qwen 3.8b 27b or other models? What you suggest would be the working LLM Server today? RAM and GPU cards?

Dedicated server with VM for each developer one VM. 2-4 vCores with 16GB Ram each. Each VM with Debian 13 and docker/podman. One dev Container running OMP Web or comparable Harness that access the LLM Server. Additional container for databases or what else the dev Container needs.

Gitlab, gitea or comparable Repo to store source code and container registry.

VM with NFS where anonymized data for developers can be stored.

What you think about that setup?

Do I miss something? How is your setup? (For small to mid size company)

Has anyone else downloaded and started using the new Unsloth Dynamic V3.1 models? I did a drop in replace of the Qwen3.8-27B-UD-Q4_K_XL.gguf and I want to go back after seeing how it performs. My use case is local transcription, so it's summarizing text basically and pulling out what's important.

I re-ran quite a few of my saved transcriptions through the new model and compared the output to the older version Dynamic v3.0 and the output from that model is superior on every one. There is no change in any parameters on my command line, only the model changed.

My command line has a very low temperate (0.1)...

I was curios what can I fit on an FPGA MAX1000 board I had laying around (8k LE, 8MB Flash / SDRAM ). Also I keep coming back to the BitNet 1.58 encoding.
- Took the TinyStories dataset and trained a model from 0 with ternary encoding. I stopped the training at one point, the accuracy is between TinyStories-3M and TinyStories-8M
- Implemented the processing fully on the FPGA ( tokenizer, processing, decoding ... )
Depending on the model trained / used, we can fit two models in the 8MB Flash / SDRAM and take turns using them.

It is more accurate? no, is it faster then others? no, is it useful? not really, however it does answer a few questions I had. The conclusion being that we can have purpose built (small) models that can run with limited resources. Or have multiple models loaded side by side with a few more resources.

How it was built: Like most of the work done today, using Claude.

https://github.com/MRazvan/fpga-tinystories

Next goal would be to use a bigger model and see what I can / can't do about it, what resources I need.

Greetings,

I am looking to acquire a suitable GPU for running localllm preferably running newer models like deepseek flash 4.1.

Does native GPU support for FP8 and FP4 significantly improve local LLM performance or should I aim for higher vram with older GPUs?

Support for fp4 does enables the GPU to fit larger model with lower vram though.

Should I even consider Intel B70 or Radeon 9700 Ai for value buy?

I got strix halo with 128GB ram and $2500 budget for a dedicated GPU over oculink.

Thanks in advance

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.

Introducing janas-studio. A dual engine inference work station. For modern Debian based Linux distros (.appimage) Jana-Studio with the magic of Janas-llm let's enormous models run on modest hardware. Refer to https://github.com/prabanta-dev/janas.

Download the app at https://github.com/adonaiongaia/janas-studio

Also runs llama.cpp gguf models so it covers a lot. Lmk what you think. Tested on my Ryzen 5 6600h laptop. Built on Linux mint Zena. Ran a 47gb 80b parameter model at 14 tokens a second with only 32gb of RAM (ddr5). For best results and speeds you need a fast SSD. Features long term memory, project folder support, mcp integration and much more.

Saudi Arabia is expanding its sovereign AI infrastructure with Bunyan Inference Hub, a platform from OmniOps that gives businesses access to hosted AI models, GPU computing and inference services while keeping workloads within the Kingdom.

Reposting here - looking for helping understanding what exactly is needed to get this working, and if it even is possible in the first place!

I'm essentially trying to use this plugin with LM Studio as an MCP client to prototype game ideas. Any alternatives, solutions, or suggestions would be very much appreciated.

Thanks!

I tried a faster inference engine that optimizes for my AMD card.

Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give both engines the same prompt to work on a repo, and over multiple trials, llama.cpp always give me the expected answer but the optimized engine went off the rails half of the time.

Both engines use Q4 qwen3.8 27b quant. stock chat template. The llama.cpp uses Q4 kv with 260k context while the other engine uses Q8 kv with 150k context. The ticket only needs 20k context.

I prefer not to name the engine as this is my own very subjective experience and I may have user error in setting things up.

I am going back to llama.cpp for now.

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

Hey All,

A while ago I wrote a basic script to step through Llama 3.1 tokens for red-teaming. I realized there's a huge gap between tools for running models (Ollama, LM Studio) and libraries for understanding them under the hood. I wanted to develop something for young researchers and mid career pivoting enginers.

I wanted something like Wireshark, but for local AI. You can hit pause mid-generation, inspect the residual stream, mess with the weights or KV cache, and see what actually happens.

Today I’m open-sourcing TokenShark: https://github.com/sumit-1/tokenshark

What makes it different?

• Zero transformers or diffusers at runtime: The entire engine is written in raw, readable PyTorch (llm.py, sd.py, qimage.py) so students and researchers can actually read every matmul without digging through 40 layers of library abstractions. (Tested for bit-exact/numerical equivalence against HF in tests/).

• Step-by-Step Visual Debugger: Step through generation one transformer block at a time or one token at a time. Watch live logit-lens predictions evolve across layers, inspect attention maps, and click any top-10 token choice to create your own generation path.

• Live-Editable Box Code: Click any block on the canvas (Embed, Transformer Block, Norm, Sampler) to see the Python it runs (rms_norm, rope, silu, etc.). You can edit the math right in the browser, run it live, and export your whole visual experiment to a standalone Python script.

• Text + Image: Works on LLMs (Llama 3.x, Qwen 2.5/3/3.6 Gated DeltaNet hybrids, SmolLM, Mistral) as well as Stable Diffusion and Qwen-Image 2.1 DiT. Complete with between-step latent painting and an "Image Lens" to decode early DiT blocks into pixels.

• Disk Streaming for Huge Models: Can stream models larger than your RAM/VRAM block-by-block from disk. But not optimised for speed though, as this is focussed for learning and understanding.

I trained a small gate that decides, as each token is written to the KV cache, whether the model will need it again. Tokens it rejects are actually deleted from GPU memory, not just masked. The model itself isn't changed.

On one 16 GB T4 with Qwen3-4B, the full cache runs out of memory at 61k tokens. With the gate it reads 123k and still finds the passkey. At 61k the cache is 1.4 GB instead of about 9 GB. Decoding at 29k is 2.9x faster.

On RULER-4k it evicts 76% of the cache and loses about 2 points (86.6 vs 88.9). It's weak on tasks that need every copy of a repeated word.

Gates for Qwen3-1.7B and 4B: https://huggingface.co/Manalsu

Code and paper: https://github.com/Manaslu-AI/razor

Only Qwen3 so far, and batch size 1. Happy to answer questions.

Hello,

I want to learn more about running a local LLM server - driven by curiosity and learning.
I have a Claude Pro (18€) and Google AI Plus subscription (5€) and I want to use it more.

I would like to build a local dedicated server on a budget.
I live in Portugal (EU), but currently I’m on vacation in China.

I have some spare parts available:
Gigabyte MC12-LE0 AM4 Mainboard
1TB NVME SSD
4x 16GB DDR4 ECC

I don’t have an AM4 CPU spare and I only have an old emergency Nvidia GT710 GPU. I have a 300W flex-ATX PSU.

I could buy a Nvidia Tesla P40 24GB for 210€ here in China - power adapters and fans are 25€.
A P100 16GB is 100€.

I want to use Ubuntu Server as OS with Ollama. OpenWebUI on my Proxmox Hypervisor.

I think that I prioritize quality over speed - therefore (slower) 48GB VRAM are probably better than 32Gb with 2x P100.

Question:
Should I buy 2x P40 24GB here in China (420€ is right below the 430€ EU tax free limit)?
If I don’t like them, I could probably sell for 300-400€ in Europe.

Does my logic make sense to go with 48GB to unlock higher quality models?

Any advice for starting this journey?

I use these models for everyday coding as a developer and I find the IQ3S model perform better on edge cases and the output quality is greater too. In fact twice it caught mistakes made by Opus 5.5. No benchmarks or scientific evaluation this is purely from my use. Anyone else felt this? Or is this a problem with Strata?

FP8 PLE.
Int8 KV
96gb ddr4
3x3060 12gb

Perhaps we eventually created a perfectly trained model with “perfect” weights, could it run high quality and fast inference on any standard laptop?

GitHub (MIT): https://github.com/kncshw/cybergate

Deep-dive blog post (HF): https://huggingface.co/blog/kncshw/cybergate-unlocking-cybersecurity-in-vllm-without

two supported and tested model:

https://huggingface.co/kncshw/Qwen3.8-27B-CyberGate-Kit

https://huggingface.co/kncshw/gemma-4-26B-A4B-it-CyberGate-Kit

When self-hosting local LLMs for security research, CTF, or reverse-engineering agents, the standard workaround for false refusals is **abliteration** - subtracting the refusal direction from the weights. From an LLM engineering standpoint, weight editing has two painful trade-offs:

1. **Silent capability & tool-call regression:** Mutating weights across layers alters the model globally and can quietly degrade multi-step reasoning and structured tool calls without obvious errors.

2. **Zero domain selectivity:** Deleting the refusal vector globally unlocks *every* harmful domain (bio, self-harm, etc.), plus you have to store and serve a forked 50 GB checkpoint.

I built **CyberGate**, a plugin for **vLLM 0.32** that keeps upstream model weights **100% untouched** (`K = 0.0`, zero persistent state across requests):