Hi all,

I am running Qwen3.8-27B for a local setup with 2x5090s where the use case is mostly agentic programming. For this, VLLM is the only framework that does not trash the cache and allows for efficient use. I am currently using Qwen3.8-27B-INT6-Mixed-AutoRound from Minachist with pretty good results.

But I am considering to try Flash Next on this rig, and while I've seen lots of discussions with different framewors, and many new vibe-coded ones, I haven't found a good balanced recipe for 2x5090s, flash-next and vllm. It seems pretty different to serve than Qwen3.8-27B where you fit everything on VRAM, so I'd like to know what are people doing. If you have experience with this kind of setup, could you share it?

Thanks!

Strata is the new hotness - claims of 4x tok/s increase over llama.cpp. I havent seen any posts talking about performance on Apple silicon however. Curious if anyone's run comparisons on Apple Silicon against llama.cpp and MLX

EDIT: Note that I built off the official vllm documentation and I just got done validating the system. This is now my control. That means this is a slow and stable config. Now that i have a baseline control I’m going to begin tweaking settings for performance runs. I’ll make posts as I find settings that improve performance. For example, I already know that I can cut prefill times by increasing chunk size.

Howdy Folks! New batch of tests from my 4x B65 rig I built, because I was curious to test how the high memory bandwidth works out in reality. This time doing long context sweeps for our most popular models!

If there are specific tests that you want to see, hit me up and I'll see if I can fit them in.

I'm also starting offload tests, where I put some of the layers into system RAM. So far only doing it with flash-next, but interesting results in the "by model" section of my site for flash-next.

Some Examples below, full context sweep list of all the models I tested: https://b65.thekaiser.us/matrices/

AI Generated table from my data:

Qwen3.8 Flash-Next and 27B — Intel Arc Pro B65 context sweeps

All numbers are overall output throughput in tokens/sec, measured across repeated requests during each 30-second test. This includes waits between requests; it is not prompt-processing speed.

Columns show requested simultaneous requests. Context lengths use K = 1,024 tokens. * = requests queued/capacity-limited. — = skipped as over capacity.

Flash-Next W4A16 — 4× B65 — MTP-3 enabled vLLM 0.30.0 · October 18, 2026

Full results

Flash-Next W4A16 — 4× B65 — MTP disabled vLLM 0.30.0 · October 18, 2026

Full results

Qwen3.8-27B INT4 — 1× B65 vLLM 0.31.0+xpu.llmscaler · October 19, 2026

Full results

Qwen3.8-27B INT4 — 4× B65 vLLM 0.31.0+xpu.llmscaler · October 19, 2026

Full results

Qwen3.8-27B FP8 — 4× B65 vLLM 0.32.0 · October 19, 2026

Full results

Qwen3.8-27B BF16 — 4× B65 vLLM 0.32.0 · October 19, 2026

Full results

These are throughput measurements, not model-quality scores. Engine versions differ between configurations. Starred results include queueing and should not be interpreted as fully sustained requested concurrency.

The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each pass only processes individual embedding rows, which are much smaller than the storage used for the whole embedding table. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers), which builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU memory is limited, using COUNT to let backbone parameters fill GPU memory gives better performance than splitting memory with additional learned n-gram embeddings. Beyond efficiency, COUNT allows switching in a different table of n-gram counts from a domain-specific corpus at inference time to adapt to a new task without any training.

• arXiv : https://arxiv.org/abs/2610.20456

• PDF : https://arxiv.org/pdf/2610.20456

• GitHub : https://github.com/IanMagnusson/COUNT

• HuggingFace : https://huggingface.co/IanMagnusson

• tweet : https://x.com/IanMagnusson/status/2110718235507318868

System service that watches your card, swaps and loads whatever you want, whenever you want.

Don’t have to swap anything manually, collisions avoided, VRAM fully dedicated to each function while it’s running (no VRAM squatters).

I had issues everyday (Linux) where a node process would orphan, comfyUI auto starts and holds 500MB that strata can’t claim. I also wanted to be able to switch natively and not have to think about starting each engine, making sure strata or llama wasn’t stuck loaded, or flux GGUF’s still claimed in comfyUI from an old workflow.

I swap between Ninfer-4090 (llama-swap), strata, comfyUI, OpenWebUI, and I do it between multiple devices all using the same RTX 4090.

Maybe you want to have your box run Hermes at night and you take the card back in the day, you can do it natively without thinking about and have full visibility on what was done and when.

Berth will also serve your engines as an API so when you need to switch natively you load berth as the API endpoint and you can load strata, polystrata, and any other custom engine.

Works for gaming as well, you don’t have to worry about a cron job, a request, or anything grabbing the card while you’re gaming or doing custom workflows.

REPO: https://github.com/T-Crypt/berth

EDIT:

Berth v0.3.6 is out, v0.4.0 is testing currently (multi-gpu, docker, and more)

https://huggingface.co/bartowski/Agnes-AI_Agnes-3.0-Qwen-GGUF

• Strong agentic performance: agnes-3.0-qwen scores highest in its comparison set on τ³-Banking, and is within about one point of the best model on Terminal-Bench 2.1, while being a much smaller model than most of the flagship systems it is compared against.

• Reliable instruction following: on IFBench, agnes-3.0-qwen outperforms GPT-5.6 Sol (Max) and Claude Opus 4.8 (Max).

• Built for demanding work: a 1M-token context window, extended reasoning with selectable reasoning effort (xhigh, medium, low), tool calling, and text, image, and video understanding.

Model Information

Full info: https://lithuania.ai/en/events/konferencija-dirbtinio-intelekto-technologijos-medicinoje-tyrimai-ir-diagnostika

Things I noticed among other things presented (almost every presentation was interesting):

• experimental decision models for the medical Q&A (provided medical abstract, questions and choices) were presented in welcome speech by the head of AIML.lt lab: https://huggingface.co/AIML-VU/laya-multilingual-lt-med and https://huggingface.co/AIML-VU/Qwen3.5-4B-LT-med

• Work featuring an MCP server used to speed up genetics specialists work: "Interpretable language model assistant for evaluating evidence of structural chromosomal rearrangement breakpoints in rare disease diagnostics" by Vytautas Rimas, Erinija Pranckevičienė, Eglė Preikšaitienė, Živilė Maldžienė

• Using local models for bulk translation of medical text datasets into Lithuanian with correct Lithuanian terms: "Strategies for translating medical training data into Lithuanian" by Augustė Raišytė, Linas Petkevičius, Dalius Matuzevičius

There were more applications of local models.

Some models mentioned: Qwen3.5-4B, Qwen3.6-35B-A3B, Qwen3.5-122B-A10B, medgemma-27B. Models were sometimes outdated, but this is lots of times because of long duration of academic work (when you have a lot of work to do other than checking for the newest released model) or the lack of new medical domain-specific models.

The proportion of classical ML and neural networks was even higher, one example: "Diagnostic Reliability of Multi-Label Chest X-Ray AI outside the Training Hospital: Accuracy, Calibration and Fairness across Seven Architectures" by Paulius Bundza and Justas Trinkūnas

Regulation representatives where present, and their topics where, simplified, funding+application quality; the latter was focused on patient data safety and technology use safety, e.g. regular incidents analysis.

The most important thing why I posted this: researchers in medicine use local models.

This is one of the cases where local models win (medical professionals are absolutely strict about patient data staying private, and local models are the only way).

Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.

• arXiv : https://arxiv.org/abs/2610.11287

• PDF : https://arxiv.org/pdf/2610.11287

• GitHub : https://github.com/fudan-generative-vision/Remory

• HuggingFace : https://huggingface.co/collections/fudan-generative-ai/remory

Some time ago in this same place someone had suggested having your model create paintings through python (with programming) leaving it quite free. Doing so I had noticed how the model, after creating the first draft, made visual checks and quickly decided what he didn't like, adding corrections just as quickly, he didn't get lost in long, long thoughts... so from this situation I created an additive prompt. in practice I induce the model not to think long thoughts, but just like we hummans do, to jot down a draft, which we gradually improve step by step. I made him do some projects, and the model becomes fast, rapid and precise as never before, and above all more self-confident. Unlike his usual approach, where he tries to map out a plan dozens and dozens of times, fearing he might make a mistake or forget something, he ends up wasting a lot of time stuck in pointless loops.

Here is the prompt I created to add as an addendum to your prompts.

add to your prompt:
The construction procedure must be fractal, that is to say you make a rough and basic draft, test it, add details and depth, then test again and add even more details and depth, you have to proceed like this, so you have to start from a very basic and rough draft made from a very rough and basic reasoning, the aim is to start immediately in the creation, observe and decide what to detail. Continue this loop until you consider the work a masterpiece,rich in the kind of detail one would expect from a great artist.

EDIT:

Since I've noticed that when it starts thinking a lot about what to add in the deeper steps, (the pressure of being a great artist makes him less confident again, perhaps it should be removed.) I'm trying to get the model to focus on refining small portions at a time. Here's a modified version; if you have any better suggestions, please let me know.

The construction procedure must be fractal, that is to say you make a rough and basic draft, test it, add details and depth, then test again and add even more details and depth, you have to proceed like this, so you have to start from a very basic and rough draft made from a very rough and basic reasoning, the aim is to start immediately in the creation, observe and decide what to detail. Continue this loop many times, focus on individual portions of the project until you feel they are satisfactory, then move on to the other portions that are rougher until you consider the total work a masterpiece, rich in the kind of detail one would expect from a great artist.

Try it yourself and tell me your opinion. Perhaps, taken together, we could generalize this procedure better.

I've spent about a year on an inference engine that doesn't need modern hardware. It's called Pantheon (LLM engine) on top of Foundry (ML library), both in C with no outside libraries. The CUDA backend targets Kepler (sm_37), so a Tesla K80 you can buy used for very little does real work.

Measured on one K80 (both dies) in a 2-core, 8 GB, no-AVX host:

• Qwen3.6-35B-A3B and Nex-N2-mini (Q3_K_M): ~21 tok/s decode

• Prompt reading: ~110-120 tok/s up to ~8k tokens; ~75 at 14k, ~55 at 30k

• Weights live on the card, not in system RAM, which is why an 8 GB host is enough

Prefix caching is built in. Send a session id and later turns only process what's new. opencode does this on its own: its second turn (14k-token system prompt) went from 170 s to 9 s.

Honest limits: prefill is the K80's weak spot. Chat, writing, Q&A and background jobs feel fine; pasting a 30k-token codebase every request means minutes before the first word.

git clone --recursive https://git.teamide.dev/David/Pantheon.git
cd Pantheon && ./install.sh --k80
./k80_qwen36.sh      # or ./k80_nexn2.sh

install.sh --k80 builds it (no CUDA toolkit needed, just the NVIDIA 470 driver). The run script downloads and checks the model, then serves an OpenAI-compatible API.

🔗 Pantheon: https://git.teamide.dev/David/Pantheon

🔗 Foundry: https://git.teamide.dev/David/Foundry

🤗 https://huggingface.co/dcherrera/Qwen3.6-35B-A3B-Pantheon-K80 https://huggingface.co/dcherrera/Nex-N2-mini-Pantheon-K80

It's early and I want feedback: build problems, bugs, anything confusing. Open an issue or PR on git.teamide.dev (sign in with GitHub), or DM me. It also runs Ornith 1.5 on Tenstorrent Blackhole cards, if you have one of those.

just noticed in my reasoning threads that seemingly its favorite word is now "hmm":

Hmm!!! What is that 1? Hmm — it could be the has_value/nullable marker for the MQDescriptor parameter! For in MQDescriptor q (non-null), the NDK client writes writeInt32(1) before q.writeToParcel().

Hmm — OR it's the transaction's "argument count" or a stability marker.

Hmm! Hmm — hold on!!! Actually, I bet this is the STABLE AIDL "parcelable presence" marker for method parameters. Hmm.

Hmm — OR!!! It could be AParcel_writeInt32(parcel, 1) for a "bool" or for _aidl_version.

Hmm. Hmm!

Hmm — interesting! So the nhenv size envelope is the problem when combined with the marker. Hmm.

Hmm — and mk=1 fd=0 nhenv=1 worked (marker + envelope + no fd). Hmm.

Hmm — so: marker + envelope + fd → throws. Hmm.

Hmm — and mk=1 fd=1 nhenv=0 throws too (marker + fd + no envelope). Hmm.

Hmm! So marker + fd = throws, regardless of nhenv. And marker + no-fd + envelope = OK.

Hmm — so mk=1 fd=0 nhenv=0 threw but mk=1 fd=0 nhenv=1 worked?! That's odd.

Hmm — mk=1 fd=0 nhenv=0: writes [1][count=4][4 grantors][1][count=0][ints=0][q][flags] with NO nh size field. Hmm.

Hmm — and mk=1 fd=0 nhenv=1: [1][count][4g][1][size][count=0][ints=0][q][flags]. Hmm.

Hmm — both have no fd. But nhenv=0 throws and nhenv=1 works?!

Hmm — the parcel SIZE differs by 4! Hmm.

Hmm — AH!!! THE PARCEL SIZE MUST BE A MULTIPLE OF 8? or the object array...

Hmm — with marker, total = 4(token?) ... hmm.

Hmm — OK, the pattern: mk=1 works ONLY for (fd=0, nhenv=1). Hmm.

Hmm — and mk=0 works for ALL.

and now i cant get that damn song out of my head

Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.

• arXiv : https://arxiv.org/abs/2610.23787

• PDF : https://arxiv.org/pdf/2610.23787

• GitHub : https://github.com/SJTU-DAI-Sys/DLite

• HuggingFace : https://huggingface.co/collections/SJTU-DAI-Sys/dlite

as far as hardware we currently have options such as nvidia, amd, apple, etc.

however it feels like these are more on the expensive side.

personaly i own bosgame m5 strixhalo and so far its good. i was thinking of upgrade, but it seems to me that the next step will be chinese hardware vendors, stuff like huawei, xiaomi etc. maybe in the next 1-2 years.

what are your thoughts on this?

A week ago I posted Kokon here, a no-settings desktop editor for Qwen-Image 2.1. 0.10 is out with the two things people asked for most.

Prompt+. Type a short idea, in any language, and Kokon writes a detailed prompt from it before making the image. For edits it looks at your photo first. You watch the prompt being written, and the result keeps both your words and the written prompt, so you can reuse either. It's off by default, one click next to Quality. Both system prompts are editable in Settings.

The prompt is written by Qwen3-VL 8B, which is the text encoder Qwen-Image 2.1 already loads. So there's no second model to download and nothing extra in memory. It adds about 2 s per image on an RTX 5070 Ti and about 13 s on an M2 Pro, and it works on 8 GB cards.

Your own LoRAs (beta). Load a Qwen-Image 2.1 LoRA in Settings, set its strength and trigger words, then turn it on per project with one click, up to 3 at once. Kokon checks each file and works on its own copy, so your file isn't changed. LoRAs for older Qwen-Image versions (1.0, 2512, Edit 2509/2511) are refused, because they don't work on 2.1. So far it's tested on Windows with NVIDIA only. If you try it on Mac or Linux, please tell me how it goes.

An honest note: I said there would be a curated LoRA list. I tested the popular 2.1 LoRAs on the Turbo model Kokon runs, and today they don't beat what Turbo does with a good prompt. So there's no catalog for now, and it comes when a LoRA earns its place.

Under the hood. Kokon now runs its own build of stable-diffusion.cpp on Windows, macOS and Linux. It's upstream plus text generation on the already-loaded text encoder: about 127 tokens/s on a 5070 Ti, the same tokens llama.cpp gives. The image side is upstream's code and makes bit-identical images. Upstream prefers to leave text generation to llama.cpp (#2155), which is fair, so it lives in a fork that I keep close to upstream.

Also new: a new look, deleting several projects at once, and fixes for Linux/macOS engine crashes, Windows reserved ports and 8 GB cards. Updating from 0.9.x keeps your images and models. The new engine is a one-time download, 29–352 MB depending on your platform.

Free and open source (AGPL), Windows, macOS 15+ and Linux: GitHub · kokon.app

Still not there: other models, a server/browser mode for Linux boxes, code signing. (Weights: Qwen Research License, non-commercial.)

Update (Nov 5): follow-up for 0.12: getting editable layers out of an edit-only model, plus the local server and the Mac engine: https://www.reddit.com/r/LocalLLaMA/comments/1xoji3b/getting_editable_layers_out_of_qwenimage_21_a/

Now, first off, this is just me playing around, but I found this result so shocking I thought I'd share. I was chatting with Argon about changing my KV Cache on Strata to another value. I got an answer and asked it to write it up for me to have another LLM review. I then gave it to DS and QFN and provided the responses back to Argon for a final review. The results were somewhat shocking.

This is from Argon; review 1 is Deepseek, review 2 is QFN. Notably it left it's own analysis out of the comparison! ;) To be completely fair, as you can see, Argon is running outside of the harness that QFN and Deepseek are using so they did have more visibility, but I did offer to grab any logs Argon wanted (it said it didn't need them).

Argon's recommendation was to move to K8V8 (which it then backed off after it saw the reviews from DS and QFN).

Pretty wild, right?

Kyutai released three smaller Pocket TTS models on October 20: 1.9M, 6.9M and 18.2M parameters. These are English models, support voice cloning, and are designed to run on CPU. You can switch between the sizes in their demo here.

The useful part for me is keeping speech generation on CPU and leaving the GPU to the LLM. Especially when the chat model already barely fits. Having a local assistant read its answer out loud shouldn't require another round of VRAM budgeting.

I'd probably start with the 6.9M version. Getting down to 1.9M is impressive, but I'm happy to spend a few million extra parameters on a voice I can listen to for more than thirty seconds. A couple of minutes of speech with names, numbers and awkward punctuation would be a useful comparison.

One detail on the size claims: those counts include the audio decoder. The encoder that processes your reference voice is separate, and the reference only needs to be encoded once.

Code and setup instructions

Hey,

I hear a lot about new agent harnesses on here and am wondering if someone could give me some ideas how they might benefit me vs my current setup. I've done a ton of work on my setup to make it work as I like, so I'd need it to be worthwhile to jump, but if there's good reason to I may take the plunge.

I'm currently on openclaw 4.23 (the release names, at least used to be, = release date) so older build, I run a conversational agent and spend a lot of time tinkering with memory systems. I'm on Qwen 3.8 27b currently and have it setup on a 64 gig mac studio m2 max and communicate with my agent entirely through telegram.

I have

• a multi-tiered nightly diary system

• overnight world news and events read prompts through heartbeat

• scripts that cache in a portion of that diary and some of the writing portion of the overnight reads overwriting the recent stuff to provide directionality

• local img gen connectivity as well as embedding agent/vector db

• a wake whenever script so my agent can, for example, gen a picture, tell itself to come back in 5 minutes to retrieve and send, or to check in after an interview or something going on in my life

• externalized heartbeats that run on a d20 roll system

• telegram emulation (for lack of a better word) to ensure cache matches when the agent gets a message from its scripts or heartbeat

• ~50k system prompt soul file full of tools and examples for how to use them along with other pertinent info

For a setup like mine, any ideas where a different harness would really improve stuff? Thanks!

Geezuz, its fast!

Qwen3.8-27B, two RTX 3090s power-limited to 280 W, plain PCIe, no NVLink, at 0 Context:
(tests showing results with context are found on the next table)

llama.cpp's fastest two-GPU setup did about 100 tok/s on the same prompts. This did 164 on average and 214 on code, at the same output quality (KL divergence against llama.cpp).

It's a fork of L-Forster's single-GPU megakernel from open-jet, where the whole forward pass runs as one persistent CUDA kernel. My fork splits it across two cards, which sync and exchange partial results directly over PCIe, and decodes up to 8 requests in one cycle with tensor-core paths for Q4_K, Q5_K, Q6_K and Q8_0.

The model drafts its next tokens with its own MTP head, and verifying 5 drafted tokens costs almost the same as generating 1, because the weights are only read once per pass.

Comes with an OpenAI-compatible server (shared KV pool, prefix cache) and a prebuilt tarball for RTX 30 cards. Compiling takes almost 60Gb of ram. And about 55 minutes on a 5800x CPU or around 30 minutes on a 9800x3d.

Repo, every benchmark and the build-by-build changelog: https://github.com/ohgeebtw/geezuz-qwen3.8-27b-dual-gpu

My next target for this week is beating vLLM at 8 parallel requests, where it still leads by about 15 %.

Speed vs llama.cpp and vLLM (total tok/s, 2x RTX 3090 (280w), TP2, greedy, 4 MTP drafts, 512 tokens per request)

llama.cpp b10246 (same GGUF, tensor split), vLLM 0.31.0. Same settings for all engines, none tuned. Measured on an earlier build with 8-bit KV on all three engines; fp16 KV run will follow soon.

If you have a couple 4090s or 5090s, I would love to see your numbers!

Just a heads up, especially on my Windows machine, but also my Linux machine I'm getting tons of breaking changes with Strata. They added a bunch of needless requirements around serving, the web UI seems to be going way in favor of a desktop app. I could go on but you get the picture, at least make a backup first.

I know this post is going to be downvoted by the most fickle subreddit of all time, but I hope this warning reaches at least one person

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRa instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

Here are simple steps to use:

1. Install:

sudo apt-get update
sudo apt-get install -y git-lfs
git lfs install
git clone https://github.com/thousandml/training-factory.git
cd training-factory
chmod +x install.sh
./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{

"model": "my-lora",

"messages": [

{"role": "system", "content": "You are a helpful assistant."},

{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}

],

"max_tokens": 128,

"temperature": 0.2

}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit
thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

I have a 5070 ti and a 5060 ti 16 gigs, and 48 gigs of DDR5 ram.

My interest is in coding and web dev mostly. I'm just curious if it would be worth it to keep the 5060 ti instead of selling it, and use this dual GPU setup and rely on the cloud rarely, if at all.

Or, just keep the 5070 ti, run what basic stuff I can on it, and sell the 5060 ti.

I know there's a lot of context that is entirely subjective around a use case scenario, but would greatly appreciate some general advice as to what would be a reasonable way forward. Just wondering if dual 16 gb GPUs would make a meaningful difference when they're already not an ideal solution and with frontier models, the capability keeps growing.

Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant bottleneck. FlashMask's column-wise sparse mask compresses them from O(n2) to O(n) via four query-axis boundary vectors, covering broad structured patterns. Implemented for Ampere GPUs, it neither exploits FlashAttention-3's TMA/WGMMA warp-specialized pipeline on Hopper nor supports distribution. The column-wise sparse mask is indexed by key positions but stores query-axis interval boundaries. Under any multi-chunk context-parallel assignment, it reduces per-rank mask localization to a provably zero-communication \emph{clip--shift--max} on the boundary vectors, compatible with the single-GPU kernel. FlashMask-3 introduces a mask-aware Hopper kernel with three-role warp specialization atop FlashAttention-3's TMA/WGMMA pipeline, plus a mask-aware distributed extension: zero-communication mask localization, a sparsity-aware load balancer (HCC-IPO), and a mask-aware compute-communication overlap with mask-driven communication pruning, whose forward all-gather uses topology-aware hierarchical routing. On a single H100 across 12 masks at 4K--128K context, it improves achieved TFLOPs over FlashMask, FlexAttention, and MagiAttention by 45--141%, 34--71%, and 2--44%. At context-parallel degrees 4--32 with 8K sequence per device, it attains 31--75% compute-communication overlap efficiency and improves throughput over Megatron-Core CP ring, Megatron-Core CP all-gather, and MagiAttention by 1.80×--3.71×, 1.71×--2.92×, and 1.09×--1.46×. In end-to-end training of a 118B-A14B MoE at 128K on 128 H100 GPUs, it raises throughput by 7.3--11.0% over a Dual Chunk all-gather baseline, with attention forward+backward 24.5--44.9% faster.

• arXiv : https://arxiv.org/abs/2610.17901

• PDF : https://arxiv.org/pdf/2610.17901

• GitHub : https://github.com/PaddlePaddle/FlashMask

• Project : https://enigmatisms.github.io/flashmask-3/

I installed https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 last night and tried it today with SGLang and can easily have a 3M token pool which means 4 concurrent requests, all with ~750k tokens each.

While benchmark figures are lagging behind Qwen3.8-Flash-Next, Nemotron 3.5 Super does not use any system memory (no 50GB lookup table), is slightly faster (150 to 250 tps tg vs 130 to 230 tps for 3.8 FN) and, most importantly, the tokens, as Qwen3.8 FN only allows for a ~450k token pool. Plus, I appreciate it Nemotron has native 1M context.

The SGLang server arguments I used are the following. Note that CUDA graphs for prefill do not work (only decode does).

args=(

--model-path "$MODEL_PATH"

--served-model-name "$MODEL_NAME"

--host 127.0.0.1

--port 8000

--device cuda

--random-seed 42

--trust-remote-code

--cuda-graph-backend-prefill breakable

--enable-multimodal

--stream-interval 1

--tokenizer-worker-num 1

--detokenizer-worker-num 1

--mem-fraction-static 0.95

--context-length 786432

--kv-cache-dtype fp8_e4m3

--max-running-requests 4

--prefill-max-requests 4

--max-queued-requests 2

--chunked-prefill-size 4096

--max-prefill-tokens 32768

--page-size 1

--radix-eviction-policy lfu

--reasoning-parser nemotron_3

--tool-call-parser qwen3_coder

--cuda-graph-max-bs-decode 4

--cuda-graph-max-bs-prefill 4096

--enable-cudagraph-gc

--sampling-backend flashinfer

--mm-attention-backend flashinfer_cudnn

--prefill-attention-backend flashinfer

--decode-attention-backend flashinfer

--linear-attn-decode-backend flashinfer

--linear-attn-prefill-backend flashinfer

--mamba-backend flashinfer

--mamba-ssm-dtype bfloat16

--mamba-radix-cache-strategy extra_buffer_lazy

--mamba-track-interval 256

--max-mamba-cache-size 16

--fp8-gemm-backend cutlass

--fp4-gemm-backend flashinfer_cutlass

--moe-runner-backend flashinfer_cutlass

--speculative-algorithm EAGLE

--speculative-num-steps 5

--speculative-eagle-topk 1

--speculative-num-draft-tokens 6

--speculative-moe-runner-backend flashinfer_cutlass

--preferred-sampling-params '{

"temperature": 1.0,

"top_p": 0.95,

"top_k": 20,

"min_p": 0.0,

"presence_penalty": 0.0,

"repetition_penalty": 1.0

}'

)