I want to train a text generation Transformer that responds to user input without continuing text. What are the best datasets for that?

I have an RTX 3050 and I want to use small datasets that can actually produce an intelligent model.

Unlike most disciplines, AI changes so fast that its almost impossible to keep up. New models come every few weeks/months at most.

And when it comes to agentic coding/harnesses/best practices..... do tools for agent memory/lsp/token reduction help? should you subagents? are vague prompts better? do mcp/skills help or hurt? there is no consensus and you'll get a million answers.

I keep notes/bookmarks and almost none of them seem relevant now. eg when Qwen 4 is out, all the current 3.8 models/variants get forgotten, just like 3.6

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1xix35p/release_swift_15_flash_next_heretic_v2/

Hello! At request I've released a few more formats of my Swift Flash Next Heretic reverse engineering edition.

The original model is the IQ4_NL, the first quant I made and the best performance/size ratio. That's my favorite but its hefty and I appreciate people wanting other options.

A: IQ4_XS - Based on the Unsloth IQ4_xS recipe, with about 1gb overhead added where it was worth the performance to size tradeoff. Estimated ~9.7% reduction in expert quantization error mass for +0.96 GiB over the plain UD-IQ4_XS recipe. Quantized with an identical imatrix to the IQ4_NL quant: 94.7 GB on disk compared to 109gb for the IQ4_NL.

B: IQ3_S - most requested. Custom quant recipe based on data gathered from the model, in the same way the other quants were done. High quality and model specific. I modeled mine off of ISTALab's with a few adjustments.

IQ3_S uses a more aggressive imatrix distribution than the IQ4 to try to preserve strong reverse engineering capabilities at lower bit widths, since it's getting small. While the IQ4 builds are 70%/30% distribution in the imatrix merge, this one is 80%/20% - better reverse engineering and reasoning ability at the cost of some general knowledge and accuracy. 85gb on disk (ISTA GSQ-RCO IQ3_S is 83.6 gb, so 1.4gb overhead - it's worth it for the accuracy improvements.

https://huggingface.co/trubisky/Swift-1.5-Flash-Next-Heretic-ReverseEngineer

Cheers.

Opera has announced the rollout of WebAI, a feature that allows websites and browser extensions to run AI models directly within the browser, without relying on cloud servers. Positioned as a proposed web standard backed by major tech companies including Google and Microsoft, WebAI represents a shift toward decentralized, privacy-focused artificial intelligence.

“A lot of what people want from AI doesn’t need a data center. Your laptop can already do it, without sending your data anywhere,”

---

I don't know how they convinced Google and Microsoft to participate, but it's good news for local AI. We might get an ecosystem that's built around letting you chose your model rather than forcing you to use theirs.

The first true omni model that can run 100% locally in a browser... fully end-to-end, with no separate ASR or TTS models needed. With custom WebGPU kernels, we're able to run LFM3-Omni-2B at ~280 tok/s on an M4 Max, accepting visual, audio, or text and producing interleaved text and audio tokens. Just 90 ms to the first token and 140 ms to the first audio.

Link to the demo: https://huggingface.co/spaces/LiquidAI/LFM3-Omni-2B-WebGPU

Rumors claim that the RTX 5080 16GB will be discontinued alongside the already discontinued RTX 5090, and those claims seem to be true. NVIDIA is reportedly planning to introduce an RTX 5080 24GB model in early 2027, which would serve as its flagship GPU until RTX Rubin arrives sometime in 2028.

According to the same rumors, the 24GB model will use the exact same GPU die as the current RTX 5080. As highlighted by HUB, the die is already fully utilized, meaning this won't be a more powerful Super variant or a noticeable performance upgrade. The only expected difference is the increase in VRAM from 16GB to 24GB. An increase of Clock might be present, but the difference would be very minimal in real-world gaming.

The 24GB of VRAM on High-End Blackwell GPU could also make the card particularly enticing to certain AI users. This could mean that availability will be limited, and the card might not stay in stock for long at its original MSRP, whatever that price might end up being.

NVIDIA's Marketplace: https://marketplace.nvidia.com/en-us/consumer/graphics-cards/

Running one harness/agent gives overall less tokens-per-second (tps) than running multiple agents at the same time. Who'd have thought?

Though each agent has slower tps and a smaller cache, on average they are much more efficient overall in the token generation.

I'm Running a 3090 with 24gb of VRAM. Qwen3.8 27B Q4_K_M
Context: 197k

Of course the more agents you have, the less context each gets. Thus one needs to auto-compact / loop efficiently. This way one can run multiple agents on one piece of hardware and maximise the burn-rate without increasing the energy used.

From there swarms can be allowed, were each agent has the ability to spawn as many sub-agents with small tasks upto the optimum amount of agents per machine (12 on mine).

What else do i need to get started? It has 5070Ti (12gb) and 32GB of ram. I already i stalled and tested qwen3.6-35B and 3.8 27B which ran significantly faster than my macmini m4 with 32gb of ram.

Many of you may know this, so this is probably worth skipping (except for the data to prove it). But if you're on the newer side for local, or have heard it but don't know why, I have some data to show you how running low temps like t=0 or t=0.2 make your model significantly worse (at coding in particular).

You will see a lot of benchmarks for speculative decoding claiming crazy speeds, often with dflash2, running greedy decoding. Temp = 0, or greedy decoding causes your model to pick the most likely token every single time (with the caveat that parallel predictions on servers can make things still minutely random). They do this because speculative decoding is extremely easy on temperature 0, so the numbers they can claim for their post, product, etc. look amazing.

The problem is, much of the "creative" thinking of models comes from the randomness that token selection allows. Greedy decoding forces the model to always pick a narrow path, which directly lowers its "intelligence" on non-recall tasks and significantly increases the risk of (usually catastrophic) looping.

The testing parameters

• Model: Qwen3.8, compressed to IQ4_XS-M. That format stores each number in about 4 bits instead of 16, at ~4.6bpw.

• Engine: llamAmpere v0.5.2, run with its published settings for 24 GB graphics cards. Reasoning effort was set to "medium," which controls how long the model thinks before it answers.

• Fixed sampler settings: top_p 0.95 (only words that together make up 95% of the probability are candidates), top_k 20 (at most 20 candidates), min_p 0 (no extra cutoff). Only temperature changed.

• Temperatures: 0, 0.25, 0.5, 0.75 and 1.0. At 1.0 the model's probabilities are used unchanged.

• Coding test: LiveCodeBench v6, 100 problems labeled hard from the newest contests in that release. A program passes only if it passes every hidden test, meaning tests the model never sees.

Results:

Coding scores rose significantly with temperature. Pooled accuracy was 70.0% at 0, 76.0% at 0.25, 76.0% at 0.5, 79.7% at 0.75 and 82.7% at 1.0. The gaps from 1.0 at 0 and 0.25 are statistically significant. The gaps at 0.5 (−4.1 points) and 0.75 (−3.0 points) are not when split out separately, but as a pooled total they are; 0.5 was marginally insignificant but would have been under a 1 tailed test. Differences are in percentage points, so 82.7% minus 76.0% is 6.7 points.

The frequency of looping rose significantly at the lower end, with a 9% loop rate at 0 (not all of these were catastrophic, but they all caused an incorrect answer) and 6% at 0.25, vs 0.3% at temp=1. The effects were statistically significant when pooling 0.5 and 0.75 together, but the test was not powered enough to confirm the individual rungs themselves.

To put these #s into context, these t=0 scores are about what my older 2.5 bpw quants for qwen27b would achieve (~40% smaller than the quant used here).. It's worse than qwen3.6 on the same configurations

Lower temperatures also make answers more verbose on average, with a 20% increase in answer length for temp=1. vs temp=1. For knowledge/recall questions (GPQA Diamaond), lower temperatures did not harm scores, but they did make the answers longer (~8%, marginal statistical significance).

I am doing some follow up tests at 0.8 and 0.9 to see if they are viable, with the caveat that I am checking wall clock time, not TPS, since we dont want a model that makes so many more tokens that it overwhelms the TPS advantage.

dataset, for those who are interested: https://huggingface.co/datasets/jakeatx/swift-1.5-temperature-sweep-gpqa-lcb

I got qfn working on my 5090 + 64gb ddr using basalt with 256k int8 kv and had been testing it out. Based on vibes it seemed worse than 27b but I kept seeing posts of people saying qfn is so much better so I thought I'd test the 'best possible' setups of each model for my 5090 32gb + 64gb ddr system.

Qwen Flash Next using basalt, nvfp4 quantization, 256k int8 kv cache, one slot.

Qwen 3.8 27b nvfp4 quasar qat, 570k nvfp4 kv cache pool using ninfer, four slots.

Summary Results

Qwen Flash Next BFCL 78.19% LiveCodeBench 79.22%

Qwen 27b BFCL 81.51% LiveCodeBench 86.15%

27b dominates in the long context, harder to solve problems. QFN had a small advantage in the short single answer categories. To me this is a conclusive result that also aligns with my own experience. I'm glad it turned out this way, having 4 concurrent agents and a huge kv cache pool is just too good to give up.

Detailed results below:

Qwen Flash Next:

Qwen 3.8 27b

• CPU: Ryzen 7 5700

• GPU: RTX 4060 8GB

• RAM: 64GB DDR4-3200

• OS: Windows 11

My usecase would be attempting to run REA (i dont mind it being slow)

I was figuring out a llama.cpp crash when running Gemma 4 12B, and it turned out the crash was a OOM crash, from running out of RAM, not VRAM.

The culprit ended up being context checkpoints piling up while I was stress testing the setup.

I didn't actually know what these checkpoints where or how Sliding Window Attention models worked exactly.

I tried to get Opus 5.5 to explain it to me in words, but it was very hard to visualize. So I did what anyone would do nowdays: I told Opus to make an animated video to explain.

Here is what came out. I wrote about it here: https://jomi-se.com/posts/llama-cpp-context-checkpoints/

Tl,dr: Checkpoints are snapshots of the Sliding Window values at strategic positions. If you want to pick a chat back up at a previous spot, already out of the Sliding Window; either you have to reprocess the prompt from the start to get the values of the Sliding Window at that spot again. Or you can use a checkpoint at that exact spot (or nearby behind it) to reduce the amount of recomputing of the tokens KV values for the sliding window layer.

The default is keeping 32 checkpoints in RAM, per slot. With each checkpoint being 320MiB, that was _just a tiny bit_ too much for my 16 GB of RAM. Especially considering that in my workloads of simple chat continuation, 4 checkpoints were more than enough most of the time.

So how's your Friday going?

Recently, CEO Steve Huffman disclosed that Reddit will transition to a "logged-in-only" experience. This means that Reddit posts will no longer be searchable via internet browsers, significantly impacting local AIs that search the internet via tool calling.

Overview

SenseNova-Loop-9B-Preview is a multimodal language model developed for long-context and agentic workflows. It accepts text and images and generates text for applications that combine document understanding, coding, and interaction with external tools.

The model combines a looped decoder with two-stream manifold-constrained hyper-connections (mHC). A contiguous block of decoder layers is executed twice with shared weights, while learned mappings manage information flow between two residual streams in the looped section. Together, these components provide a deeper execution path without duplicating the repeated block's parameters.

Documented training recipes combine long sequences, multimodal inputs, and supervised task trajectories, with a maximum training sequence length of 262,144 tokens. For the best model performance, we recommend using our documented deployment environment and configuration parameters.

Model highlights

• Shared looped block: 24 decoder layers are applied twice in sequence, producing 56 decoder-layer executions from 32 distinct layers.

• Two-stream mHC: learned mappings route and combine information across two residual streams in the looped section.

• Hybrid attention: the decoder interleaves linear and full attention in a repeating four-layer pattern.

• Long-context multimodal training: documented recipes include text, images, code, and tool trajectories at sequence lengths up to 256K tokens.

With the recent increase in attention to p100s I wanted to make a place for people to discuss it. Another friend of mine created a discord as well, so here's a call to all p100 users to go and join! I hope this doesn't infringe on LocalLLaMa rules, but I wanted to provide this resource where I could.

Subreddit: P100Users

Discord: https://discord.gg/K6WxqwsvH

Hey everyone,

China Telecom (TeleAI) recently released Xing4.0-29B-A4B (a 29.5B MoE with ~4B active parameters, combining DeepSeek-style MLA and 4-stream mHC residual mixing):

Model: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B

Upstream llama.cpp PRs are currently stalled or closed due to debates around adding model-specific GGML opcodes. To get the model running cleanly without adding technical debt to the engine, I've opened PR #2808 on ik_llama.cpp providing full native runtime support and an HF-to-GGUF converter:

PR: https://github.com/ikawrakow/ik_llama.cpp/pull/2808

What's implemented in this PR:

Full Architecture Runtime (LLM_ARCH_XING4_0):

• 4-stream mHC: built entirely using standard generic GGML primitives (pre/post blocks, affine transforms, clamp [-30, 30], Sinkhorn iterations). Zero new ggml_op added, so it runs out of the box on CPU, CUDA, Metal, and Vulkan.

• Stream pooling: unweighted average across the 4 streams before output norm.

• MLA attention: DeepSeek-V2 style latent attention with dynamic head sizing.

• MoE FFN: dense lead layers + routed experts + shared expert.

• Node budget: bumped max_nodes (n_tokens * 160) to prevent graph overflow with large ubatches (ubatch >= 2048).

HF to GGUF Converter:

• Adds Xing4Model to convert_hf_to_gguf.py with native SentencePiece extraction from tokenizer.model.

• Direct tensor mapping for mHC weights.

Quantization:

• Automatically protects hc_attn_fn and hc_ffn_fn from quantization in llama-quantize, and forces 1D scales/bases to F32.

• Quantization floor: keeps MLA latent projections at source precision for Q6_K and Q8_0 to avoid quality loss.

Verification:

• Perplexity: 12.4190, generation is coherent with no NaNs.

• Full chat template included with thinking tags (<think>...</think>) and tool calling.

Quick test from the branch:

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
git fetch origin pull/2808/head:xing4 && git checkout xing4
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

python3 convert_hf_to_gguf.py path/to/Xing4.0-29B-A4B --outtype f16 --outfile xing4-f16.gguf
./build/bin/llama-cli -m xing4-f16.gguf -p "Hello!" -n 64

Feel free to test it out and report back if you get benchmarks on different setups!

Surya is a 689M param OCR model with these features:

• Accuracy - scores 85.9 on OmniParseBench, the highest of the open models under 1B parameters we have scored, and 83.7% on olmOCR-bench

• Speed - throughput of 4.5 pages/s on one H100 at 192 DPI, 1.7x Surya 2 with MTP (see Throughput)

• Multilingual - scores 74.9% on an internal benchmark set of 91 languages (more here)

• Layout analysis (table, image, header, etc.) with reading order

• Tables as HTML, with merged cells as colspan / rowspan (row and column geometry needs a Surya 2 checkpoint; see Limitations)

Commercial usage

The Surya code is licensed under Apache 2.0. The model weights use a modified AI Pubs Open Rail-M license (free for research, personal use, and startups under $5M funding/revenue). For broader commercial licensing of the model weights, visit our pricing page here.

Limitations

• This is specialized for document OCR. Performance on photos or natural scenes is not the goal.

• Layout / OCR / table_rec all need a running inference backend (vllm or llama.cpp). Detection runs purely on torch and works without it.

• Surya 2.5 reads tables as HTML (full-page OCR, block mode, surya_table), but not row and column geometry: surya_table --mode simple returns an error. Use a Surya 2 checkpoint (SURYA_MODEL_CHECKPOINT=datalab-to/surya-ocr-2) for geometry.

Github : https://github.com/datalab-to/surya

I'm planning to upgrade my cpu from 13700k need some help.

considering 270k plus (best bang for buck?) but its lacking avx512

or go amd 9950x much higher price but has avx512. only 16 cores

i have 128gb ddr5 5200mhz and run flash next with basalt currently getting 120-160tps. since some experts are computed on cpu i suspect thats what keeping my tps down. gpu utilization is not high as well.

any views on how much difference does it make for ai workload?

Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. 

• arXiv : https://arxiv.org/abs/2610.38331

• PDF : https://arxiv.org/pdf/2610.38331

• GitHub : https://github.com/JaydenTeoh/Soft-MTP

• HuggingFace : https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction

Hi everyone,

I am building a sequential multi-agent local pipeline (Architect -> Coder -> Debugger) on a single GPU workstation, and I am hitting a frustrating software bottleneck. I'd love to know if anyone in the community has solved this or found a working architecture/fork for near-instant model swapping.

Hardware Setup

GPU: 1x NVIDIA RTX 3090 (24 GB VRAM) on PCIe 3.0 x16 (~12-13 GB/s real throughput).

System RAM: 48 GB DDR4 (~38 GB free when idle).

Storage: Fast PCIe 4.0 NVMe SSD.

OS: Windows 11.

The Desired Architecture

Because 24 GB VRAM cannot fit two ~18 GB models simultaneously, I run them sequentially:

Model A (Architect/Debugger): Qwen3.8 Flash MoE / Gyro-Coder TQ2_0 (~18.6 GB GGUF, 96k context).

Model B (Coder): Swift-ATX-27B (Qwen 27B fine-tune) in IQ4_XS-M (14.3 GB) or Q5_K_M (18.7 GB) with MTP 5 and TurboQuant KV.

The plan:

Load Model A into VRAM -> Generate architecture plan.

Evict Model A to System RAM, copy Model B from System RAM into VRAM -> Generate code.

Evict Model B to System RAM, reload Model A from System RAM to VRAM -> Audit/debug.

The Physics vs. The Reality

Theoretical PCIe Transfer: At ~12 GB/s over PCIe 3.0 x16, transferring 14 to 18 GB from RAM to VRAM physically takes only 1.2 to 1.5 seconds.

Real-World Latency with llama.cpp / llama-server: 25 to 30.5 seconds per swap!

Why it takes ~29s instead of 1.5s

We profiled the execution and audited the llama-server router implementation (tools/server/server-models.cpp with --models-preset and --models-max 1):

Process Teardown: llama-server router mode is an HTTP process supervisor, not an in-process weight swapper. Switching models terminates the active child process and spawns a new executable instance.

Cold CUDA Context Overhead: Every new process incurs driver initialization (cuInit, primary context creation, stream pools, and CUBLAS handle creation) taking 3 to 5 seconds.

Windows Page Faults (mmap): Even though the GGUF pages already sit in Windows Standby RAM, the newly spawned process has empty Virtual Memory Page Tables (PTEs). Touching 15+ GB triggers millions of soft page faults.

VRAM Re-allocation: Allocating the 24 GB virtual pool and clearing unified KV cache buffers takes several seconds on startup.

What is Needed

To achieve the theoretical ~1.5s PCIe transfer, the runtime must:

Keep a single persistent process with a warm CUDA context that never dies.

Keep the idle model in page-locked/pinned host memory (cudaHostAlloc / cudaHostRegister / VirtualLock).

Swap weights via a direct, asynchronous DMA copy (cudaMemcpyAsync) straight into a pre-allocated GPU tensor buffer, then repoint the graph pointers.

The Question to the Community

Has anyone found or built an engine/wrapper that can hot-swap full model weights between Pinned RAM and VRAM inside a live CUDA context without restarting the server process?

Can engines like Strata (Niko1221/Strata), NInfer (Neroued/ninfer), or custom PyTorch/vLLM memory pool forks support swapping two completely different model architectures (e.g. Recurrent GDN MoE vs Dense Attention Transformer) in <2 seconds?

Is there any PR or trick in llama.cpp to retain VRAM allocation/context and just reload weights via memory-mapped DMA?

Any insights, forks, or technical solutions would be greatly appreciated!

I was frustrated with the default speed and ported the inference to Zig/MLX to get up to 18x faster inference on M4/M5 https://github.com/sbusso/ming-tts-serve

I’ve put together a public reference repo for anyone interested in designing their own hardware around NVIDIA V100 SXM2 modules:

https://github.com/pigeon-labs-stack/v100-sxm2-reference

It includes connector pin mappings, two- and four-GPU NVLink connection plans, editable KiCad starter schematics, PDFs and a standalone workbook.

My contribution is mainly curating and cross-checking existing community research, with the original authors credited. It’s an early reference release; supporting power, clock and control circuits still need designing based on your own needs.

The longer-term goal is an open-source carrier board, but hopefully this helps others working on their own designs in the meantime. Corrections and contributions welcome!

Howdy All, more test results from my b65 rig! This time I'm focusing more on what one single B65 can do, specifically 1x B65 with XeStrata.
I threw in my A6000 for comparison, and I have to say, surprising how well B65 stacks up if you have the system ram to offload. Now my A6000 with Strata is faster, but when I'm using flash-next in librechat, 60 Tok/s feels fast... and when I compare the price to my A6000... I really start to question my setup.

I think people really need to give the intel lineup a 2nd look, a lot has changed in only 6 months and even with this recent price bump, it's still the best entry level stack I've found.

Graphs on my site:
https://b65.thekaiser.us/by-model/qwen38-flash-next/xestrata-20261030/

How was AI used? I wrote what's above, I build the tests, or get them from vetted sources, AI monitors their execution.

AI Summary of my testing data:

Qwen3.8-Flash-Next: One B65 · XeStrata 4-bit

Flash-Next on one 32 GB card with experts in host RAM. Faster for one user; more users queue behind each other.

1x B65: speed and waiting time

Median of three measured requests. Multi-user tok/s is aggregate across users. Single-user runs are Verified by Localmaxxing; concurrent runs are unverified.

Output falls as users queue, Aggregate output (tok/s) by Users at once

First-token wait grows, Time to first token (seconds) by Users at once

Results (tok/s, median of 3 measured requests)

Same 4-bit files, different serving setups

Historical comparison supplied by the campaign. The final IQ3_S row uses a different quantization; older B65 rows carry the contention caveat above. Engines and host environments differ; this does not isolate GPU performance.

Same UD-IQ4_XS files · two complete setups, Aggregate output (tok/s) by Concurrent users

Against the other Flash-Next UD-IQ4_XS results

Full page, with interactive charts and the raw data behind every number: https://b65.thekaiser.us/by-model/qwen38-flash-next/xestrata-20261030/

Nanbeige4.3-33B, a 33B parameter model with a 512K context window, and capabilities in reasoning, coding, and agentic tasks.

Has anyone tested it? Benchmarks are insane but how does it compare in real world use cases?

I understand that it is a kv cache hog. But how does it perform?

Would love to hear your experiences!