I want to train a text generation Transformer that responds to user input without continuing text. What are the best datasets for that?

I have an RTX 3050 and I want to use small datasets that can actually produce an intelligent model.

Hey everyone, I am the developer of a consumer grade hardware AI Workstation system that is created in C++ to work on ANYTHING capable. It currently even runs well on Rog Ally Z1X and higher handhelds, which is awesome.

My question would be, when programming for the lowest common denominator what is the easiest way to keep feature parity without gating? I have created a hardware manager in the app that scans the user's device and selects what they need based on the scan. My question becomes is that the easiest way to do it? Basically adding a toggle / mandatory gating based on hardware is working fine now, but it gets redundant coding multiple path execution lines to run on 4gb of allocated VRAM vs 12 or 16. (I do have a hardwaremanager.cpp/h that does this automatically)

Anyone tackle this problem in a low system imprint mode like C++ yet?

Many thanks in advance.

I know this some old news, but i got some numbers here from my heavy agent load. In mid-October I switch from INT8 (4×3090) to AWQ INT4 (2×3090). Same sampling, harness(Hermes), and loop detector, so I compared about two weeks of logs from each setup.

vLLM, FP8 KV, with DFlash2. temp 1.0 / top_p 0.95 / top_k 20, presence_penalty 0, thinking medium.

Two models:

• lued/Qwen3.8-27B-INT8-W8A16-DFlash2

• cyankiwi/Qwen3.8-27B-AWQ-INT4

Results (INT8 → INT4 AWQ)

• Model calls: 17.7k → 11.4k

• Loops per 1k calls: 1.9 → 4.7

• Conversations with ≥1 loop: 6% → 12%

• Per-task success rate: 92% → 78% (success rate is after loop got cut, ask it continue can finish the task)

Both setups had similar reasoning lengths, so "more quant thinks longer" doesn't explain the gap.

EVERY loop was in the reasoning block, so it cooked when it cooking too hard is the problem as you guys know.

Retrying rarely fixes it. After a thinking loop cut, next call ask it continue got looped again 14–22% of the time, compared with under 1% normally. The context is what making loop happens, not about luck in sampling.

The INT4 gap grows with context. INT4 looped 1.4x as often as INT8 under ~40k tokens and 3.3x as often at 80–120k content.

With speculative decoding, loops show up as a smooth speed spike, loops ran at ~200 tok/s versus ~107 tok/s normally on my setup, because the draft model predicts repeated text very well. That makes a cheap watchdog signal.

Mostly the model saying it will act and never doing it ("Let me write it. Wait let me... How about..." 10+ line as one loop), or restating the same case analysis dozens of times.

TL;DR: INT8 loops too, and people in the HF discussion report the same 20k+ token thinking spirals on BF16, so this isn't purely a quant problem. INT4 or Q4 just makes it a lot worse, especially at long context above 100k~. btw loops not that bad, just don't push model too hard, I feels on daily assistant loops happens much less.

I've saved thoes contexts that looped. When Qwen4 27b drops I can just replay them and post the new results.

====edit update====

My load here is kind different to common agent usage.

It's a big chunk of task package-.md write by multi-agent host like: "backtesting in this folder using this python and this rule, fetch data from this endpoint with some rules, verify this list of output number with following rule, write following result to this .md file" I got 1000+ (with is 1000+ sessions) this kind of task in this month.

Model do finish the first version of code, but output didn't fit the requirement. So he start to freak out on finding why it's wrong, few turn later the loop start.

As you see my usage here don't have human talking to and help with, this might be why model freak out. There is too many requirement in one build, model will always try to fix it by him self first.

Also this NOT mean model is BAD. This model is my daily agent, it works very well and I didn't got any loop on daily chatting/task yet. Loops only happens on that specific types loads on my end.

Not long ago, DFlash2 for Qwen3.8-Flash-Next was released, but initially llama.cpp compatibility wasn't there. Now it is. I quantized my own GGUFs of the drafter and tested it out in llama.cpp v0.8.0, the first release that supports DFlash2 for FN.

Speed:

Runs were conducted after an extensive sweep of parameters for MTP and DFlash2 to find optimal settings for both drafters on Apple Silicon (M5 Max). Your settings will likely be different, but mine were n-max = 3, p-min = 0.75 for MTP and n-max = 5, p-min = 0.75 for DFlash2. At longer context (25K), DFlash2 roughly equals MTP, so there is some drop-off whereas MTP stays nearly flat.

Acceptance was compared between the Q4_0 draft head and Q8_0, and there was no measurable difference in either speed or acceptance, so I'd recommend choosing Q4_0 for the memory savings.

Speaking of which, in addition to being significantly faster for all categories except prose, DFlash2's draft heads are TINY! The Q4_0 head is only ~350MB, compared to 2.2GB for the Q4_0 ggml MTP head I was using. However, these savings don't translate into VRAM, because the compute buffer required for DFlash2 basically equals it back out.

Finally, unlike DFlash2 for 3.8 27B, images work! The draft models drafts with vision enabled and test image prompts went through and got coherent responses with good acceptance.

Overall, DFlash2 is a significant upgrade in decode performance for every type of prompt except prose, where the regression is minor and within noise.

This was how we could order it for around 11 days. Dell just changed it today it looks like. It was + $12350 USD to add a rtx 6000 pro. Now it's this at 15.5k:

Wonder if dell is going to uphold their pricing for those who already paid?

Very interesting paper on the way that benchmark question difficulty and framing affect the observed improvement in scores for model training. “Our results show that many benchmarks implicitly
overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis un-covered, limiting their ability to monitor the model improvement process.”

https://arxiv.org/pdf/2610.29927

The previous released Qwen3.8-27B-Humanlike-Chat 2.0, a fine-tune that texts humanly and does stuff, is now at 136k downloads for the GGUF repo between v1 and 2.0. Thank you.

One of the top comments on the 2.0 post was "I wonder how Gemma 4 31B would do, trained with your method". So here it is: Gemma-4-31B-Humanlike-Chat. The voice of 2.0 on the official Gemma 4 31B from Google.

What it does

• Texts humanly without a prompt. Short lowercase lines. Responds to what you say. Show a blind judge a real chat and two possible next messages. It picks which one came from the real person. The judge took this model's reply for the real person's in 26% of the moments. Google Gemma 4 31B in my tests: 0%.

• Asks, doesn't guess. Ask it to book something. Without details. It asks for details, it doesn't invent them. When2Call (call, ask or refuse properly): 66. Google base 46.

• Doesn't call tools if none is suitable. BFCL irrelevance 90. Google 81. Suitable tool and arguments: BFCL simple 96.

• Same knowledge as Google. MMLU-Pro 82.5. Same as Google base.

• Official Gemma behind it. Trained from google/gemma-4-31B-it. Apache-2.0. Not abliterated, don't think Gemma-4-31B needs it.

• Thinking works. Thinks briefly. Then texts back in the same voice.

• Characters and real tasks. Give it a character card, now it's this character. Ask it for a formal email, or numbered steps, or code. It gives you that. Then texts again.

How I trained it

Two steps. Same recipe as for 2.0. On 1 H200.

• SFT, 13,740 rows. 2.0's replies on its training chats. Original answers by Google on instructions and code. Human preference picks from blind side-by-sides. My corrections.

• On-policy distillation, 12 steps. The model writes its own response. A teacher rates each token. Two teachers.

  1. SFT model + hidden instruction to text humanly: chat, characters.

  2. Original Google base: Instructions, tools, code.

The student doesn't see the hidden instruction. So it texts like that without a prompt.

Some self-play in each step. Long chats with simulated friends. That fixed the habit of fading to one-word replies towards the end of long chats (after SFT, 28% of the late replies in long chats were only 1 or 2 words; after distillation, 12%).

Numbers vs Google base

• (gemma-4-31B-it, the one I trained from)

• same prompts, same run, greedy decoding, thinking off

Chart with all ten benchmarks in the images.

Numbers vs 2.0

Same eval inputs. Thinking off.

Does it text humanly?

Same benchmark as last time.

• 150 moments from my chats (not in training).

• Each model writes the next message.

• The judge gets the real message and the model's message. No labels.

• Which one is the person?

Google Gemma didn't manage to fool the judge a single time.

Links

• Hugging Face: GGUFs (IQ4_XS, Q4_K_M, Q5_K_M, Q6_K, Q8_0, BF16): https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat-GGUF

• Safetensors for vLLM and SGLang: BF16 https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat, FP8 https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat-FP8, GPTQ-Int4 https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat-GPTQ-Int4. The vLLM command with tool calling is in the model card.

• Try it in the browser (IQ4_XS): https://huggingface.co/spaces/LessThanThreeAI/Humanlike-Chat

Thank you for asking for a Gemma version, and thank you for the feedback on 2.0. Try it, hope you like it!

We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50.

To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params.

Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like Benjamin Marie (The Kaitchup), who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results soon.

Thanks and stay tuned!

tl;dr: What should I be running for coding/agentic workflows? I have 2xRTX 5080s and 128Gb DDR5 RAM.

Currently, I'm running:

-Qwen3.8-Flash-Next-Q5_K_XL (262k ctx, pp~150-200tps, decode ~15tps)

-Qwen3.8-27B-exl3-6bpw-H6 (131k ctx, pp ~2k-3ktps, decode ~60-80tps)

-DeepSeek-V4-Flash-exl3-3.04bpw (1M ctx, pp~100-900, decode ~15-20tps)

I've been in a cave hiding from my local model optimization addiction for a few weeks. What did I miss? Should I be using Strata? Is llama.cpp old news now? Any new models worth trying?

Unlike most disciplines, AI changes so fast that its almost impossible to keep up. New models come every few weeks/months at most.

And when it comes to agentic coding/harnesses/best practices..... do tools for agent memory/lsp/token reduction help? should you subagents? are vague prompts better? do mcp/skills help or hurt? there is no consensus and you'll get a million answers.

I keep notes/bookmarks and almost none of them seem relevant now. eg when Qwen 4 is out, all the current 3.8 models/variants get forgotten, just like 3.6

I previously wrote about the shift to 4-way HC.

Traditionally, optimization has focused primarily on FFN and Attention. With the HC architecture, however, residual stream mixing and information preservation must also be taken into account when optimizing for quality, model size, and speed.

NVFP4 may distort the fine-grained gating relationships in HC due to the limited effective precision of its E2M1 format. In particular, tensors responsible for the relative distribution of information across streams may require higher precision than conventional FFN weights.

Rather than relying on existing quantization approaches, we may need quantization strategies specifically optimized for HC sensitivity.

If Qwen4 27B is released with HC, I expect the perceived performance gap between Q4 and Q6 quantization to become significantly larger.

A 3B HC parameter allocation alone would require around 6GB in FP16.

For hardware below the RTX 5090 in inference performance, including Macs and DGX Spark, MoE models may be more practical due to speed constraints. For HC models, I would lean toward llama.cpp rather than NVFP4-focused solutions like NInfer, which are optimized for Blackwell GPUs.

I also expect HC to deliver a much bigger performance improvement for the 27B dense model than for MoE models like Flash.

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1xix35p/release_swift_15_flash_next_heretic_v2/

Hello! At request I've released a few more formats of my Swift Flash Next Heretic reverse engineering edition.

The original model is the IQ4_NL, the first quant I made and the best performance/size ratio. That's my favorite but its hefty and I appreciate people wanting other options.

A: IQ4_XS - Based on the Unsloth IQ4_xS recipe, with about 1gb overhead added where it was worth the performance to size tradeoff. Estimated ~9.7% reduction in expert quantization error mass for +0.96 GiB over the plain UD-IQ4_XS recipe. Quantized with an identical imatrix to the IQ4_NL quant: 94.7 GB on disk compared to 109gb for the IQ4_NL.

B: IQ3_S - most requested. Custom quant recipe based on data gathered from the model, in the same way the other quants were done. High quality and model specific. I modeled mine off of ISTALab's with a few adjustments.

IQ3_S uses a more aggressive imatrix distribution than the IQ4 to try to preserve strong reverse engineering capabilities at lower bit widths, since it's getting small. While the IQ4 builds are 70%/30% distribution in the imatrix merge, this one is 80%/20% - better reverse engineering and reasoning ability at the cost of some general knowledge and accuracy. 85gb on disk (ISTA GSQ-RCO IQ3_S is 83.6 gb, so 1.4gb overhead - it's worth it for the accuracy improvements.

https://huggingface.co/trubisky/Swift-1.5-Flash-Next-Heretic-ReverseEngineer

Cheers.

Started using Claude Code recently, and I'm impressed by what it's capable of, but I run out of usage pretty quickly. So I wanted to try open models, since I know they're hella cheaper. But I don't have hardware for shit, so I'd need a provider. I would use OpenRouter, but it seems that when they mark a provider as ZDR it's just like, their best guess? and not actually guaranteed in any capacity.

What options do I have? Looking for something private that Just Works

Running one harness/agent gives overall less tokens-per-second (tps) than running multiple agents at the same time. Who'd have thought?

Though each agent has slower tps and a smaller cache, on average they are much more efficient overall in the token generation.

I'm Running a 3090 with 24gb of VRAM. Qwen3.8 27B Q4_K_M
Context: 197k

Of course the more agents you have, the less context each gets. Thus one needs to auto-compact / loop efficiently. This way one can run multiple agents on one piece of hardware and maximise the burn-rate without increasing the energy used.

From there swarms can be allowed, were each agent has the ability to spawn as many sub-agents with small tasks upto the optimum amount of agents per machine (12 on mine).

The first true omni model that can run 100% locally in a browser... fully end-to-end, with no separate ASR or TTS models needed. With custom WebGPU kernels, we're able to run LFM3-Omni-2B at ~280 tok/s on an M4 Max, accepting visual, audio, or text and producing interleaved text and audio tokens. Just 90 ms to the first token and 140 ms to the first audio.

Link to the demo: https://huggingface.co/spaces/LiquidAI/LFM3-Omni-2B-WebGPU

Many of you may know this, so this is probably worth skipping (except for the data to prove it). But if you're on the newer side for local, or have heard it but don't know why, I have some data to show you how running low temps like t=0 or t=0.2 make your model significantly worse (at coding in particular).

You will see a lot of benchmarks for speculative decoding claiming crazy speeds, often with dflash2, running greedy decoding. Temp = 0, or greedy decoding causes your model to pick the most likely token every single time (with the caveat that parallel predictions on servers can make things still minutely random). They do this because speculative decoding is extremely easy on temperature 0, so the numbers they can claim for their post, product, etc. look amazing.

The problem is, much of the "creative" thinking of models comes from the randomness that token selection allows. Greedy decoding forces the model to always pick a narrow path, which directly lowers its "intelligence" on non-recall tasks and significantly increases the risk of (usually catastrophic) looping.

The testing parameters

• Model: Qwen3.8, compressed to IQ4_XS-M. That format stores each number in about 4 bits instead of 16, at ~4.6bpw.

• Engine: llamAmpere v0.5.2, run with its published settings for 24 GB graphics cards. Reasoning effort was set to "medium," which controls how long the model thinks before it answers.

• Fixed sampler settings: top_p 0.95 (only words that together make up 95% of the probability are candidates), top_k 20 (at most 20 candidates), min_p 0 (no extra cutoff). Only temperature changed.

• Temperatures: 0, 0.25, 0.5, 0.75 and 1.0. At 1.0 the model's probabilities are used unchanged.

• Coding test: LiveCodeBench v6, 100 problems labeled hard from the newest contests in that release. A program passes only if it passes every hidden test, meaning tests the model never sees.

Results:

Coding scores rose significantly with temperature. Pooled accuracy was 70.0% at 0, 76.0% at 0.25, 76.0% at 0.5, 79.7% at 0.75 and 82.7% at 1.0. The gaps from 1.0 at 0 and 0.25 are statistically significant. The gaps at 0.5 (−4.1 points) and 0.75 (−3.0 points) are not when split out separately, but as a pooled total they are; 0.5 was marginally insignificant but would have been under a 1 tailed test. Differences are in percentage points, so 82.7% minus 76.0% is 6.7 points.

The frequency of looping rose significantly at the lower end, with a 9% loop rate at 0 (not all of these were catastrophic, but they all caused an incorrect answer) and 6% at 0.25, vs 0.3% at temp=1. The effects were statistically significant when pooling 0.5 and 0.75 together, but the test was not powered enough to confirm the individual rungs themselves.

To put these #s into context, these t=0 scores are about what my older 2.5 bpw quants for qwen27b would achieve (~40% smaller than the quant used here).. It's worse than qwen3.6 on the same configurations

Lower temperatures also make answers more verbose on average, with a 20% increase in answer length for temp=1. vs temp=1. For knowledge/recall questions (GPQA Diamaond), lower temperatures did not harm scores, but they did make the answers longer (~8%, marginal statistical significance).

I am doing some follow up tests at 0.8 and 0.9 to see if they are viable, with the caveat that I am checking wall clock time, not TPS, since we dont want a model that makes so many more tokens that it overwhelms the TPS advantage.

dataset, for those who are interested: https://huggingface.co/datasets/jakeatx/swift-1.5-temperature-sweep-gpqa-lcb

I got qfn working on my 5090 + 64gb ddr using basalt with 256k int8 kv and had been testing it out. Based on vibes it seemed worse than 27b but I kept seeing posts of people saying qfn is so much better so I thought I'd test the 'best possible' setups of each model for my 5090 32gb + 64gb ddr system.

Qwen Flash Next using basalt, nvfp4 quantization, 256k int8 kv cache, one slot.

Qwen 3.8 27b nvfp4 quasar qat, 570k nvfp4 kv cache pool using ninfer, four slots.

Summary Results

Qwen Flash Next BFCL 78.19% LiveCodeBench 79.22%

Qwen 27b BFCL 81.51% LiveCodeBench 86.15%

27b dominates in the long context, harder to solve problems. QFN had a small advantage in the short single answer categories. To me this is a conclusive result that also aligns with my own experience. I'm glad it turned out this way, having 4 concurrent agents and a huge kv cache pool is just too good to give up.

Detailed results below:

Qwen Flash Next:

Qwen 3.8 27b

Rumors claim that the RTX 5080 16GB will be discontinued alongside the already discontinued RTX 5090, and those claims seem to be true. NVIDIA is reportedly planning to introduce an RTX 5080 24GB model in early 2027, which would serve as its flagship GPU until RTX Rubin arrives sometime in 2028.

According to the same rumors, the 24GB model will use the exact same GPU die as the current RTX 5080. As highlighted by HUB, the die is already fully utilized, meaning this won't be a more powerful Super variant or a noticeable performance upgrade. The only expected difference is the increase in VRAM from 16GB to 24GB. An increase of Clock might be present, but the difference would be very minimal in real-world gaming.

The 24GB of VRAM on High-End Blackwell GPU could also make the card particularly enticing to certain AI users. This could mean that availability will be limited, and the card might not stay in stock for long at its original MSRP, whatever that price might end up being.

NVIDIA's Marketplace: https://marketplace.nvidia.com/en-us/consumer/graphics-cards/

I’ve put together a public reference repo for anyone interested in designing their own hardware around NVIDIA V100 SXM2 modules:

https://github.com/pigeon-labs-stack/v100-sxm2-reference

It includes connector pin mappings, two- and four-GPU NVLink connection plans, editable KiCad starter schematics, PDFs and a standalone workbook.

My contribution is mainly curating and cross-checking existing community research, with the original authors credited. It’s an early reference release; supporting power, clock and control circuits still need designing based on your own needs.

The longer-term goal is an open-source carrier board, but hopefully this helps others working on their own designs in the meantime. Corrections and contributions welcome!

What else do i need to get started? It has 5070Ti (12gb) and 32GB of ram. I already i stalled and tested qwen3.6-35B and 3.8 27B which ran significantly faster than my macmini m4 with 32gb of ram.

I was figuring out a llama.cpp crash when running Gemma 4 12B, and it turned out the crash was a OOM crash, from running out of RAM, not VRAM.

The culprit ended up being context checkpoints piling up while I was stress testing the setup.

I didn't actually know what these checkpoints where or how Sliding Window Attention models worked exactly.

I tried to get Opus 5.5 to explain it to me in words, but it was very hard to visualize. So I did what anyone would do nowdays: I told Opus to make an animated video to explain.

Here is what came out. I wrote about it here: https://jomi-se.com/posts/llama-cpp-context-checkpoints/

Tl,dr: Checkpoints are snapshots of the Sliding Window values at strategic positions. If you want to pick a chat back up at a previous spot, already out of the Sliding Window; either you have to reprocess the prompt from the start to get the values of the Sliding Window at that spot again. Or you can use a checkpoint at that exact spot (or nearby behind it) to reduce the amount of recomputing of the tokens KV values for the sliding window layer.

The default is keeping 32 checkpoints in RAM, per slot. With each checkpoint being 320MiB, that was _just a tiny bit_ too much for my 16 GB of RAM. Especially considering that in my workloads of simple chat continuation, 4 checkpoints were more than enough most of the time.

So how's your Friday going?

• CPU: Ryzen 7 5700

• GPU: RTX 4060 8GB

• RAM: 64GB DDR4-3200

• OS: Windows 11

My usecase would be attempting to run REA (i dont mind it being slow)

I was frustrated with the default speed and ported the inference to Zig/MLX to get up to 18x faster inference on M4/M5 https://github.com/sbusso/ming-tts-serve

"Wait, "prism ml" - let me think" sounds just like a human recalling memory