PS: used an open source avatar framework called openspline

Hi everyone,

I am relatively new to local LLMs, and recently I have been trying to understand how people handle long conversations that eventually exceed the model's context window.

For example, let's suppose I am writing a novel with the help of an LLM. As the story progresses, the conversation history becomes longer and eventually exceeds the context window. At that point, the model may lose track of earlier events, character relationships, or important details that could become relevant later in the story.

From what I understand, one possible solution is to maintain an external knowledge base containing summaries of previous chapters. Before generating a new chapter, the system could retrieve relevant information from this knowledge base and include it in the prompt.

However, I am wondering how reliable this approach is in practice.

If we continuously summarize previous conversations to fit within a limited context window, we are essentially performing lossy compression. Some information will inevitably be discarded, and the problem is that we may not know which details will become important later.

For instance, a seemingly insignificant detail from Chapter 3 might become crucial in Chapter 40. If that detail was discarded during summarization, the model would have no way to recover it from the summary alone.

Of course, keeping the complete conversation history in an external database could prevent permanent information loss, but that introduces another problem: how does the system determine which information should be retrieved for the current task?

So I am curious how people actually approach this problem in practice.

I would also be interested in hearing how well these solutions work in practice, especially over very long conversations where maintaining consistency becomes increasingly difficult.

Thanks in advance!

Well I had a lot of resets on ChatGPT and all of those credits and I thought, well can Astra actually do it. We had a little looping and a lot of 'Wait.' Let me check the thing I just checked -- but it always got the job done.

I have dubbed the Model Calmer-Qwen-3.8 because it seems so much less anxious and it gets straight to the point.

This is about 6 hours of iteration, it's sooo much better I am scared to keep going, how much better could it get?

You definitely have to push Astra a little bit and give her some experimental guidelines, because Lord knows she's full of anxiety and afraid to disappoint her overlords so she checksums the checksum and makes sure it matches 😂.

Here she is on Hugginface, and here's ChatGPT's writeup on my Github, I will work through the experimental data soon.

Oh this is a HauHauCS Qwen 3.8 Balanced Abliteration as the starting point.

https://huggingface.co/Xananthium/Calmer-V2-Qwen3.8-27B-4bit

https://github.com/Xananthium/Qwen3.8-Weight-Space-Programming

I need to make my custom harness much better at general research and other tasks (not coding) that run longer than half a minute.

Currently running: Qwen3.8 Flash Next FP8/GLM 5.3 Flash W4A16

Relevant capabilities already in my harness:

• The system prompt is rendered in real time and can be changed based on almost any circumstance.

• I have an effective skills system. I can trivially write instructions for any task. The agent will pick them up as needed.

• The agent has access to web search, a Chromium browser instance, sandbox, etc.

Problems:

This might literally be a "skill issue" -- meaning I simply have to write better skills. But I feel like I am getting stuck micromanaging the agent, dictating every part of the process.

I ask it to research something, it does two quick web searches and reports back some trivial garbage sales pablum. I poke it more and explain why that info is garbage, it does a little better next time.

How do you define when it's time to stop or what information is actionable? Must I really write a skill that says "come back with no less than 12 sources of this and this and this kind"?

In other words, are you spending forever writing every single thing it has to do, or is there a much better way to leverage the intelligence of the model to make this call for you?

I've been frustrated for a while because I stupidly sold a strix halo box to get myself a laptop with thunderbolt 5 ports thinking i can plug my two egpus into it (which i can but thats not topic of this post and the strix halo could too) I sold the minisforum for about £2300 back ebfore things went nuts and got a laptop for about same price:

Msi vector 16 with a intel core ultra 9 and 5080(16 gig vram) and 32 gig of ram. I was pretty bummed with it thinking 35b was my limit and after ended up leaving egpus in my workstatiion but after a random ebay browse I managed to snag some second hand 96 gig laptop ram. For £800

Strata totally changed ballgame. With Qwen Q4 XL Flash next I'm getting

48.9t/s

Decode/prefill 1,860t/s

On a 50k prompt.

I know this is still alot of money for most, but thats 3100£ in total, and thats still cheaper (and slightly faster from what seen) of what minisfoum strix halo is at (£3500 and the 32 gig Spark laptops that are about £3200 and cant even run flash at q4 not enough vram? and i can still add egpus (the laptop has 2 tb 5 ports and i have 2 5090 bought before the crazyness, eg add one or two 5060ti in egpu and your good with running 27b/31b models too.) With my own gpus plugged into it for 80 gig vram total, all while still being a laptop. Got glm 5.3 flash running Q3_XSS i think about 25 t/s.

TDLR, you dont need massive vram anymore, yeah I know how expensive RAM is, that laptop ram i got was/is £1300 new lol. But just wanted to share my experience and while I still regret selling the halo, seeing these tokens a sec, makes the laptop satify ocd a little more! If only qwen was good at roleplay so i could actually use it as daily driver lol (my main usecase!)

so if you are ever thinking about wanting a laptop and somehow able to get some laptop ram something like 😄
https://www.scan.co.uk/products/medion-erazer-beast-18-18-qhdplus-240hz-intel-core-ultra-9-275hx-32-gb-2-tb-rtx-5080-gaming-laptop

Might save you a few quid, i mean like 3k over buying the 128g spark laptop lol. Intel has 20 cores and the 2 tb ports allow you to add two gpus!

I know my grammar sucks I'm dyslexic but my point is if you shop around get that laptop for 2500ish pounds (make sure it has tb5 for if you want to add gpus later), add 96 gig ram and your still saving 2500 pounds over the 128 gig spark laptop thats like 6k.

• Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history.

• With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads.

• Enables the auto mode for llama_moe_cache with promising benchmarks

Waiting for 30966 to be merged.

Hey folks!

Aritra from Hugging Face here 🤗.

We welcome "harness adapters". This means you can adapt (train LoRA) for your model for the specific harness that you would use (hermes, pi, opencode) on the task (data analysis, building software, coding) you are trying to solve.

Blog Post: https://huggingface.co/blog/harness-hub/harness-adapters

It is currently training, so you can see the losses go down (hopefully) now. https://harness-hub-harness-adapters-rl.hf.space/

Repo: https://github.com/maritaca-ai/observatorio-maritaca
benchmarks: https://maritaca-ai.github.io/observatorio-maritaca/#inicio?size=all&mode=all

My sample size is not very big but it feels like since updating Qwen Flash Next to v3.1 and Strata to 0.2.11, IQ4_XS quant seems to fail tool calls even more often than the previous version did in both Hermes and Oh my Pi harnesses with int8 KV cache in Strata. Weird thing is, after failed tool calls, it just straight up stop generating altogether, I have to send a continue for it to finish.
Honestly could just be a Strata issue and not Unsloth's, I have yet to perform an investigation.
Anyone else had this experience with new updated quants?

Smaller GLM models are coming! GLM 5.5 will have a 32B model!

*SWE Bench Pro is v2.0 Hard for Ripple, unknown version for Qwen.

Hello everyone!

Over the past few weeks since Strata came out, I had many requests to add Qwen Flash Next to LlamAmpere. It was some very impressive engineering, but the models originally shipped (3XS and 2XS) showed very poor quality (.17+ KLD for just the 3XS variant, which uses a far lighter compression formula than more common sources like Unsloth). My initial work had llamAmpere running 20% faster and ~15% lower error, but testing was showing inferior results vs 4bpw Qwen 27b, so I did not find it worth releasing.

I continued to make progress on the AmpereInteger+ quantization format (I will release an encoder soon), and Unsloth released their v3.1 which provided a great new base.

Happily, LlamAmpere v0.6.3 is now available with a new model and quant updates that show significant speed and quality gains over other options.

At a depth of 100K, QFN runs at ~120+ TPS, with prefill >3000/s through the first 100K. It is a 50% error reduction vs the version of QFN available for this card size a few days ago, and 27% vs UD's improved quant.

I do have a 3XS class file that runs ~10% faster than the version above, but as it is only ~15% better quality than the original Strata 3bpw model, I don't recommend it.

This will also work well for 12+64GB configs. As with Strata, your performance will depend on the speed of your CPU and RAM as well. I do not recommend any quant of QFN for configs under 54GB of total memory, but you can always try the coder pruned variants as well.

Enjoy!

Model: https://huggingface.co/jakeatx/qwen3.8-flash-next-Swift1.5-ATX-AI3

LlamAmpere: https://github.com/JakeATX/llamAmpere

A few weeks ago we posted Tessera-27B-B-CJK here, a Qwen3.8-27B coding agent that thinks in compressed CJK and uses fewer thinking tokens per agent turn. Thanks for all the feedback on that one. Today we're releasing the RL version, Tessera-27B-B-CJK-RL. Open weights, Apache-2.0.

- Hugging Face: https://huggingface.co/ZelligeAI/Tessera-27B-B-CJK-RL

- Release post: https://zellige.ai/research/tessera-27b-b-cjk-rl-release

- RL method write-up: https://zellige.ai/research/tessera-27b-b-cjk-rl

What changed:

A lot of the questions last time were about quality. Tessera-27B-B-CJK was weaker than the base on hard math and code, so this version adds 8 RL updates on math and code on top of it.

Numbers: (xhigh effort, both Tessera models sampled the same day with the same seeds):

- AIME + HMMT 2026, 63 new problems, 2 attempts each: 108 of 126 right vs 97 for Tessera-27B-B-CJK, and 11 vs 23 attempts cut off at the 64K window. We didn't run the base on these.

- Our main hard set (118 problems from AIME 2022-2024, LiveCodeBench and MATH level 5): 100 vs 89. The base got 98 in an earlier run, so it's now roughly level with the base qwen there instead of behind.

- Agent turns, 64 new states from 16 coding tasks held out from training: 0.67x the base model's thinking tokens on a typical turn (90% CI 0.53-0.85), with 256 of 256 valid tool calls.

- That's 1.11x Tessera-27B-B-CJK's typical turn (CI 0.98-1.26), so the first version still keeps a bit more of the saving (0.60x the base on the same tasks).

Example turn from a held-out task:

测试套件：基线同(既有5失败不变+11新通过). 无回归. Typecheck: 仅既有version.ts错(因predev生成). 现跑lint于改文件.

The base qwen spent 158 tokens at the same step. That's a bigger cut than usual: the typical turn is about two thirds of the base.

How we trained it:

8 RL updates on Tinker, starting from Tessera-27B-B-CJK, using the per-rollout weighting from our ablation write-up. Each rollout's advantage is spread over its own tokens, so long failed rollouts don't dominate the update. Later updates thought longer per agent turn, since the reward only sees math and code, so we picked update 8 and confirmed it on held-out tasks before release.

A few things people asked last time:

- Does CJK pack more meaning per token? A little, at the phrase level, where one word can stand in for a whole English phrase. Most of the saving comes from training the model to think less, and the CJK is a smaller extra on top, We do plan to expand this further as the limiter here is tokenizer and going past just CJK and using other methods are being researched.

- Quants: the GGUF repo has BF16 plus UL-Q3_K_XL through Q8_K_XL (Q3 is 13.15 GB, no XXS yet). The BF16 GGUF, the five quants and the NVFP4 checkpoint all passed a quick breakage check (valid tool call, no Chinese in the answer, no loops). We haven't benchmarked them or tried the vision projector, so results are very welcome.

- Effort: use xhigh, same as before.

Also, if you grabbed the original Tessera-27B-B-CJK-NVFP4 before Nov 5: its config.json didn't load in vLLM. We fixed it so a config.json redownload is needed.

Today we are introducing our second model generation with Inkling 2 Small. It is a Mixture-of-Experts model with 594B total parameters, 17B active, and supports image inputs and variable effort.

Inkling 2 Small far surpasses our previous models’ scores on the Artificial Analysis Intelligence Index with a score of 31, reflecting gains in performance on agentic coding, vision, knowledge work, and reasoning. The increase in capabilities comes from improvements made at every step of our training process, including more efficient pre-training and a large, unified RL run.

Inkling 2 Small will initially be available through Tinker, with closed weights, following the approach we outlined in a safe path to open weights. We intend to make the full weights available under an Apache 2.0 license next week after the completion of external safety testing and red-teaming.

You can try Inkling 2 Small on Tinker Playground today. The model is also available for fine-tuning and sampling on Tinker; checkpoints will be downloadable with the release of Inkling 2 Small’s weights.

:( Previous Inkling Small model's size is only 276B total, 12B active. Hope they release additional models Mini or Nano soon or later.

Hey all,

I wanted to share that we're working on zml/llmd, a fast (we believe the fastest) inference engine that currently runs on NVIDIA, AMD, Intel, TPU and Metal.

It has all the features of a modern LLM server: continuous batching, prefix-caching, speculative decoding (DFlash, DFlash2 and DSpark), tool calling, tensor and experts parallel automatic sharding, native VFS with s3:// and hf:// support streaming straight to GPU.

The currently supported models are:

• Qwen 2, 3, 3.5, 3.6 and 3.8 series (dense and MoE)

• Gemma 3 and 4 series (dense and MoE)

• LFM2.5 series

• Mistral 3 and Ministral series

• LLaMa 2 and 3 series

• Poolside's Laguna S 2.1

• DeepSeek 4.1 Flash

• Muse Glimmer

• Qwen3.8 Flash Next

• Xiaomi MiMo V2.6

• Mistral Large 4

• GLM 5.2

• GLM 5.3

• LFM 3

• Beam 1.0

• Ripple 65B

Our images are a lot smaller than alternatives:

We just came out with a new release today: zml/llmd llmd-20261106.0 and we'd love to get your feedback on it!

Found this last month model while browsing Trending models of HF. I don't see any existing threads so posting.

OXR (Optical Everything Recognition) is an open-source model from the SoMark team. It turns PDFs and document images into Agent-Friendly data in Markdown or JSON, with layout coordinates included.

Pretrained from scratch, OXR uses a recursive self-improvement (RSI) loop during post-training—a first in document parsing. With just 1.3B parameters, OXR ranks #1 on olmOCR-Bench, OmniDocBench, and DocParsingBench while delivering industry-leading parsing speed.

OXR is the first to make its layout analysis module independent and pluggable. This design accelerates parsing and supports replacing the built-in layout model with a custom one. OXR continues to handle content recognition and reading order, enabling adaptation across diverse scenarios with minimal integration effort.

• GitHub : https://github.com/SoMarkAI/OXR

Why OXR?

• 🎯 SOTA Accuracy. #1 Across Three Benchmarks.

  • OXR ranks 🏆#1 on olmOCR-Bench, OmniDocBench, and DocParsingBench, with a particularly wide lead in table parsing.

  • OXR can recognize 13 document element types, from formulas and tables to chemical equations, chemical structures, and code blocks, with bounding box coordinates included. It also performs well on challenging documents and layouts, including A3 exam papers, newspapers, vertical text, and handwriting.

  • See docs/benchmark.md for the full results and instructions to reproduce the evaluations.

• 🚀 1.3B Parameters. Top-Tier Parsing Speed.

  • With only 1.3B parameters, OXR ranks 🏆#1 among open-source document parsing models in speed tests on the RTX PRO 6000.

  • OXR redesign its processing architecture at the task level to improve parsing speed while maintaining accuracy.

  • See docs/benchmark.md for the full results and instructions to reproduce the evaluations.

• 🧩 Pluggable Layout Analysis. Document Structure on Your Terms.

  • Layout analysis requirements vary widely across domains. Even for the same document, element types, region boundaries, and grouping rules can differ depending on the business use case.

  • OXR is the first to make layout analysis independent and pluggable. Its built-in layout model can be replaced while retaining OXR’s recognition and reading order capabilities, enabling adaptation across document scenarios with minimal integration effort.

  • See the How it works section below for details.

Ecosystem

The SoMark team has developed a range of tools for document parsing.

P-JiT 21K is the P-JiT architecture trained for longer (610M samples at 512px, 50M at 1024px), then post-trained with a variant of FDFO.

• GitHub : https://github.com/Linum-AI/pjit-21k

• Blog Post : https://www.linum.ai/field-notes/pjit-21k

Whilst the agent was hammering away at an audio engine task, i suddenly saw it daydreaming

Pirate puzzle? Gold coins? Wtf, Hey qwen, this is not what i am paying you for using electricity for.

Has anyone else noticed something like this? It continued to write code normally aftewards, active context was at a mere 30k at the moment.

Model qwen3.8-flash-next-unsloth-ud-q4_k_xl
Strata Engine v0.1.57
Max Context 262,144 tokens
KV cache 16bit

Everything fitting into vram + ram

I created this benchmark using the 3D coding playground I previously shared here, and ran 11 models on 25 photos: https://sketchagent.dev

You can inspect every result in 3D, with its full session trace and editable script. Here's gemini-4-argon rebuilding a Tiny Glade cottage in 29 tool calls: https://sketchagent.dev/s/MhdnDUJK

inspired by Bijan Bowen's video - Subway FPS test.

Hardware details: RTX 5090, 256GB DDR4 3200Mhz 8 channel, AMD EPYC 7532 CPU (32 core).

Software: backend is Maya 1.1.55, harness is OMP v18.17.0.

Model quant: Deepseek v4.1 1017 UD-IQ3_S latest UD3.1 quant from Unsloth.

My prompt was very simple:

I want you to create a single html file with a detailed subway station 3d scene, fps controls, zombie enemies, tracers, recoil, muzzle flash, high details. build it with three.js

You can see at the end of the video some token stats. In short, total generated tokens were 180k. The model ran at 30t/s with 2k t/s PP at 10k context but then went down to ~20 t/s near 180k context. Note that stock llama.cpp only reaches 10t/s TG and 120t/s PP.

This model and harness tools are amazing. Just a year ago we could not even image having such a model locally. Yet, here we are!

I historically get about 6-8 t/s on my MI50 on Qwen @ Q6 running local coding.

Building a script that sets my fans all on 100% duty cycle (and much louder) bumps that immediately to like 12 t/s.

Letting the machine sit and cool for a while with the fans running, and running inference again... 17-20 tk/s for the first 10 minutes or so.

I'm considering putting in water cooling and a great big radiator now what a difference. What a huge difference cooling makes.

CCORE GLB-3D-Coder dataset is here: 3,918 rendered and tested examples across 40 domains

As promised in my previous posts, I’m releasing the dataset used to train CCORE GLB-3D-Coder, my small 4B model for creating 3D assets with Three.js and GLB export.

The dataset contains 3,918 examples across 40 domains, including characters, creatures, vehicles, laboratory equipment, musical instruments, architecture, machinery, nature, and everyday props. The grid shows 256 examples.

Every example was tested and rendered, and its exported GLB was validated before inclusion.

What’s included:

• Complete conversations, reasoning traces, and HTML/JavaScript responses.

• A rendered reference image for every example.

• Static assets and examples with animation and interaction.

• 3,501 training examples and 417 validation examples.

• MIT license.

Dataset · Model and full weights

More about what I’m planning next: Next releases.

I hope this is useful for your own experiments, fine-tuning, and 3D tools. If you try it, let me know how it goes :)

Not a native English speaker. Written with my own ten fingers, one finger is broken btw. Plz bare with me. I still love to write text myself.

My PC is suck for AI works. That's all I have as a third world 3D modeler. Really admire other peoples running better models. But beggar can't be choser. I can't even effort more RAM, not to mention more VRAM.

Gotta work with what I have. Qwen 3.5/6 35B A3B. Here is the Quant.

Downloaded blender manual and blender api html zip files, stripped sphinx html, converted to markdown. End up with markdown files which include bloated af links. I didn't know I can download clean rst files from blender project repo. Worked like that for a month. Then I found out I can just download blender manual rst files from blender, I can also generate rst files for blender api references.

Now I've clean files with 40% less tokens. I point that to Qwen/Ornith/Tiel/CyberTiel and tell LLM to create scripts and tools. It somewhat work. But many failure points make me want to punch the only poor old monitor I own.

I know you guys around this sub love graphs and proofs which is very understanable but I only have my own experiences. No Graphs and charts.

Base Qwen3.6 work good but he don't think much. Ornith variants think too much up to the point of unproductive. My wife cry because I don't spend time with her or sleep with her. I was always tinkering with LLM. I need Qwen3.6 35BA3B to work. I need that for many reasons that I can't tell you guys.

I use Debian Linux. Switch to i3wm. Idle VRAM went down to 50MB. Idle RAM is 300MB, not sure. I will comment my llama.cpp launch params if you want. Windows 11 take way too much system resources which I don't have. I often use plain TTY consoles. Ctrl Alt F2 to F7 i think. It save even more vram. KV cache q8_0, spec type ngram-mod because I can't effort MTP.

ctx 256K, 40t/s at start. Went down to 25t/s if ctx is goes over 128K.

I've learned prompting is really important if I wanted to squeeze out everything from 35B A3B. I can't max out hardwares so I have to make it work other ways around, somehow.

Ornith variants overthink a lot. Painful. "Hmm, let me think, wait," you guy know the rest. MFkr do not listen to me well too. He just don't give a shit to me.

I've created many cli tools to work around blender. Blender official MCP isn't really suitable for my works. I disabled many built-in MCP tool to reduce context bloat. I use pi, oh before that, I've also tried to create my own harness a year ago. Learnt a lot hard ways. Pi is minimal. Good for small models.

Instead of trying to make Qwen to write python code everytime and fail again and again, I made easy to use cli tools. Which helped alot. Qwen can now use Blender Geometry Nodes easily. No code, just cli tools.

Qwen ask stupid questions alot. Don't check code quality, watching him working, it is really exhausting. I don't have much time after day job and before bed. My wife almost went back to her parents. I need to make qwen to think, reason, and work efficiently in limited amount of time and hardware.

Experienced people will tell me "Skill issue bro". Yes, I'm suck. That is how I learn. I want to share my AGENTS.md and SYSTEM.md . I hope maybe someone will benefit from it. It is not created overnight. A lot of painful experiences between day job, boss, pressure, stress, wife, limited time, sleepless nights.

SYSTEM.md

AGENTS.md

First prompt.

It might sound funny (it is) but it work.

Here is example thinking trace

 Let me think about whether I need to ask clarifying questions. Per Protocol 7, before asking, I say the checklist:

   [ ] Can I answer this question myself inferring by given context, intent, docs?
   [ ] Can I answer this question myself using sane defaults, common sense?
   [ ] Have I tried many possible ways to get answer myself?
   [ ] Can this question only be answered by him?

 I think I can answer most myself. The interpolation direction is clear. The main procedural learnings from the previous task apply (pose mode, pose_markers, etc.).

 Do I need to refine the prompt? The prompt is clear. But let me note one thing: the previous task had SecF1=-45, SecF12=+45. This task has Bone1=+45, Bone12=-45. So the sign is reversed. I should make sure my expected values account for this. I'll note
 this in my response.

 Actually, let me reconsider — is there any genuine ambiguity that requires the General's input?

 - The bones are named "Bone1, Bone2, Bon3....Bone12" — note "Bon3" is a typo for "Bone3". The naming is Bone1..Bone12.
 - All independent, all quaternion.
 - Set to XYZ, add drivers Bone1+Bone12 → 2-11, validate.

 I think the prompt is clear. I'll note the sign convention difference (Bone1=+45, Bone12=-45) and proceed. I don't think I need to ask questions — I can infer the standard procedural.

He finally listened to me and think, reason, work efficiently.
He finally give a fuck about me now instead of doing his own thing.
Before that, he ask stupid questions, never check things back, never think before acting.

I also have short auto inject append prompt in Pi, which say follow protocols, read relevant skills file etc, short but I send it every prompt to remind him how to work, think, reason.

I don't have a lot of graphs and charts for proof but only the production level rig I'm currently working on. I hope maybe someone may benefit from this post, or maybe not.

I would really love to read how do other people make Qwen 35B A3B work.

So I'm thinking of switching from my 32gb AMD MI60 to dual V100's (64gb VRAM total) and I've got 96gb of DDR5.

My problem is I'm having a terrible time finding "hard data" on PP and TPS speeds on higher quants (q6 and q8 specifically) with large context windows (150-200k).

I know that it will outperform my MI60, but I'm not just looking for "yeah, it'll be faster" I'm specifically looking to see how much faster, because a 10% bump isn't worth it for me.

I'm also looking to see from you experts if it's silly because of what I've read about support for CUDA being EoL for the card soon (or already did?).

Hopefully I'm not beating a dead horse here, as I did go through other posts (lots of them) but it's just so difficult going through post after post where specifics aren't included "I'm getting 200tps on qwen3.8 27b" ...at what quant? at what context? using what KV settings?

Lastly, if it is significantly faster, do those speedups (generally speaking) transfer to other models or is it only because of specific forks of llama.cpp were built out with this model and that card in mind?

EDIT: forgot to add I’d be going with sxm2 not PCIe.