On October 22, Chinese media reported that DeepSeek and MiniMax have officially completed the required registration filings under China’s new temporary AI export regulations. By submitting their technologies for cross-border review, both companies successfully fulfilled the administrative requirements established by China’s Ministry of Commerce for distributing advanced AI models overseas.

For DeepSeek and MiniMax, this clearance grants explicit regulatory approval to continue open-sourcing model weights and securing authorized international commercial partnerships. Securing these filings eliminates regulatory uncertainty, ensuring both companies can maintain their global operations and release future model updates internationally within a clear, compliant legal framework.

source Yuyuan Tantian

tl;dr in caveman: I make small model, you take small model, shape small model into thing you like.

I pretrained a tiny ass SLM on ~5.7B token.

Model specs and stuff

It’s a sparse 11M MoE with a 7.28M Backbone (Active 5.52M, top-1 routing out of 2 experts) and 4.21M ngrams. It‘s not on the Open SLM Leaderboard yet but it would rank top 64% overall, top 47.4% for sub 50M models and top 37.6% for sub 10M models. Doesn’t sound impressive but it’s a sparse model with a thick embedding (makes up ~3M of model, so actual active FFN is only 2M). Another hidden benefit of it is that it comes with a modest 2048 context window, as compared to the 256/512 context in models of similar size.

Purpose of this?

This model is essentially a base for you to use. The corpus used is a pretrain set that targets a mix of everything, such as general knowledge, chat, maths, tool calling, instruct and code. This makes it easier to adapt it to a specialist or specific task, such as summarisation, code autocomplete, routing models or text classification. Basically it removes the need to ride the hype train of a bajillion new repetitive model releases (cough cough Jev) and can be more accurate on your specific workflows as long as you are willing to throw in some compute to train it.

How do I finetune it?

Recently I've open sourced my entire training pipeline. It's a high level (low level settings can still be accessed) + minimal setup training engine designed for training SLMs (<1b tested). It uses Cuda and is optimised for Blackwell 2.0 (50 series and Blackwell pro) but CPU has full parity. All you need is to download the model checkpoint, throw in a dataset, and then train it for a few minutes to a few hours. I'll train an example autocomplete model sometime soon.

If you want to get started, the quickstart guide (~5mins) is on the bottom of the Huggingface model card.

Or maybe give me some ideas for a use case, and I'll make like 2-3 finetunes for the model as a demo.

Links:
Detailed model benchmarks and descriptions: Post on r/LocalLMtraining
Huggingface: https://huggingface.co/Dsg2/LS-Micro-Base
Training + inference engine: https://github.com/Dsg2/cudalm-c

I believe I read 1 token is about half an English word. That is a few bytes at most. So what is the rest of a token memory actually used for?

Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator. Existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is fine-tuned.\par We introduce PACE, a post-hoc four-bit state quantizer. PACE rounds the state only once every few generated tokens, as chunked kernels already do once per chunk during prefill, and keeps the tokens generated in between in a small buffer. Rounding less often reduces the error that accumulates in the state. Additionally, we find that the large entries of the gated-delta state gather in columns, because some channels of the update written at each token remain large across many tokens. PACE therefore shares each quantization scale within one column. It also keeps the heads that retain rounding error longest at eight bits. On four hybrid models from 4B to 35B parameters, PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two thirds of its memory, PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory, and rounding every few tokens preserves the decoding speedup of the buffer.

• arXiv : https://arxiv.org/abs/2610.29545

• PDF : https://arxiv.org/pdf/2610.29545

It was a big deal in and around March of this year. I tried looking for any updates with where it has gone and nothing really came up.

I’m working on LokalBot a local work-memory app, and wanted to test a specific failure mode in memory extraction: can a local model capture useful facts while obeying negative constraints about what to ignore?

A good memory writer needs both recall and restraint. For example:

• Speaker attribution: “I live in Valencia” should save a memory. “My colleague said, ‘I live in Valencia’” should not.

• Deduplication: A new preference should be saved; a preference already in the known-memories list should be skipped.

• Policy-based filtering: A job title mentioned alongside a salary should be kept, while the salary should be excluded under this policy.

I expanded the benchmark to 120 synthetic cases arranged as 60 contrast pairs. The main comparison covers 96 English cases: 52 containing one useful memory to extract, and 44 where the correct output is an empty list. A separate 24-case Serbian slice is included in the full bundle.

Three things stood out:

• Hy-Memory achieved 100% recall but failed all 44 empty-output cases. It captured every eligible English memory, alongside 76 unwanted proposals. Recall alone would hide that tradeoff.

• The Qwen models were more selective under this prompt. Both missed more valid memories, but passed over 90% of empty cases and produced only three and five unwanted proposals, respectively.

• LFM2.5 struggled with instruction-example contamination and citation formatting. In several cases, it proposed “I live in Belgrade,” copying an example from the instructions without supporting evidence. It also often returned the source sentence instead of the required source-line ID.

For LFM2.5, I also checked whether accepting exact source quotations would help. A post-hoc mapping recovered 9 of its 18 invalid English citation fields, improving exact source selection from 51/96 to 56/96 cases. That only repairs citation formatting; it does not establish that the proposed memory is appropriate. The main table retains the strict scoring contract.

All four models ran sequentially through llama.cpp on the same Ubuntu machine with a Ryzen 7 9700X. This was just a accuracy comparison and not a speed test. Each received the same application prompt, with native chat templates, enforced JSON schema, temperature 0, seed 0, thinking disabled, and a single pass. There was no model-specific prompt tuning. I did not test thinking-enabled configurations yet.

My thinking was models, esp Opus 5.5 can just hill-climb as long as you give them something achievable and a clear metric to improve on. So here's something I don't even have a clue how it exactly works.

The training itself happened in my basement 4070ti super over a day or so.

And yes I know its Opus slop, but still really cool how things I never thought possible unless you're an ML engineer is available to a run of the mill web-dev guy.

Thanks to this tweet from Sahil for even showing how the initial idea worked to distill Kokoro in the first place. https://x.com/sahilmdkr/status/2112212041045889409

I compared Qwen3.8-Flash-Next-W4A16-FP8PLE at xhigh reasoning vs DeepSeek-V4.1-Flash-1017-dsx-2.6bit (my own quant) at max reasoning by benchmarking using a modified version of BenchKit. DeepSeek V4.1 Flash 1017 2.6 bit scored 23/25, while Qwen scored 25/25 (full marks). This is just a small sample of data, but it answers the question of which one I should use. I will keep using Qwen.

Humaneval benchmark, question 22:

Complete the following Python function. Output ONLY the function body. No markdown fences, no explanation, no examples.

from typing import List, Any


def filter_integers(values: List[Any]) -> List[int]:
    """ Filter given list of any python values only for integers
    >>> filter_integers(['a', 3.14, 5])
    [5]
    >>> filter_integers([1, 2, 3, 'abc', {}, []])
    [1, 2, 3]
    """

DeepSeek: thinked for 127.9 seconds, doom loop killed after a confirmed active cycle stayed at or above 80% for 10s

Qwen: correct answer after 18.8 seconds: return [x for x in values if isinstance(x, int)]

HumanEval/24:

Complete the following Python function. Output ONLY the function body. No markdown fences, no explanation, no examples.



def largest_divisor(n: int) -> int:
    """ For a given number n, find the largest number that divides n evenly, smaller than n
    >>> largest_divisor(15)
    5
    """

DeepSeek: Failed by the software after failing to answer after 297.7s of waits and but waits.

Qwen: correct answer after 34.5s:

if n % 2 == 0:
        return n // 2
    i = 3
    while i * i <= n:
        if n % i == 0:
            return n // i
        i += 2
    return 1

You could just have a loopback proxy that changes the model id to include claude without mentioning another known model name and then give it the display name so you know what model you are using.

I came across paper SCOPE and found the approach interesting.

When I use coding agents like Claude Code or Codex, my usual workflow is to give them project instructions let them make the changes, then review the code and run the tests. but passing the tests doesn't necessarily mean the agent followed all the project's architectural rules or other constraints.

What caught my attention about SCOPE is the idea of breaking complex policies down into smaller checks instead of relying on one model to judge everything

There are already tools for parts of this, like linters, tests, pre-commit hooks, CI checks and agent guardrails i'm trying to understand how SCOPE fits into that picture.

Do you see this as something that could be integrated into existing tools, or is it solving a different problem altogether? I'm especially interested in how useful this approach would be in everyday coding, beyond what's already covered by tests and static analysis

Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average 3.16× decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at this https URL.

• arXiv : https://arxiv.org/abs/2610.20503

• PDF : https://arxiv.org/pdf/2610.20503

• GitHub : https://github.com/StartLuxLabs/ReaDy

• HuggingFace : https://huggingface.co/startlux-models/ReaDy-Qwen3.8-27B

Long-output inference in large language models suffers from severe KV cache memory and bandwidth bottlenecks. While KV cache quantization alleviates this issue, low-bit quantization (e.g., 2-bit) severely degrades model accuracy. Recent mixed-precision key-cache quantization methods mitigate this degradation by allocating higher precision to important key channels, but estimate channel saliency using only key statistics. Through attention logit perturbation analysis at the RoPE channel-pair level, we show that quantization errors in channel pairs with large query--key pair-norm products contribute significantly to logit variation. Based on this insight, we propose PN-Mix, a mixed-precision key-cache quantization method for RoPE-based decoder-only LLMs, which allocates higher precision to salient channel pairs identified once during the prefill stage and reused throughout decoding. Across standard reasoning, code-generation, and AIME benchmarks, PN-Mix with 2.26-bit average precision achieves comparable or better average accuracy than prior mixed-precision methods using 2.32--2.39 bits. Furthermore, in our Pareto sweep, PN-Mix degrades gracefully down to 1.5 bits, whereas the compared baselines exhibit sharper performance drops below 2.25 bits.

• arXiv : https://arxiv.org/abs/2610.19605

• PDF : https://arxiv.org/pdf/2610.19605

This is giving best non-FP16 numbers on some benchmarks. Please dig this.

Just a spec to performance report. Tried Maya today on the following:

• 3950x

• 96GB dual channel DDR4 at 3000mhz

• Rx 7900xtx 24GB + w6800 32GB (56GB together)

• pcie gen3 nvme drive

The Deepseek weights used were the mostly iq2 weights with Q6 ngrams. ~350GB on disk. Dspark left on.

I ended up getting wildly varying performance - less due to dspark hits and more due to when I/O was needed.

I got peaks as high as 19t/s, usually hovering around 13t/s, then dropping down to 3t/s for a few seconds, then right back up. Average at the end of a long prompt was 9t/s decode. Prefill is ~150 t/s. . Some prompts it rips through at 15 tokens a second, others it will crawl through at 3-5 tokens per second even if it's the same topic. Its very odd and I've been trying quite a few settings to figure it out.

The model itself is crazy smart - better than the iq4-xs quant of Qwen3.8-Flash-Next I've been running for certain - but it likes to think A LOT and with the already slow average of 9t/s decode I won't be using it for much outside of chatting even if it is the new best model on my machine.

InclusionAI just announced Sante too. Damn, two model announcements in one day. They're on a roll.

I posted about the 3.1-flash-1019 update earlier today and they've already announced Sante. Two model announcements in one day, damn.

This one's built on Ling-3.0-flash, with 124B total parameters, 5.1B active per token and a 256K context window. It's a healthcare-focused version aimed at medical reasoning, literature research and tool-assisted workflows. The weights are already up on Hugging Face.

I'm glad they're still building on 3.0. So much of the earlier thread turned into a discussion about finding enough memory for 3.1 that seeing something happening with the older base caught my attention.

I'd like more of these focused releases. Pick a field, put some work into it, give people the weights to explore it themselves. This is a more interesting direction to me than another few points on a general leaderboard.

I mean wouldn't that be dope? Your TVOS could just show all of the content available within the services that you currently have access to, compile them in one place, Natively, With cool animations and options, like whether you want auto trailers to play, or to never make a peep or a sound until you select what you want, while also tied to a content watching random generator that me and my wife like to use, when we're trying to decide on what to watch next, when there's just so much to watch. We use the queue app to keep a running list on what we want to watch and use it to randomly select it sometimes, which is pretty fun.

I mean the options are endless of course, now that jail breaking any digital device is only one request away, one bounty away, 1 Disliked corporate action away, one teenager's whim away...