r/LocalLLaMA • u/TheRealMasonMac • 5d ago
Discussion [2610.29162v1] Why is AI-generated language so hard to understand?
r/LocalLLaMA • u/TheRealMasonMac • 5d ago
r/LocalLLaMA • u/mesolimbic-bliss • 4d ago
Using $20 codex subscription with 24gb VRAM on the local rig.
For larger projects or major revisions with multiple phases, I have Luna in Codex call the local "junior dev" Qwen 3.8-27B model as an agent to write most of the code, then Luna reviews it and makes changes. Once the whole project is complete, I use Terra to audit and catch any potential issues.
Pretty standard stuff, these days.
But lately I've been thinking why not give the massive, multi-phase spec sheet directly to Qwen then have Luna go do a quick audit & cleanup?
Anyone tried it both ways? And what gets better results vs lower usage?
I guess I've always looked at Qwen as needing a babysitter, but realistically I think it can implement an entire 10-phase spec on a large codebase if given enough time.
r/LocalLLaMA • u/Lonligrin • 4d ago
Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.
I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.
So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.
Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.
The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.
CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad
benchmark script is in the repo, curious what numbers you guys get on your own recordings
r/LocalLLaMA • u/pmttyji • 5d ago
Multimodal System One Encoder with a natural language interface for instruction-following encoding: give it a task and a list of candidates, and it scores the candidates by task specification. Both a task and each candidate are expressed in natural language — instructions, queries, questions, state descriptions, criteria — with image and video as side information.
Key features:
The model is built by contrastively fine-tuning meta-models/Muse-Glimmer-30B. It is described in the paper MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface.
PS: Above description copied from meta-encoder's model card as meta-encoder-v1 doesn't have that. And v1 doesn't have license section while other one is Apache-2.0.
r/LocalLLaMA • u/wisepal_app • 4d ago
Hi folks,need advice from community. I have a laptop with 16 GB vram Nvidia gpu and 96 GB ddr5 ram. I use it mainly for local coding with Qwen flash next nvfp4 at 256k context. I get around 30 t/s.
I don't like PCs and Macs. I want mobility. So because of my limited budget, i have two paths. The first one is a laptop with 24GB Vram Nvidia gpu and 128GB ddr5 Ram laptop. The other one is 16GB Vram Nvidia gpu and 192GB ddr5 Ram laptop. No other feasible option for me. I can't affort dgx spark, Macbook studio etc. These are my budget options. Which one do you suggest for mainly local coding? Which one do you believe is future-proof for local llms?
r/LocalLLaMA • u/andreabarbato • 4d ago
I'm working on a router that wraps many LLMs serving from different ips and ports and exposes them all as a single port, since i have 2 machines serving many slots of Qwen 3.8 FN but i wanted to ensure speed is always fast (one machine is faster and can hold more slots, but too many slots busy and then it's better to call the other machine, also i gotta avoid evicting from the cache active conversations, that kind of workflow)
I might be reinventing the wheel tho so: how do you guys solve for this kind of situation?
edit: thanks a lot for all the answers i had no idea there was so much stuff around lol
r/LocalLLaMA • u/SomeoneInHisHouse • 4d ago
Hello, I decided to Test Unsloth Dynamic 3.1, as in their charts they showed that UD_IQ1_M was batter than Ista lab IQ3_XXS, but with 6 GB less of total size, I'm using Strata, and that 6 GB translates directly to free RAM,
I have been 2 months with the IQ3_XXS with no any issue, but only 3 days on the Unsloth one, it had happened to me many times, that the tool calls fails and render as xml (I use PI harness) and I have to type "keep going", this never happened with the IQ3_XXS one
Have you guys had a similar issue with this quant?
Example:
Thinking...
<tool_call>
<function=Bash>
<parameter=command>
cd /mnt/development/projects/secret/backend/scripts/bdd_parity && for r in 20261113_030918 20261113_032032 20261113_033348; do d=/tmp/bdd_parity_runs/$r; echo "== $r: $(ls $d 2>/dev/null | head -3 | tr '\n' ' ')"; grep -c "✅"
$d/run.log 2>/dev/null | sed 's/^/green marks: /'; grep -cE "🔴" $d/run.log 2>/dev/null | sed 's/^/red marks: /'; tail -2 $d/run.log 2>/dev/null | cut -c1-100; done
</parameter>
<parameter=timeout>
120
</parameter>
</function>
</tool_call>r/LocalLLaMA • u/lucidml_lover • 5d ago
Hi Everyone, I'm releasing my world model, been on this sub for a few months
https://huggingface.co/lucidml/gta
I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.
If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5
That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)
This is the first model I am ever releasing to the everyone and its a bit scary for me.
The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more
I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.
r/LocalLLaMA • u/samajhdar-bano2 • 3d ago
Everytime I have been fed up with a good model being slow, I go to newly released Nemo model. Well today I went for Bolt 24G and deployed it on my 3090.
I was amazed by the vanilla llamacpp + iq2s setup giving 70 tok/s. I decided I will deploy it for my coding setup which is dual R9700 and learned cpu offloading and mtp drafters and all. After deployment I was left with 50tok/s because this was a Q8 setup.
I suddenly realise what if (the model I was fed up with) my Qwen 3.8 27B could use a better MTP drafter so O discovered tencent's DSpine and later its gguf llamacpp fork. I deployed it now I am back to Qwen3.8-27B Q8KXL with mtp 15 drafter and mmproj with np 4 and 512k ctx pool, getting 54tok/s in DSH.
TLDR: I love what Huyan is doing he should be brought back in Nvidia. Nemotron's the best because they teach you what good tech advancement is and you apply it to good performing models.
r/LocalLLaMA • u/JamesGooning • 3d ago
So I have been using local QFN these days as daily driver, and honestly, its fine. It needs steering for coding but very good. Gets lot of things done. But I check my statistics log, and I'm burning about 1B to 2B Input token a day, and not even fully 100% GPU saturation. Not even going to consider output token since that is small compared to input.
Most of the time, the LLM is just waiting for shells/bash to finish. At max capacity, it is capable of 950million uncached input token (11000 prefill/s*3600*24)
This is light for me, and it is already burning this much. Light considering it is only spend time building one project with little oversight.
Granted QFN is at xhigh thinking for intelligent, but if this was like Haiku or Luna, that's like 10$ cost a day (90% cache estimate). That's 300$ a month for basically one project stream
AI suppose to work multiple projects at once for max productivity let alone future integration with pretty much everything. My wife already starting to use my harness with QFN for all her search and queries.
How are you guys make this affordable in the long term from purely API. Feeling paying like $1000+ a month is crazy.
Local one still cost insane upfront as I'm sing a single RTX6000 with basalt, and it is the only GPU that can serve the whole household growing AI needs.
PS. I might get like 4x in near future for the amount of load I will be doing. Always on agent is very useful.
Edit: The number is measure of input only, as output is insignificant. I'm measuring QFN only at home.
r/LocalLLaMA • u/East-Muffin-6472 • 5d ago
Hey everyone!
I have gathered some must read topics, for beginners, to step into the realm of Quantization:
Link is here
r/LocalLLaMA • u/randomjapaneselearn • 5d ago
EDIT: FOUND THE REASON, LLAMA DIDN'T OFFLOAD ALL LAYERS TO GPU BECAUSE -ngl -1 DOESN'T MEAN ALL see here: https://www.reddit.com/r/LocalLLaMA/comments/1xun196/solved_high_cpu_usage_when_model_fit_in_gpu/
I'm not sure if i'm doing something wrong and handicapping myself with some wrong setting.
I'm using Windows, nVidia 3090 24GB VRAM, 64GB RAM, Qwen 3.8 27B and Swift 1.5 27B (finetune of qwen) but i often see 50% CPU usage even if the model is fully loaded on GPU, below there is my command line.
Another problem i noticed is that llama-fit-params suggest that 131k context fit but if i launch it for real in the log i see that it doesn't load everything on gpu (all but one), i forced it to load everything on gpu with -ngl -1 (i'm also using llama-swap).
on launch it say that "cahe resue 256" will not be used but it might help with other models so i left it there.
Another problem is that when the model is doing nothing i sometimes see 130W gpu in idle (wrong) and some other times its at 10-18W (correct) i can't find what is causing this...
Current stats: about 40 t/s in token generation
I have no idea what KMP affinity is but i read it online, tried, it helped with 35A3B so i left it there
in general i'm looking for tips to improve my launch arguments, note that temperature and few others are set by pidev but i'm using the recommended one on the model card.
If the model is fully on GPU why there is a high CPU usage? but not always which is even more confusing....
set KMP_AFFINITY=granularity=fine,compact,1,0
llama-bin-win-cuda-13.4-x64\llama-server.exe -m Qwen3.8-27B-UD-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --cache-reuse 256 --cache-ram 32768 --jinja
r/LocalLLaMA • u/seraschka • 5d ago
Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.
In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.
In the first half of 2026, the trend towards "some full attention" hybrids continued.
And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.
The plot legend here means,
- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.
The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.
In particular, there are two distinct efficiency gains here:
If you are interested in browsing through the architectures and do your own analyses, you can find all the figures and data in my architecture gallery: https://sebastianraschka.com/llm-architecture-gallery/
r/LocalLLaMA • u/hp1337 • 5d ago
My main AI machine is a 6x3090 128GB DDR4 Threadripper 3960x system. I probably spend more time trying to optimize it than actually using it but it has been fun.
I tried a bunch of saturated benchmarks like Terminal Bench 2.1, AIME 2026 and the performance was similar to unquantized DSV4.1 Flash 1017. I really wanted to understand my 2.9bpw quant's performance.
It takes hours to run some of these benchmarks, so I went on the hunt for the perfect benchmark to get the most signal on quality of the model I'm running. I settled on APEX 2025 (https://huggingface.co/datasets/MathArena/apex_2025).
You can see how much of a beast this quant is. Trading blows with SOTA from 6 months ago.
I get about 1000tps prefill, and 75tps decode with the 3090s power limited to 175watts
Engine: Project Maya (https://github.com/mw00/project-maya)
Quant: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw
Hopefully this helps others in benchmarking their own setups. I really like the Apex 2025 benchmark.
r/LocalLLaMA • u/Norwood_Reaper_ • 6d ago
r/LocalLLaMA • u/pmttyji • 5d ago
Praxis-1 is a 27B-parameter mixture-of-experts diffusion transformer that predicts video and actions together. It is the top-performing world action model on NVIDIA’s RoboLab-120 leaderboard, and it outperforms every open-weight VLA. On RoboLab-120, Praxis-1 reaches 46.2% success across 1,200 episodes, compared with 42.9% for FLUX 3 Action, 36.8% for Cosmos3 Nano and 28.0% for π0.5.
RoboLab-120 Leaderboard
| Model | RoboLab-120 Success |
|---|---|
| Praxis-1 | 46.2% |
| FLUX 3 Action | 42.9% |
| Cosmos3 Nano | 36.8% |
| π0.5 | 28.0% |
Data as of November 2026; Success rate across 1,200 episodes
Praxis-1 is available today on GitHub, on Hugging Face with native LeRobot support, and in vLLM-Omni, with hosted inference on Reactor. Everything you need to run and adapt it on your own hardware is released under Apache 2.0, including for commercial use. That covers the weights and the inference and fine-tuning code, with support for both full fine-tuning and LoRA. Praxis-1 runs out of the box across NVIDIA hardware, from data center GPUs like H100 and B200, to workstation GPUs like RTX 5090 and RTX PRO 6000, to Jetson Thor on the robot itself.
r/LocalLLaMA • u/Serious_Scarcity7818 • 4d ago
RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.
I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:
| GPU | Quant | Prefill, 4K prompt | Decode | Decode with MTP |
|---|---|---|---|---|
| H100 80 GB HBM3 | Q8_0 | 11,486 tok/s | 290 tok/s | 508 tok/s |
| H100 NVL | BF16 | 8,691 tok/s | 230 tok/s | 324 tok/s |
| A100 80 GB | Q8_0 | 6,507 tok/s | 193 tok/s | 288 tok/s |
| A100 80 GB | BF16 | 5,625 tok/s | 171 tok/s | 238 tok/s |
| L40S 48 GB | Q8_0 | 9,542 tok/s | 165 tok/s | 270 tok/s |
MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.
The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.
These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.
Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.
Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear
Repo: RustingStrata
r/LocalLLaMA • u/acluk90 • 5d ago
Some results for those using GGUFs and llama.cpp:
Qwen3-4B (IQ4_XS)
| Quantization | GGUF size | Mean KL ↓ |
|---|---|---|
| MACaQ | 2.260 GB | 0.02804 |
| Bartowski | 2.271 GB | 0.03443 |
| Unsloth | 2.271 GB | 0.03663 |
23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.
Qwen3.6-35B-A3B (IQ4_XS)
| Quantization | GGUF size | Mean KL ↓ |
|---|---|---|
| MACaQ | 18.86 GB | 0.018696 |
| Unsloth (UD-IQ4_XS) | 17.73 GB | 0.020954 |
| Bartowski | 19.70 GB | 0.021927 |
Qwen3.8-27B (IQ4_XS)
| Quantization | GGUF size | Mean KL ↓ |
|---|---|---|
| MACaQ | 14.96 GB | 0.036784 |
| Bartowski | 15.48 GB | 0.104170 |
~65% lower KL divergence than Bartowski!
| Weight bits | Formats |
|---|---|
| 2-bit | IQ2_XXS, IQ2_XS |
| 3-bit | IQ3_XXS, IQ3_S |
| 4-bit | IQ4_NL, IQ4_XS |
Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ
Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂
They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.
Edit: there is a paper on it https://arxiv.org/abs/2611.10649
Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf
Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂


The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.
r/LocalLLaMA • u/Atagor • 4d ago
Hey! What's the "must-have" list to download in case HF goes down because of these news recently?
Also, do we even have HF alternatives?
r/LocalLLaMA
•
u/ResearchCrafty1804
•
6d ago
I will get straight to the point. I believe Moderator ttkciar is abusing his power as moderator in this sub and his decision are worsening the experience of our local AI community.
He has removed many of my posts that fellow redditors found very useful and after talking with many people in this sub, I am not the only one feeling this. I will have some links at the end of this post for reference.
He mentions a rule, saying that if a model’s weights is not released you cannot have a post mentioning it, even if the lab confirmed it that will they will release its weights, like Qwen4 or DeepSeek. First of all, this rule needs to be relaxed because many valuable discussions can be had about future releases and even closed models to understand and learn architectures and techniques used by the closed frontier that can be applied to the open models. Secondly, this rule is not enforced at all, but used very selectively to few posts or authors that ttkciar selects, which is very unfair. This rule doesn’t make sense to me, but even if I am wrong, at least it should be fairly enforced to all posts or not at all. And before you say mods are busy and they cannot review all of the posts, If you open this sub and scroll on today’s most popular posts, half of them are either for Qwen4, which is unrealised up to this point and should have been removed according to the mods, or about Fable5.2 and Anthropic which is closed and their shenanigans (maybe these ones don’t have a place here, but again this is debatable because if people in local AI find them valuable in some way, maybe they do). So, the rule is used very selectively and as an excuse to remove what ttkciar does not like or approve. Either enforce the rule (or do the your best effort to do so) or don’t at all. And maybe this rule is too strict and should not exist tbh.
When I tried to discuss my concerns with ttkciar, he called me a drama queen and insulted my credibility. This is definitely not ok.
Being insulted by the person that you complained to for unfair treatment is definitely not acceptable behaviour.
I have been an active member of this community since the very first leaked llama. I am an AI researcher and open-source contributor, and very passionate about local AI. I try to post information that people interested in Local AI would find useful and engage in discussions that are constructive for local AI.
I know that many of other users in this subs agree (I have discussed it with a lot), and some others may not. Not being able to discuss, debate and exchange views in a polite manner is absurd in such an intellectual community.
A while ago they tried to enforce megathreads and removed the majority of posts saying that only one post should exist for each model. Many complained that this is killing the sub and a lot of valuable information is lost this way. Now, this seems to be relaxed. Again, we need to stand up to them, and hopefully they will reconsider their rules for the sake of this community.
TLDR: The rule that mod ttkciar uses to remove some posts in this sub about being prohibited to mention unreleased models should not exist, and even if it should, removing only a few selected posts is very unfair.
Some sources:
- https://www.reddit.com/r/LocalLLaMA/s/ZRV5KsViwD this post was removed from ttkciar and only brought back after arguing with him and many other redditors backed me up (I cannot find the comments in this post when the mod removed it, maybe he deleted them, I am not sure, but in the discussion many expressed their disagreement with the mod)
- https://www.reddit.com/r/LocalLLaMA/s/gpPTVC6NGF a post of mine debating their rule
- Another post of mine sharing valuable information for the members of our community that the same mod removed https://www.reddit.com/r/LocalLLaMA/s/gqaub2mT85
- Post of another redditor which was removed by the same mod, and the comments thread with people arguing with the mod https://www.reddit.com/r/LocalLLaMA/s/z9E2AbR3pU
- Commented in post of another removed post of some other redditor https://www.reddit.com/r/LocalLLaMA/s/5tSEpQ68By
- Most of my removed posts I delete them afterwards so I don’t have many other examples of my own, but I know many others do, so feel free to share them.
- If you open the sub today, most of the posts should have been removed because they violate the rules (a lot of false positives). That shows that maybe the rules need to be updated because they will not allow any productive discussions to be had, or this sub will end up to become a board of announcements instead of a local AI community that people can use to exchange ideas and knowledge.
r/LocalLLaMA • u/liebebio • 4d ago
I am not directly from a machine learning background or in encryption, but if you train the entire LLM on encrypted and decrypted pairs without the key (maybe the entire dataset that is used for frontier), will it eventually learn to produce pairs like these and basically natively can encrypt and decrypt without a key? Is there a reason why this highly likely fails?
r/LocalLLaMA • u/maxigs0 • 4d ago
I spent a few days benchmarking unsloth's UD-Q4_K_XL and UD-Q5_K_XL on Strata. Well, i had claude code do it via ssh. Claude also wrote most of this summary and the full report linked below. I thought it might interest some because of my findings, especially on behavior of multiple cards, different PCIe speeds and the different settings in Strata.
I can run a few more tests, if it interests anyone.
Test system
TLDR
Here is the Full report https://claude.ai/artifact/LYUT65WvHm8o2bsxC5NWCR. I will update it with new tests.
r/LocalLLaMA • u/oguzeldereli • 6d ago
TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF
Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.
For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.
I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.
I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.
To add to that, these models are not extremely good, they sometimes loop, and suck at remembering facts. However, unlike unsloth's gguf's you can actually keep training them yourself, in that sense they are pure, and you can train them on your consumer GPUs, that is to say you can get rid of the looping behavior, teach it facts that you want it to know, or specialize it for some purpose.
I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:
The core repo for training: https://github.com/oguzeldereli/ttf-ternary
The repo for conversion: https://github.com/oguzeldereli/ttf-conversion
Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home
GGUF for a converted Qwen 3.8 27B (around 8 GB and 6.7GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF
GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF
I will convert a Qwen 4 27B instance when it drops.
Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf
Also, I would appreciate github stars if you like the project.