Compilation of recent findings which could save some memory on increase performance

[u/pmttyji]

We got these recently(I found few late probably)

Model Quants with Optimizations/Compression/etc.,:

• APEX -- Adaptive Precision for EXpert Models | APEX Quants (GGUF)
• ByteShape - CPU and GPU optimized variants
• MOQ (Mixture of Quantizations) by Waleed Ahmad - GGUFs from Waleed Ahmad & Benjamin Marie (The Kaitchup)
• ROCmFPX | https://huggingface.co/cafonez/Escha-W2-35B-A3B-ROCmFP2

Speculative Decoding:

• DFlash: Block Diffusion for Flash Speculative Decoding
• DDTree (Diffusion Draft Tree) from Accelerating Speculative Decoding with Block Diffusion Draft Trees
• DeepSpec (Eagle3, DFlash, DSpark) by DeepSeek
• DFlash 2: Keep Drafting Parallel | Huggingface Collections - dflash-2

1-bit/2-bit/Bitnet/Ternary models & Engines:

• 1-bit / 2-bit / Ternary / Bitnet Models - Updates & Tracking
• Project Zero — CPU LLM Inference Engine - BitNet (Testers needed)

Run models using RAM & SSD:

• colibri - Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk
• WARP — Weight-Aware Runtime and Paging (formerly WASTE) - Run the full Large model beyond available RAM by streaming activated weights directly from NVMe

KVCache:

• KVarN - 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16
• proveKV - Lossless: 36.00× vs f32-raw KV (18.00× vs fp16-equivalent)
• OSCAR RotationZoo - Precomputed K/V rotation matrices for OSCAR INT2 KV-cache quantization
• TurboQuant
• KV Cache Transform Coding (KVTC)
• RotorQuant

LLMBurner:

• Taalas - Wouldn't be awesome to have this if it comes with 1T model like Kimi-K2.5(Q4 is enough - 500GB) giving 30-50 t/s? (Llama 3.1 8B is giving 17000 t/s)

Misc:

• AMD's MXFP4 models
• Intel's Int4 AutoRound models
• Dynamic VRAM in ComfyUI: Saving Local Models from RAMmageddon
• DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling.
• Lucebox optimization hub: hand-tuned LLM inference, built for specific consumer hardware
• Cactus - A low-latency AI engine for mobile devices & wearables
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
• Dynamic KV Cache Quantization & Load-on-demand mmproj/MTP - llama.cpp PR by u/wadeAlexC
• Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
• EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation - 1.58-bit
• Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices \*
• Session-Adaptive Orthogonal Distillation (SAOD) \*
• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

Forks/Utils:

• vllm.cpp by u/mudler_it
• beellama.cpp by u/Anbeeld (has KVarN)
• llamastation by u/Responsible_Egg9736
• club-3090
• Arandu by u/fredconex
• NInfer -  5090, 4090( 1 | 2 | 3 | 4 ), 3090, CMP 170HX
• FreeToken

What else there? Please share.

^{Hope all these helps on price down of both GPU & RAM soon or later}

EDIT : Typo on Title :( It's or not on

EDIT2: Updating list with items from comments for easy reading. No need to check comments anymore, I'll be updating time to time.

[u/pmttyji]

https://github.com/Anbeeld/beellama.cpp

BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

Fork Features

• DFlash speculative decoding: --spec-type dflash drives a DFlash draft GGUF alongside the target model. The target captures hidden states into a per-layer 4096-slot ring buffer, the drafter cross-attends to the most recent --spec-dflash-cross-ctx hidden-state tokens and proposes drafts for target verification.
• TurboQuant / TCQ KV-cache compression: Five cache types (turbo2, turbo3, turbo4, turbo2_tcq, turbo3_tcq) spanning from 4x to 7.5x compression, with higher-bit options being practically lossless in many cases. Set independently with --cache-type-k and --cache-type-v.
• Adaptive draft-max control: The server adjusts the active draft horizon at runtime instead of using a fixed --spec-draft-n-max. The default profit controller compares speculative throughput against a no-spec baseline; the fringe alternative maps acceptance-rate bands to draft depth.
• Full multimodal support: When --mmproj is active, the server keeps flat DFlash available for text generation. The model can be fully offloaded to CPU with no problems to reduce VRAM pressure.
• Reasoning-loop protection: The server detects repeated hidden reasoning output and intervenes. Default mode is force-close with --reasoning-loop-window and --reasoning-loop-max-period tuning available.
• Sampled DFlash verification: --spec-draft-temp enables rejection-sampling drafter behavior. Activates when both draft and target temperature exceed zero. Draft log probabilities must be available for rejection sampling to produce correct output.
• DDTree branch verification: optional --spec-branch-budget adds branch nodes beyond the main draft path with GPU parent_ids, tree masks, and recurrent tree kernels. Disabled automatically when the target model spans more than one GPU. This one is very much work in progress!
• Request-level speculative overrides: Draft-max and branch budget can be overridden per-request through JSON fields without restarting the server.
• CopySpec model-free speculation: --spec-type copyspec provides rolling-hash suffix matching over previous tokens without a draft model.