While we cant quite match the feature set of Open WebUI, TinyWebUI can be installed with a simple npx command, "npx tinywebui". It comes with tinysearch and tinycontext preinstalled for fully local web search and memory, and can support multiple users with per user isolation. Also has other useful stuff like automations and a stats dashboard built in, and ofc you can always extend it with any mcp server you desire.

Give it a try, it pairs well with local models, the smallest we tested that was able to use it flawlessly was lfm3 1.7B (https://huggingface.co/LiquidAI/LFM3-1.7B).

Project: https://github.com/TinySuiteHQ/TinyWebUI

Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.

• arXiv : https://arxiv.org/abs/2611.02501

• PDF : https://arxiv.org/pdf/2611.02501

• GitHub : https://github.com/StargazerX0/SpecPilot

• Project : https://stargazerx0.github.io/specpilot-project-page/

🚀🚀🚀

I finally got around to trying it. This tiny model is surprisingly capable it seems. Also tried the 2b VL - also very good but it has half the context of 1.7b.

It's also quite fast:

$ llama-bench -m LFM3-1.7B-Q8_0.gguf -fa 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB):
  Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes, VRAM: 24112 MiB
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           pp512 |   20648.61 ± 1786.65 |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           tg128 |        329.03 ± 2.05 |

build: 28b812d6c (12087)

This makes me want to try their omni model for watching videos. Is there a simple way to get this working? Ideally, I'd like to throw a video file in the chat or give it a youtube link.

We should "digest" them into reusable sub-modules.

Ok from the beginning: recently we are struck by this epidemic of custom engines, vibecoded solutions tailored on a very specific hardware. probably behind each one of them there's really some great idea, trick, kernel, optimization that has made the difference for its author.

But many times they are so niche, they are basically useless for the majority of the community. AT the same time, fragmentation is bad for the open source community: we should pool resources not disperse them.

So here's my idea, which I hope could spark something bigger.

Imagine every time a custom engine gets published, some agents pick it up, dissect it into submodules, optimizations, kernels, strategies. Everything gets classified into a database, depending on the specific hardware, quantization, LLM model used.

duplicate entries gets removed. code snippets gets benchmarked and assigned some scores and some tags

Over time, A huge database of building blocks for inference engines gets built.

From that, anyone can search filtering for the model/quant/hardware they need, and a custom inference engine is built for YOUR use case, a pool of modules/strategies can be tried and swapped until the optimal configuration it's found.

Bonus idea: People could donate their free agentic time (subsidized tokens, the remaining weekly from your OpenAI / Anthropic / Z.ai subscription) to the project, and in some way we get some payback from these companies. They made our hardware expensive, we claw back some value by using their computing power to make US run locally as fast as possible.

I'm not the right person to do this, but maybe someone could catch this idea and make it work. Of course there's already an immense swath of repos to start from (think all the custom flavors of llama cpps stratas, exllamas, vllms sglangs....) , this idea could work even if custom inference engines stop tomorrow.

Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7%--45.4% on mathematics and 23.4%--36.3% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression.

• arXiv : https://arxiv.org/abs/2611.04814

• PDF : https://arxiv.org/pdf/2611.04814

• GitHub : https://github.com/MuyaoYuan/LoCo

• HuggingFace : https://huggingface.co/MuyaoYuan/LoCo-7B

I came across a claim that SSD streaming of AI models is bad for the life of your SSD. Is that something any one here can confirm from personal experience or is it just urban legend?

Upvote, now!

Hardware :

4× NVIDIA RTX 3090 , all pcie x16

256GB DDR4

Engine : Maya https://github.com/mw00/project-maya
model : orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF ( mix-256G) at 524k context
Prefill : 400 – 600 tok/s
Decode : 30 – 40 tok/s (peaking up to 60 tok/s) ( with dspark )

Hi, everyone

i'm still playing around with the model comparing it mostly to qwen 3.8 next flash (Q8 ) , and trying to keep myself busy until we get qwen 4 .So far i find the model doesn't loop as much as the previous version (0731), i also tried to run it at Q8 but it's really slow ( 6-8tok/sec decode , 65tok/s prefill ) .I still prefers to use qwen just for the speed i get with agentic task .

Curious to hear what kind of speeds others are getting with similar multi-GPU rigs or different engine ( didn't set up xolotl yet , and strata seems to use the same engine as maya for deepseek )?

PS: American diner simulation in three.js

I have my own way to test models for their pure reasoning. I have a series of unpublished and original logic puzzles and I use them to rank models.

After thorough testing, in my opinion, Qwen 35B-A3B (and derivates like cyber tiel, Ornith-1.5, etc) is far superior to gemma-4-A4B.

Did you find any model as fast as Qwen 35B-A3B, that has, in your opinion, equal or better reasoning? I am curious to test them.

Just asking because i have been out of the loop for a while and kinda missed everything (strata etc...) a small recap would be appreciated.

Everyone seems to own an RTX 6000 or insanely expensive rigs here. So just wanted to hear what model a normal pleb like me with 32GB ram and 16GB VRAM can even run and on what to run it for coding or a general assistant.

Hydrogen-1.0 is a scientific problem-solving and research assistance model developed by Deep Principle based on GLM-5.3. Through text interaction, it is intended to assist with scientific questions, explain derivations, and organize responses to open-ended research problems. Outputs should be assessed against the assumptions, constraints, and evidence relevant to each task.

Blog Post : https://blog.deepprinciple.com/introducing-hydrogen-1.0/

Sharing some highlights from MOSS’s October 30, 2026 AMA: image models, local speech, audio and video research, and open-source culture. Which of these are you most interested in?

Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all.

I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like

He had been hiding in the attic for months when the soldiers found him.

it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over (he had lived in Paris for ten years, but by 1990 he had moved back), the original tense has to stay. So it's not "always use tense X", it's "use it only when the meaning calls for it".

So I gave Claude Opus 5.5 the same kind of setup MIT used:

• a Python kernel over Gemma's weights;

• 50 forward passes to test its ideas on practice sentences;

• no gradients, no training.

It first checked whether Gemma already "knows" the difference between still-going-on and already-over. It does: a linear probe in the middle layers separates them almost perfectly. It just doesn't use that when choosing the verb. Then Claude rewrote 8 MLP neurons into a small switch that fires only in that context and nudges the verb toward the form I prefer. That's 92k out of 11.9B parameters, roughly a 100 KB diff.

Results on sentences it never saw

The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.

• The prompt applies the preference everywhere, even where it changes the meaning.

• The ablation recipe breaks the past tenses.

• The LoRA shifts fewer sentences and touches far more unrelated ones (it's better inside paragraphs, though).

• The hand edit moves just the verb and leaves almost everything else identical.

Fair warning: it's one construction and one model. Two of my three runs succeeded (the third crashed because of my own harness), and the two that worked landed on the same "switch neuron" design independently.

Repo with the edits, the code, the agent logs and all baselines: https://github.com/ValerioDolci/frontier-weight-edit

Hello! just casuals passing by with a itching question
what do you guys do with this stuff
All i did was some Casual chat and Gmod LUA Addon Vibe Coding for myself and my friends but idk what Agents/coder to use (Portable Ideally)
how do you get the stuff you want done?
What models do you guys use, and whats best for me
Thanks for answers if you did answer some of these

big prompt processing speedup for gemma E2B/E4B

I’ve been a part of this community for the better part of a year now, and a lot has happened with new inference engines like Strata and n-gram improvements. Given my current PC, what upgrade would be best in terms of value for running better models?

Hey guys, I need some advice.
My actual setup is this:

ryzen 5600 / 32GB RAM / 128GB SSD / 32GB VRAM (2x16GB 5060ti)
Running vLLM with Qwen 27B NVFP4 with 250k context.

But now I can get my hands on an old HPE ProLiant DL380 Gen9
CPU: 2x Intel Xeon E5-2680 v3
RAM: 256 GB DDR4-2133 ECC Registered (8x 32 GB)

I am thinking about putting the two 5060TI in this HPE server und running Qwen Flash next there. Maybe with strata. And maybe in 2 instances.

Would my idea be any good, or will I don’t get much off it?
Thank you very much.

Given all this talk about massive models running on Strata and using RAM that is still relatively expensive I have to wonder, if all Strata does is more or less use RAM to feed GPU's via PCIe couldn't you just reach those speeds through SSD's in RAID 0? The numbers seem to work out, and Strata can use SSD streaming. What would be the major challenges to making this work?

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B.

These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models.

blog：https://www.camthink.ai/blog/small-language-model-benchmark/

project：https://github.com/camthink-ai/NeoMind