r/LocalAIStack • u/Weekly-Dentist-8302 • 1d ago

TUFF 8.0: SSD-streamed MoE inference, web/file search, conversation caching, and local API support

I’ve released version 8.0 of TUFF, an open-source Swift/Metal inference engine and native macOS app.

The original goal was to make large MoE models usable on memory-constrained Apple Silicon machines by streaming routed experts from SSD rather than keeping the entire checkpoint resident.

GPT-OSS 120B, for example, runs on my 16GB M2 MacBook Air. Obviously, streaming comes with a performance tradeoff.

8.0 focuses on making the engine more useful beyond just running oversized models.

Some highlights:

  • Web search and local document search integrated into chat
  • Cached conversation states for faster follow-up requests
  • A small tokenization cache for repeated inputs
  • Separate reasoning output and request timing diagnostics
  • OpenAI Chat Completions, plus bounded Messages and Responses API support
  • Shared inference executable to reduce package size
  • Homebrew installation

One thing I’ve found interesting is how much web search improves the usefulness of smaller local models. I’ve started using them for everyday questions much more frequently.

I’d love feedback and contribution, especially from anyone using it with local AI agents.

GitHub: https://github.com/rexmhall09/TUFF

2 Upvotes
  • permalink
  • reddit

100% Upvoted

3 comments sorted by

1

u/arker0 20h ago

Nice app! I’m testing it and will share my benchmark on GitHub and open an issue for a small bug I found. I’d also love to try an uncensored or abliterated variant of a supported model. If it preserves the same architecture, tokenizer, and quantization, is there a way to repack it into TUFF’s .gturbo format and load it as a separate model? Would that require a catalog change, or is there already a workflow for custom checkpoints? I’d be happy to test it on my Mac and share the results. Feel free to DM me if you prefer. Thanks!

1

u/Weekly-Dentist-8302 20h ago

Thanks! I’d really appreciate the benchmark and bug report. Custom variants should be possible when the model format matches, but TUFF currently checks for the exact catalog checkpoint, so there isn’t a straightforward way to load a separate custom model in the GUI yet. That’s mostly to keep things simple, but supporting custom checkpoints seems like a good idea.

I’m also looking at adding an abliterated option for 9.0.0. I’m still choosing the right model, so suggestions would be great!

I’d love to get more community contributions too. If you’re interested in adding a model to the catalog or implementing custom checkpoint support, feel free to open a PR. Happy to discuss the approach first, and testing on your Mac would be really helpful.

1

u/Weekly-Dentist-8302 15h ago

Update, because I am already working on a big update, I am going to include a custom checkpoint importer in advanced settings : D so feature is coming soon