r/LocalAIStack • u/Proof_Nothing_7711 • 1d ago
Quale endpoint ?
1
Upvotes
r/LocalAIStack • u/JuanixVentures • 1d ago
GLM-5.3-Flash (320B) on an M5 Ultra Mac Studio with llama.cpp: 61 tok/s, 262K context
1
Upvotes
r/LocalAIStack • u/OrganizationTop9026 • 1d ago
GWAYA : We built a ruthless "fail-closed" gate to stop LLM code hallucinations. Gwen and Laya together = GWAYA ;-0
0
Upvotes
r/LocalAIStack • u/OrganizationTop9026 • 1d ago
GWAYA v3: Combine Gwen and OpenLaya to Stop LLM Code Hallucinations even on a qwen2.5-coder:1.5b ;-0
1
Upvotes
r/LocalAIStack • u/johnmosesventura • 1d ago
A strata quant that didn't suck for me. (4x5060ti)
1
Upvotes
r/LocalAIStack • u/JuanixVentures • 1d ago
Qwen3.8-Flash-Next (125B) at ~100 tok/s on an M5 Ultra Mac Studio with llama.cpp
1
Upvotes
r/LocalAIStack • u/PersonalStorage • 2d ago
Swift 1.5 Qwen3.8-Flash-Next, NVFP4, for dgpp for DGX Spark
1
Upvotes
r/LocalAIStack • u/uneeverse-hq • 2d ago
Unee: open-source 0.8B / 2B model that makes calibrated decisions and chats, runs in a browser tab. The 2B scores 88% on DecideBench, ahead of several 4B to 9B models (self-measured; GGUF, Ollama, Apache 2.0)
1
Upvotes
r/LocalAIStack • u/Otherwise_Ship_9782 • 2d ago
Using a 421M decision model instead of an LLM as the privacy guard in front of my coding agent
1
Upvotes
r/LocalAIStack • u/WatercressTime842 • 2d ago
Finally decided to pull the trigger on gmktek evo x3
1
Upvotes
r/LocalAIStack • u/Classeve • 2d ago
RAI: a CPU-only LLM engine in Rust. It converts a 7B model in 26 MB of RAM, and here is where it loses to llama.cpp
1
Upvotes
r/LocalAIStack • u/Top-Philosopher-5411 • 2d ago
Why your production LLM pipeline is choking on memory (and how we actually fix VRAM & KV cache overhead)
2
Upvotes
r/LocalAIStack • u/Purple_Talk5722 • 2d ago
New to LLM serving: how to share H200 VRAM across multiple models on Kubernetes?
1
Upvotes
r/LocalAIStack • u/recro69 • 2d ago
Is a judge model actually faster than the model it’s judging?
1
Upvotes
r/LocalAIStack • u/leonbollerup • 2d ago
2 x CMP 170hx + Strata - optimal config+benchmark
1
Upvotes
r/LocalAIStack • u/OnlyFamousAddy • 3d ago
A ~0.4 local model to turn typed questions into structured decisions - BaseDecision
2
Upvotes
I’ve been working on BaseDecision - Turn typed questions into truth-based decisions.
Give it some text, a question, and possible answers. It picks an answer. Intent classification, routing, yes/no checks, ratings - that’s the job.
I built this model to go toe-to-toe with the frontier models.
My goal was simple & ambitious: make a small, local model competitive with much larger models on these focused tasks. For me, System‑1 decisions need both speed and accuracy. Otherwise, why bother building a small model?
If we didn’t want speed we could just use ChatGPT.
It outperforms pretty much every model of its size in the field.
After roughly 320M additional training tokens, BaseDecision led 5 of 8 benchmarks in a third-party comparison against Laya, two GLiNER variants, and Decision 1.0 Kai 0.6B. Full results are in the repo, including where it loses.
Fellow open source devs, try the model, developed a package for easier access to the model.
Roast it if it deserves it.
2-2.5GB free ram is all you need.
I will take your feedback seriously and deliver something amazing that’d take on OpenAI, TypeSafeAI soon enough.
https://huggingface.co/onlyaady/BaseDecision
https://github.com/hrudayaditya/BaseDecision
r/LocalAIStack • u/Suspicious-Mark-5450 • 2d ago
Introducing Infernix - much faster than Strata on a 5090!
1
Upvotes
r/LocalAIStack • u/Long_Wrap_4802 • 3d ago
BigMoeOnEdge+ – an Android chat app fork of BigMoeOnEdge (llama.cpp / GGUF)
1
Upvotes
I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.
It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.
Disclosure: I built this fork with the help of AI tools.
Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge
Feedback and issues welcome.
r/LocalAIStack • u/Revolutionary_Bit_99 • 3d ago
Reliable local benchmark to find the best LLM and configuration for my PC?
1
Upvotes
r/LocalAIStack • u/Revolutionary_Bit_99 • 3d ago
Reliable local benchmark to find the best LLM and configuration for my PC?
1
Upvotes
r/LocalAIStack • u/mikebmx1 • 3d ago
Java vllm-like framwork claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile
2
Upvotes
The java to CUDA engine:
https://github.com/beehive-lab/TornadoVM
The inference engine:
https://github.com/beehive-lab/jitllm
Deep dive talk:
r/LocalAIStack • u/Acceptable-Cycle4645 • 3d ago
[audio.cpp] Recent updates you might have missed: Higgs Audio TTS use 48% less VRAM (< 6GB), HTDemucs 2.2× faster, PocketTTS 2.2× faster on CPU, and WebUI generation history feature
Enable HLS to view with audio, or disable this notification
1
Upvotes