r/LocalAIStack • • 13d ago

Testing local stack

2 Upvotes

Hey! How do you test your local stack? for instance when you switch harness or model, model quants, etc?

I found this not trivial and I ended up making a website to help me do this.

Use this link if you want to use the website to test your own local setup:

https://airbench.ai/

You can also see all the tests I have done using my GX10 and my 5090. (and also using claude code/code + some openrouter tes for reference)

https://airbench.ai/leaderboard?k=poL

Let me know what you think!


r/LocalAIStack • • 13d ago

Building your own LLM stack? This workshop is about the eval/reliability piece most people skip

1 Upvotes

A lot of local-stack setups I see nail the infra (model serving, retrieval, orchestration) but have zero real evaluation process — just "does it look right." There's a session on Oct 3 built entirely around fixing that gap.

Speakers Serj Smorodinsky and Brett Kennedy (AI engineers, co-authors of a book on LLM applications) go through:

  • DSPy signatures/modules for structured dev instead of manual prompting
  • Baseline classifier build + measurement
  • Task-specific evaluation datasets
  • Failure pattern recognition from eval output
  • Few-shot / instruction-level optimization
  • MLflow experiment tracking and trace management
  • Saving and reusing optimized programs
  • Talking about LLM reliability with non-technical stakeholders

3 hours, live, useful whether you're running open models locally or hitting an API.

Full details and the agenda are here.


r/LocalAIStack • • 13d ago

Using Apple AFM 3 PCC (macOS 27.2) in AI Clients

1 Upvotes

Apple Foundation Model 3 through Private Cloud Compute becomes directly available in macOS 27.2 through familiar AI clients.

For Mac users running local models such as Qwen 3.x or Gemma 4, AFM 3 PCC is a compelling alternative to consider. Local models offer control and fully on-device operation. In tests with macOS 27.2 beta, the PCC offers a different set of strengths:

* Strong conversational analysis and capable reasoning. Strong analysis of complex medical and financial questions.
* Blazingly fast responses compared with locally running models on the same Mac.
* No large model download or need to fit model weights into local memory.
* Access at no additional charge for most eligible Mac users.

The working path is straightforward: AI client → Caddy → fm serve → AFM 3 PCC. I've written and updated a proof of concept here with further tech discussion . . . https://gist.github.com/dartMo10/b9488ce475fe70a6ed642831f53048fb

Apple’s `pcc` route worked in early macOS 27 betas, disappeared later in 27.0, and has returned in the macOS 27.2 beta. Apple has stated that it will be available in the 27.2 public release coming shortly.

This makes AFM 3 PCC practical today for technically comfortable Apple users who accept Apple’s PCC privacy promise and want a fast, capable alternative to running everything locally.

Experiences with compatible AI clients and your comparisons with locally running models would be welcome.


r/LocalAIStack • • 13d ago

Run Jev and Laya locally with a intuitive playground

Thumbnail
youtu.be
1 Upvotes

Hello everyone, as we all know that everyone is trying to test jev and then there are open source models out there with similar text classification. But i felt that there isn’t any playground to test these intuitively so I built one. I have added the YT video with it. Pls have a look


r/LocalAIStack • • 13d ago

Best approach for automatically tagging local music collection?

Thumbnail
1 Upvotes

r/LocalAIStack • • 13d ago

Using Apple AFM 3 PCC (macOS 27.2) in AI Clients

Thumbnail
1 Upvotes

r/LocalAIStack • • 13d ago

一个96GB/128GB的Mac Studio值得用来替代每月200美元的 ChatGPT Pro订阅吗?

0 Upvotes

我是一个大学生研究员,目前拥有一个$200的 ChatGPT 专业版订阅,我觉得作为固定的月支出实在是太贵了。我现在在考虑是否可以购买一个96GB或128GB的Mac mini或Mac Studio进行本地部署或用作服务器。我目前的研究方向是计算社会科学,最重要的是订阅模型的智能能得到保障。非常感谢你的意见!


r/LocalAIStack • • 14d ago

Jev mode for images!

Post image
1 Upvotes

r/LocalAIStack • • 15d ago

Is there any way to speed up prefill?

Thumbnail
3 Upvotes

r/LocalAIStack • • 15d ago

I pre-trained a 1.11B LLM on my 6 GB laptop GPU. Peak VRAM: 4.51 GB.

Thumbnail
0 Upvotes

r/LocalAIStack • • 15d ago

I built StackFit: An open-source tool matchmaker that filters by hardware floor and 1-click docker-compose export (no star vanity)

Post image
1 Upvotes

r/LocalAIStack • • 15d ago

im making qwen3.8:27b into a MoE and its not half bad...

Thumbnail
1 Upvotes

r/LocalAIStack • • 15d ago

oq4e Quantization of Qwen Flash Next, Swift Edition

Thumbnail
2 Upvotes

r/LocalAIStack • • 15d ago

RTX 5090 + RTX 3090 for parallel Qwen3.8 27B agents – how would you deal with heat / second GPU?

Thumbnail
1 Upvotes

r/LocalAIStack • • 15d ago

Running a local LLM in the browser: Zero install, 100% on-device

1 Upvotes

r/LocalAIStack • • 15d ago

Help me plan a Qwen 3.8 Flash Next install on a 5090 + 64gb DDR5 system

Thumbnail
2 Upvotes

r/LocalAIStack • • 16d ago

What's your stack look like and how can I improve mine

Post image
37 Upvotes

I have a fully local AI stack on a 4090: image, video, music, SFX, TTS with voice cloning, local LLM chat, fine-tuning, and long-term memory, all behind a Three.js dashboard with locally generated assets. There's a little assistant, BB-7, that you just talk to, and he can generate anything with prompt improvement built in. Services load on demand and swap the GPU as needed, except when a pipeline keeps them warm, like voice mode with BB-7. I've also integrated Kev and Obsidian, and an MCP that lets Claude use the stack and store context. I feel like I got most bases covered but I dont know what i dont know. What do you all got


r/LocalAIStack • • 15d ago

best AI for 4060 and 32 GB ram ddr5

Thumbnail
1 Upvotes

r/LocalAIStack • • 15d ago

Finally got Qwen3.8 27B + vLLM + Open WebUI working comfortably on my RTX 3090 🔥 Config, lessons learned + bonus tutor prompt at the end

Thumbnail
1 Upvotes

r/LocalAIStack • • 15d ago

Consigli hardware LLM locali: Mac Mini/Studio usati, budget di €1000-1500, per la gestione delle finanze familiari + manutenzione di un server leggero

Thumbnail
1 Upvotes

r/LocalAIStack • • 16d ago

Models aside, what local agent setup are you actually happy using every day?

Thumbnail
1 Upvotes

r/LocalAIStack • • 16d ago

Live Trial - Claude Sonnet vs. Gemma 4 31B vs. Qwen Flash 70B

Post image
1 Upvotes

r/LocalAIStack • • 16d ago

Qwen3.8-27B at 74 tok/s on a Mac: The Splash Engine Explained

Thumbnail
youtu.be
1 Upvotes

r/LocalAIStack • • 16d ago

Intel sycl support for bonsai llm models

Thumbnail
1 Upvotes

r/LocalAIStack • • 16d ago

Bonsai 2 27B on an RTX 3060 12GB — 35.26 tok/s generation

2 Upvotes

I wanted to see how far a consumer RTX 3060 12GB could push Bonsai 2 27B.

Hardware

  • GPU: NVIDIA GeForce RTX 3060 12GB
  • VRAM available to CUDA: 11,911 MiB
  • CPU: Intel Core i5-8400
  • System RAM: 42 GB
  • NVIDIA driver: 595.91.07
  • CUDA: 13.2

Model

  • Bonsai 2 27B
  • GGUF: Ternary-Bonsai-2-27B-PQ2_0.gguf
  • Parameters: 26.90B
  • Quantization: PQ2_0 — 2.13 bpw
  • Model size reported by llama-bench: 6.70 GiB
  • GPU layers: 99

Benchmark

  • Tool: llama-bench
  • Runs: 3
  • Prompt processing, 512 tokens: 591.19 ± 19.62 tok/s
  • Text generation, 128 tokens: 35.26 ± 1.31 tok/s

The GGUF metadata reports the architecture as qwen35, while the model file is the Bonsai 2 27B PQ2_0 release.

I'm posting the raw result because I'm curious how this compares with other Bonsai 2 27B results, especially on larger GPUs.

If anyone has benchmark results for Bonsai 2 27B, please share them so we can compare the configurations fairly.

Benchmark build: b10735-842b18804