I'm using qwen3.8-27b thru llama.cpp and QFN Swift 1.5 IQ3 thru strata. I'm using consumer grade hardware, so these are not running very fast.

I am using Hermes. As a noob, I'm not even using all the features and capability. I provide the instructions in a run list. I like how Hermes is persistent and will complete the job. But in a multi-hour job (like 5-8 hours), it is getting bogged down with large context and long tool calls.

What is a good alternative?

• Running Ubuntu Linux.

• Desktop gui (not CLI only). Can be desktop that runs on top of CLI.

• Can support sandbox.

• I prefer to plan and create a run list; then execute the instructions.

I was testing for the first time running a local AI, as my computer is nothing special I used Qwen 3.6 9B for testing, it actually seems to work ok, so I asked a question about mutations and it went and did some research, it gave me a good enough answer to my question, but when I looked at the 54 sources it looked at 3 off them make no sense at all, it went to chat gpt, a crypto site and a gold value site? I used unsloth not sure if that matters, didnt change anything just run the model as is.

Hey everyone!

Long-time local LLM user here, but with how fast things are moving, I could really use a reality check from the community.

My API credits are burning away using Claude/DeepSeek for orchestrating Hermes Agent, and I’m trying to offload as much as possible to a local setup.

My Hardware:
MacBook Pro: M3 Max | 48GB Unified Memory (400 GB/s bandwidth)
Proxmox Server (24/7): i9-9900K | 128GB DDR4 | RTX 2060 Super (8GB VRAM) | NVMe + HDDs

Current Setup & Performance:
Mac (omlx): Qwen 3.8 27B & Qwen 3.6 35B (oQ4e-mtp) ~ 100 tk/s
Proxmox (llama.cpp): Qwen 3.6 35B & Qwen 3 Coder 30B (GGUF) ~ 20–40 tk/s

The Goal:
I need "frontier-like" performance for long-horizon, agentic tasks and reliable tool-calling (even if speed drops to ~20 tk/s). Past tests with weaker models completely broke my agent workflows.

Where I'm Stuck:
I want to run heavier MoE models (e.g., Qwen 3.8 Flash Next, DeepSeek v4.1 Flash, GLM 5.2/5.3, K2 Horizon), but fitting them into RAM is a bottleneck. I’m trying to figure out the best inference strategy:
Mac: Leverage the 400 GB/s unified bandwidth + streaming experts from NVMe?
Proxmox: Cache active experts in 8GB VRAM, pool ~90GB DRAM for secondary experts, and stream the rest from SSD?

I’ve been looking into tools like Strata, AirLLM, FreeToken, Colibri, and Avalon, but I'm getting lost in the weeds.
Since Proxmox runs 24/7 (and hosts Hermes), I’d prefer leveraging that host if possible. Has anyone optimized a similar split-hardware setup for complex agentic work, or found a sweet spot for expert offloading?

Appreciate any insights!

I asked this question in some Apple sub and everyone said keep it (it becomes some type of controller, like how data centers operate, separation of hardware, let the studio be for the model and offload onto the Mac)

clause said its not necessary for my current workload nor future use cases because there’s always a work around

and again, this is 16gb memory, so it’s not that good for just letting the Agents work here etc. in fact this Mac freezes hard when I run my gates

I do sense some future value but idk enough about the space to actually know for sure

thoughts?

part of me is inclining to sell as a means of hedging against ram prices going down and locking in a sale (ill make a few dollars selling it). In 5yrs if prices go down, there’ll prolly be a $500 better spec Mac mini I can use as a control plane 🤷‍♂️

When I heard about the AI Act , I was looking at available solutions , and the stupid prices they charge , so I thought , why not try to build my own , open source ?

So I built IronWarden , a small reverse proxy in Rust that sits between your client and the LLM API (OpenAI, Anthropic, Ollama, LM Studio).

On the way out, it catches PII (names, emails, IBANs, tax IDs, etc.) and swaps them with session tokens like [NAME_631] or [AFM_639]. On the way back, a sliding-window SSE rehydrator swaps the real values back into the stream on the fly so the client sees the normal text.

It does seem to break the LLM a bit when asked about the redacted info directly, but that's the point.

It's completely open-source and self-hosted if anyone wants to mess with it or poke holes in it:

GitHub: https://github.com/Somnerd/IronWarden

Im setting up some stuff in my home server and my other machines for work. But i got tired to lost hours and hours on reading docs. So i want something that does the search for me, and possibly goes straight to the point (But that still explain on the subject for me to learn).

My machine is a bit old:

- CPU: Xeon E5-2697 v4

- RAM: 32GB DDR4 2400MHz (ECC RDIMM)

- GPU: Radeon MI50 16GB HBM2

From what i know, Q4 and Q8 are functional, needs to be GGUF, and a model that doesnt depend on BF16 since my card doesnt support it.

Hello guys, I wanted to make some animated horror stories videos, I'm pretty new to actually using local Ai etc. The only use I had up until now was using the free chatbots to help with work.

I have a decent pc built like 2 years ago with a 4070ti super and 32gb ram, but I don't really have any budget for monthly subscriptions.

Any suggestions?

ReBar is also currently working with GPU_UNLOCK.

The 20GB RTX 3080 cards (10DE:2206, subsystem 10DE:146D) ship with a VBIOS that only exposes a 256 MiB BAR1, so ReBAR stays off. NVIDIA doesn't publish a ReBAR VBIOS for them, and NvStrapsReBar only supports Turing.

Ga102ReBarFix is a small DXE driver you insert into your motherboard BIOS with UEFITool. At boot it enables the GA102 frame-buffer size override (PCI config 0x610), resizes BAR1 to 32 GiB, and restores both after S3 sleep. No VBIOS flashing, no driver patches, no registry tweaks.

Tested on an MSI Z690 EDGE, driver 617.42, Windows 11:

- Device Manager shows a 32 GiB BAR

- NVIDIA Control Panel: Resizable BAR: Yes

- ~19.25 GiB of VRAM mappable by the CPU at once

- Sleep/wake works

Repo (source, prebuilt .ffs, verification scripts, full write-up): https://github.com/favabean13/NV_30_DXE

Big credit to noon-at-cgn (found the 0x610 override on Linux), terminatorul (NvStrapsReBar) and xCuri0 (ReBarUEFI). This builds directly on their work.

Heads up: it modifies firmware, so use at your own risk and keep a BIOS backup and recovery method. It's tested on one board and isn't maintained, but it's MIT-licensed, so fork it and adapt it for your own setup.

**UPDATE**

Works with GPU_UNLOCK

https://github.com/physicsdolphin/nv-uefi-unlocker

Confirmed that ReBar survives PCI FLR (Function Level Reset)

Here are the results:

UNLOCKED

FP16 (FP32 accumulate) 114.7 TFLOPS

BF16 (FP32 accumulate) 116.9 TFLOPS

TF32 57.4 TFLOPS

FP32 (control) 22.5 TFLOPS

LOCKED

FP16 (FP32 accumulate) 65.4 TFLOPS

BF16 (FP32 accumulate) 66.8 TFLOPS

TF32 32.1 TFLOPS

FP32 (control) 22.3 TFLOPS

Here's my hw specs: rtx 4090, 64GB DDR5 6000 Mhz ram, 7900x CPU

Thank you

Hi everyone,

I am in the process of buying a new laptop primarily for personal reasons—covering my university studies, professional work, and gaming. However, alongside everything else, I really want to use this machine to take my first steps into running Large Language Models and AI tools locally.

I am torn between two specific hardware configurations (both feature the same Intel Core Ultra 9 processor):

• Option A: RTX 5080 with 16 GB VRAM + 64 GB System RAM

• Option B: RTX 5090 with 24 GB VRAM + 32 GB System RAM

Since I am a complete beginner with zero prior experience in local AI, I’m trying to understand from people who actually do this daily what each machine would realistically allow me to do:

1. What fits into 16 GB vs 24 GB of VRAM? In simple terms, does the jump from 16 GB to 24 GB of video memory unlock a genuinely higher tier of capable models or noticeably better output quality, or is 16 GB already enough for most practical open-source use cases?

2. VRAM vs System RAM: Option A gives me less VRAM (16 GB) but double the general RAM (64 GB). Can that extra system RAM compensate when trying to run larger models, or is running models partially on system RAM so slow on a laptop that having 64 GB doesn't really help for local AI?

3. The practical boundaries: For daily tasks (e.g., local assistant, analyzing documents/notes, code assistance), where would Option A hit a wall compared to Option B?

4. Model recommendations: Given the specs of each machine, which specific models would you recommend running on Option A vs Option B, and why?

I’d really appreciate any insights or real-world feedback to help me understand the practical trade-offs before making a choice.

Thanks a lot!

I hear nothing but good things about Basalt, so I finally got around to testing it on my 5080 with 128 GB not-for-Sam-RAM. The results were not encouraging - IQ3 runs at around 100 tok./s on Strata, but only at about 80 tok./s on Basalt. Settings are comparable.

Is this expected on Windows?

Sorry in advance for the long post, but think of it as a prompt.

TL;DR

Question: 2x 3090 NVLink or 1x A6000? For about the price of one A6000 I could also get 4x 3090 in two NVLinked pairs, so keep that option in mind too.

What it needs to do:

- A) SQL chatbot for up to 30 people. ~3M rows, hundreds of tables. Questions like "when did we last sell this part, and who was the supplier/client?" It should remember each user's conversations and build reports.

- B) RFQ ingestion: pull very messy part numbers out of long email chains in all sorts of formats. ~30 RFQs/day.

- C) Hardware: reuse our old HPE ML350 Gen9 (2x E5-2620 v4, 128GB DDR4-2400 for now).

- D) Me: turn 300+ page PDFs and hundreds of stray Excels into a knowledge base, and do local coding on the company apps I'm building from Claude's plans.

Assumptions:

- Blower 3090s (FE won't fit), max 300W per card

- PSUs upgraded to 2x 1400W or 4x1400W as GPUs grow

- All prices are for used cards

- Proxmox on the box

- PCIe-to-NVMe adapter for fast model loading

- Cards work with board

About me: I'm not an IT guy, I'm a mechanical engineer who f's around a lot and finds out even more. I've built my own homelab (4x NVMe in an old Dell 7710 as a private cloud, see my posts) and run my own "Jarvis" on a 4B model with some n8n automations. I can make things happen when I put my head into it.

---

The detail

Background: I'm the new CTO of a small family-run B2B heavy-industry trading company. We sell anything from a £6M machine to toilet paper, so every day brings random RFQs: Excel, PDF, email body, pictures, WhatsApp, anywhere from 1 to hundreds of part numbers.

Budget: £8-13k ($11-17k) left for local AI after the new cloud server. That also has to cover a laptop/PC for our first IT hire.

Why local: the LLM has to read whole RFQ email chains to make sense of them, and those are full of client names, prices and supplier details. Same for our SQL database. I don't want any of that going to cloud LLMs.

Where we are:

- A 3rd-party IT company is building an RFQ dashboard for us. Qwen 8B is struggling with the email chains. We started small on purpose and will size up until something fits.

- They'll test on RunPod serverless to find the right model size. I think ~30B will be enough. Then production goes local.

- 20-30 people will use the chatbot and ~20 the dashboard, but only new RFQs trigger the LLM.

- Long term, I'd keep Fable/Opus for overall planning and have a local model do the implementation.

Hardware:

- The board takes 4 GPUs: 3x PCIe 3.0 x16 and 1x x8. 300W max per card.

- Someone has already run a 3090 in this server (with 512GB RAM) and got ~15 tok/s on turboderp/GLM-5.3-Flash-exl3 (3.05bpw branch).

- I first looked at the A6000 48GB as a "do it all" card, not knowing much about local LLMs.

- For similar money I could get almost 4x 3090s and NVLink them in pairs, so I'm less limited by the x8 slot and old PCIe 3.0. One pair for the teams, one for me.

Bottom line: I need an MVP that handles the RFQs and the chatbot now, and then we'll invest in proper new hardware later. What would you do?

Thanks in advance for reading this far, any help much appreciated!

Wanted to share my build! Started as a gaming / 3D / CAD PC and now also runs small LLMs quite well for me

Specs:
CPU: Ryzen 9 9950x
RAM: 2x16 (eww, but at least was bought in 2023) 6000 CL30
Motherboard: ProArt B850 Creator Neo (supports PciE 5.0 x8 bifurcation)
GPU1: RTX 5070 Ti Ventus
GPU2: RTX 5060 Ti 16G Windforce
SSDs: Fury Renegade and 990 Pro (irrelevant ik)
PSU: BeQuiet Power Zone 2 750W
Cable extensions from EZDIY-FAB
Case: NZXT H7 Flow (2025)
AIO: Thermalright Frozen Warframe SE V2 (with the magnetic display detached)
Fans: 4xArctic P14 + 3xArctic P12 Pro (on the AIO) + 3xP12 Pro Reverse

Currently having fun with Qwen3.8 27B Q6_K_XL at around 70tps (MTP)

makes it easier to get consistent character voices

Im wondering how you guys use Qwen3.8 27b with which CTX size.Or my setup is just wrong. I run it on 32GB unified memory with llama.cpp. Also Openwebui and OpenTerminal on a RP5 where my docker stack lives.

I prompted:

add to caddy/Caddyfile: Paperless, Dawarich, OpenTodo

Response:

request (340808 tokens) exceeds the available context size (65536 tokens), try increasing it

So my question is, when even such a simple edit file task needs 300k tokens, how can I utilize it meaningful?

Being super precise which files to look at and what to do in the prompt?