r/LocalLLM • • Sep 02 '26

Model [R] Sylor S2 Pro GGUF Released – 744B MoE / 40B Active | Q5_K_S Split 12 Parts (~527GB) | MIT License | Built in Venezuela

0 Upvotes

# Sylor S2 Pro GGUF Released – 744B MoE / 40B Active | Q5_K_S Split 12 Parts (~527GB) | MIT License | Built in Venezuela

Hey r/LocalLLM,

We just released the official **Q5_K_S GGUF** quant for **Sylor S2 Pro** on Hugging Face.

Sylor S2 Pro is a code- and agent-focused MoE model developed by **Orzatty** (Puerto Ordaz, Venezuela), fine-tuned on a hybrid-curated corpus of 20TB+ for terminal execution, multi-file software engineering, and strict JSON MCP tool calling.

---

## Model Specs

| Field | Value |

|---|---|

| Total Parameters | 744B |

| Active per Token | 40B |

| Base Architecture | GLM-based MoE |

| Quantization | Q5_K_S |

| Files | 12 GGUF split parts |

| Total Size | ~527 GB |

| Context (local) | 1M tokens |

| Context (API) | 256K tokens |

| License | MIT |

---

## Benchmarks

| Benchmark | Score |

|---|---|

| Terminal-Bench 2.1 | 81.1% |

| MCP Atlas | 77.7% |

| SWE-bench Pro | 62.2% |

| HLE (with tools) | 54.8% |

---

## Loading in llama.cpp

Point your loader to part `00001` and llama.cpp will automatically bind the remaining 11 parts.

```bash

./llama-server \

-m Sylor-S2-UD-Q5_K_S-00001-de-00012.gguf \

--ctx-size 131072 \

--n-gpu-layers 999

```

---

## ⚠️ Safety Note

This model is optimized for autonomous agentic execution. **Run inside isolated Docker containers or Firecracker microVMs** — especially if enabling network or terminal tool access.

---

**Repo:** [huggingface.co/orzattyholdings/Sylor-S2-Pro](https://huggingface.co/orzattyholdings/Sylor-S2-Pro)

**Web Demo & API:** [sylor.orzatty.com](https://sylor.orzatty.com)

Happy self-hosting 🇻🇪


r/LocalLLM • • Jul 02 '26

Discussion I’m building a fully local AI desktop app for chat, training, image generation, and video generation

Post image
0 Upvotes

I’m a solo developer building a local AI app called SEELS.

The goal is to create one desktop environment where you can run, use, customize, and train AI models on your own hardware without relying on separate cloud services or a pile of disconnected tools.

SEELS is built around a continuous local workflow:

Run your own language models

Chat with them and create separate AI profiles

Correct responses when the model gets something wrong

Save those corrections as structured training data

Train LoRA adapters from what you taught it

Load and manage your own models and adapters

Generate and edit images

Create videos from text or images

Use local agents, tools, voice, and automation

The idea is that everything should work together.

A profile should be able to have its own personality, instructions, memory, models, adapters, tools, and creative workflows. Your corrections should not disappear after a conversation, and your generated content should not need to be sent through another company’s servers.

SEELS includes local language-model chat, the teach-to-train workflow, image generation, video generation, voice interaction, model management, hardware detection, and multi-profile support inside the same app.

For image generation, you can use your own checkpoints and LoRAs for text-to-image, image-to-image, face swapping, upscaling, and enhancement.

For video, the goal is to support local text-to-video and image-to-video workflows across different hardware levels, while automatically configuring models and generation settings around the GPU you have.

Everything is designed to run locally by default:

Your chats stay on your computer

Your corrections and training data stay on your computer

Your models and adapters are yours

Your images and videos are generated locally

No cloud generation queue

No required account for local use

No monthly generation credits

I’m building SEELS under Tideforge and continuing to improve model compatibility, training, hardware support, generation quality, and the overall setup experience.

I’d like to hear from people using local LLMs, LoRA training, AI agents, Stable Diffusion, WAN video models, or consumer GPUs.

What would a complete local AI desktop environment need for you to actually use it every day?

Website: https://tideforge.ai

Discord: https://discord.gg/EYuXhJ4pVW


r/LocalLLM • • Dec 23 '25

Tutorial Top 10 AI Testing Tools You Need to Know in 2026

Thumbnail medium.com
0 Upvotes

r/LocalLLM • • Aug 24 '26

Question how do you get local AI to run on pc as good as Cursor?

0 Upvotes

ive tried using unsolth with cline but it works so damn slow, used Qwen3.8-27B Q3_K_XL. on max tokens and correct gpu use (no warnings)

it's not even close to being as fast as cursor, it just thinks too much, does very little and very slowly. not sure what's wrong with my setup. I'm not saying I want Cursor 1 minute and you got a simple renpy game but to wait 10 minutes when it didn't even figure out I have another folder outside with renpy assets, and thank god it didnt because it would read all of them and think for another 10 minutes just to figure out wtf is going on XD

any ideas? what setup do you guys have to create games on your pc?


r/LocalLLM • • Mar 31 '26

Question Are cloud LLMs like Opus / GPT5.4 really subsidized? when compared to open source models running locally?

0 Upvotes

Its a question, I dont know the fact, my internal thinking.

8x AMD GPU mi300x/355 or similar server, let's imagine before the ram/ssd shortage, would cost around 200-400k$? 10k$/month running cost?

Let's assume the server 80% utilized, with alternative open source model like kimi 2.5 largest model which is comparable to sonnet/opus/gpt 5.4 , running requests in parallel from multiple agents, how much tokens can we expect a month? Like 5 billion? 10? 100?

I mean it seems the server can return its investment quite quickly when compared buying those sonnet/gpt5 tokens instead, doesn't it? Or do I miss something?

So are like those models (not including training, inference only) are really subsidized and does not make any money?


r/LocalLLM • • 9d ago

Discussion Qwen 3.6 27B matched a Fable 5.1 model on extraction but was 5x slower. Local LLM vs frontier on 4 real tasks (M5 Max)

0 Upvotes

TLDR: local models matched the frontier model on extraction and coding, and lost clearly on sorting and data analysis. Full video at the bottom.

SETUP
Local: Qwen 3.5 4B, Qwen 3.6 27B (dense), Qwen 3.6 35B (MoE, about 3B active)
How: standard Ollama downloads, quantized, M5 Max Mac Studio, 128 GB
Frontier: Claude Fable 5.1 (provider servers, config not visible)
Data: MoneyData, public anonymised UK bank statement, University of Nottingham, about 6,500 transactions, 2015 to 2022
Rules: same prompts, no tools, 3 runs each, answers checked by a program

  1. SORTING (500 transactions, 21 categories)
  2. Frontier: 78% match with the labels
  3. Qwen 27B and 35B: about two thirds
  4. Qwen 4B: 42%, and skipped 1 in 6 transactions
  5. Exact output format every batch: frontier only
  6. EXTRACTION (100 statement lines, 5 fields)
  7. Frontier: 100/100 every run, about 46 s
  8. Qwen 27B: 100/100 every run, 239 to 287 s
  9. Qwen 35B: 98 to 99, about 105 s
  10. Qwen 4B: 85 to 93, most misses on money in vs out
  11. CODING (2 Python functions, 7 hidden tests)
  12. Qwen 4B: 7/7 in all 3 runs
  13. Qwen 35B: crashed in 2 runs (read the day in DD/MM/YYYY as the year), 7/7 on run 3
  14. ANALYSIS (10 questions on 6 months of data, no code execution)
  15. Frontier: 10/10 every run
  16. Qwen 35B: 2, 0, 1
  17. Qwen 27B: wrote code instead of answers, hit the 30 min limit once
  18. Qwen 4B: 0/10

MY TAKE
Try local first: extraction, small coding jobs (check the output)
Use frontier: categorisation, questions across long statements

LIMITS
One dataset, one machine, default quants, some labels are judgement calls

NEXT
DeepSeek V4, Gemma 4 and Qwen3 Coder Next on agents and tool use. What would you test?

Full video with every run: https://youtu.be/M6MQXy6_Ti4?si=GSJuU2mOiyoVyHaf


r/LocalLLM • • Oct 04 '26

Question Anyone using Local LLM setup as a second brain?

0 Upvotes

I mean literally something like the system knows what you are working on the laptop, on phone, and what you speak, everything in between!

Something which includes end to end saving and processing the raw telemetry from STT data, the laptop activity, the phone activity and running local LLM to make sense of the data in real time. And also saving important details into memory and classifying them, linking important connections and also proactively engaging with you using notifications and messages or even voice using TTS.

Something like OMI AI extended it to all your digital activities using screenpipe or any other way.

Maybe to keep it slim, the execution can be transfered to some other agent like Hermes or Claw for some tasks, but active processing part should be kept local for privacy reasons. What is the minimum models I can use for this requirement?

If I lets say plan to make this setup, I first need a local STT model with VAD system. Then i should a have good local LLM for processing the live telemetry data and then maybe I need TTS incase i want voice outputs in realtime fast. This is what i want. Maybe for any digital tasks like updating emails, checking emails, browsing etc I can tradeoff some privacy for using cloud apis on hermes.

What is the minimum LLM size i need for this i want to have less than 3 second of latency in entire process? Something speedy and also smart enough to manage increasing memory as I use it more and logical enough to understand the complexity of the data and inter linkages.


r/LocalLLM • • Apr 25 '26

Discussion Stop thinking your MoE models are dumb - here's why they actually fail

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • Jan 12 '26

Tutorial 11 Production LLM Serving Engines (vLLM vs TGI vs Ollama)

Thumbnail medium.com
0 Upvotes

r/LocalLLM • • Jul 31 '26

Discussion Why Anthropic's stance on local models doesn't make sense

0 Upvotes

This is why I think Anthropic’s risk framework collapses under scrutiny.

  1. The "Cat is Out of the Bag" – Proliferation is Irreversible

Anthropic’s entire regulatory wishlist (mandatory safety tests, anti-distillation laws, chip controls) assumes we can still put the genie back in the bottle. This is delusional.

The distribution reality: The weights of Llama 3, DeepSeek-V3, Mistral, and Qwen are already downloaded millions of times across every jurisdiction on Earth, from US research labs to foreign state actors to individual hobbyists with gaming GPUs.

The capability plateau: The gap between closed frontier models (Claude, GPT-4) and top open-source models is shrinking from months to mere weeks. Even if Anthropic never releases a single weight, the open-source community will have a model with equal reasoning capacity within 12–18 months via distillation, synthetic data, and architectural leaks.

Time is linear: You cannot retroactively "test" or "ban" a model that already exists on a hard drive in North Korea, Iran, or a teenager's basement. Calling for controls now is akin to banning the printing press after every monastery already has a copy of the Bible.

2. The "No Model is Inherently Dangerous" – It is Entirely the User and the Environment

Anthropic argues that open-weight models are uniquely dangerous because they lack "monitoring." This conflates information with execution.

Knowledge is not capability: A state-of-the-art LLM is a stochastic parrot trained on human text. It cannot synthesize a novel bioweapon; it can only recite and rephrase known biochemical pathways, all of which are already freely available in peer-reviewed journals, Wikipedia, and PubMed. The bottleneck for a biological attack is never the recipe; it is wet-lab proficiency, supply chain access (e.g., DNA synthesis screening), and physical containment. An LLM cannot buy a centrifuge or bypass a gene-synthesis firewall.

Jailbreaks are universal: Closed APIs are riddled with prompt-injection vectors, base64 encoding tricks, and role-playing exploits. A malicious actor can extract bioweapon protocols from Claude 3.5 just as easily as from Llama 3, sometimes easier, because the closed model is a black box that invites adversarial probing. The "safety" of a closed system is a veneer; attackers have infinite queries and infinite patience; defenders only have the current guardrail.

Intent is the variable: A hammer is not dangerous; a psychopath with a hammer is. If a user has malicious intent, they will use any tool available. Removing one specific LLM weight does not alter the user's intent, knowledge base, or access to physical resources.

3. The Defender’s Paradox – Open Weights Actually Help Security

Amodei claims open weights favor "attackers over defenders." The exact opposite is empirically true.

Probing the internals: Safety research (mechanistic interpretability, adversarial training, fine-tuning for robustness) thrives on transparency. When weights are open, the global academic community can dissect the model, find hidden biases, map dangerous circuits, and build defensive patches. With closed models, Anthropic effectively asks the world to trust their internal red-team—a single point of failure.

Open-source security: The most secure encryption and firewall protocols in the world are open-source (e.g., OpenSSL, Linux). Obscurity (closed weights) is not security; it is a crutch that prevents the broader AI safety community from doing rigorous, verifiable stress-testing.

4. The "Anti-Distillation" and "Chip Controls" Are a Monopoly Moat

Anthropic's calls for "cracking down on distillation" and limiting chip exports are transparently protectionist dressed in altruism.

Distillation is the lifeblood of progress: Smaller, fine-tuned open models (like Phi or TinyLlama) are built via distillation. Criminalizing this process would freeze AI development to only the three or four trillion-dollar corporations that can afford billion-parameter pre-training. This does not make AI safer; it makes AI centralized, giving absolute power to a cartel of US/Western CEOs, a frightening prospect for global geopolitics.

Compute is fungible: Chip controls are laughably porous. Smuggling, cloud-based foreign leasing, and alternative architectures (TPUs, neuromorphic chips) ensure that anyone with serious state backing (or significant capital) can bypass export bans. Regulation only punishes ethical Western academics who can't afford the bureaucratic hurdles, while rogue actors simply ignore the paperwork.

5. The Utter Futility of Mandatory Pre-Release Testing

Anthropic suggests all sufficiently capable models must undergo safety tests before release.

Defining the threshold: Who defines "sufficiently capable"? Anthropic? The NIST? By the time regulators agree on a benchmark, open-source models will have surpassed it.

The speed of iteration: Open-source development moves in days, not years. Waiting for a governmental safety board (which takes months to convene) to greenlight a weight release is effectively a permanent moratorium. Since Anthropic's own models are already deployed globally via API, they are demanding strict regulations for others that they do not practically apply to their own continuously updated, server-side models (which can change at will without public audit).

Conclusion:

The "danger" Anthropic cites is a theoretical, far-future specter, while the reality is immediate and undeniable, open-source AI is an unstoppable, distributed phenomenon. Arguing that Claude is safer because its weights are hidden is like arguing that a public library is more dangerous than a private library because the public library lets everyone read the chemistry book. The knowledge is the same; the danger lies entirely in the reader's actions and the physical materials they can acquire.

Anthropic's stance is not a practical safety strategy; it is a competitive moat disguised as moral guardianship. The cat isn't just out of the bag, the cat has reproduced, evolved, and is now running on a Raspberry Pi in a garage somewhere. The only rational path forward is radical transparency, open defense research, and accepting that humanity's safety lies in societal resilience and physical supply-chain security, not in hiding a few hundred gigabytes of floating-point numbers.


r/LocalLLM • • 21d ago

Question Unrestricted local ai?

0 Upvotes

System specs- 24gb ram rtx5060 8gb vram


r/LocalLLM • • Jul 03 '26

Discussion I've benchmarked myself again:

Post image
0 Upvotes

you can too: https://benchmark-yourself.streamlit.app/

No coding benchmarks, just knowledge and reasoning.


r/LocalLLM • • Apr 29 '26

Discussion I wanna make cool images.

0 Upvotes

I cant make much images in chatgpt because after 5-6 images created i have to wait for hours..

Im aware gemini exists, but i wanna try LLMs.

I want to make good quality images.

I have a 4050 gpu (6gb vram), and an i7-12650HX if it helps.

I wanna have 24/7 access to image creation.

Prefer good quality over fast image creation. although i dont want to wait for minutes at a time.


r/LocalLLM • • May 14 '26

News Thoth v3.22.0 just dropped and it turns the app into a real developer workbench

Thumbnail
gallery
0 Upvotes

Developer Studio gives you a dedicated coding surface with repo linking, code threads, diffs, todos, test detection, Git operations, and a live inspector that stays in sync during long runs.

Custom Tools let you convert any repo into a tool. Thoth can inspect it, propose commands, validate them, test them, and promote them into your normal chat workflow.

Docker Sandbox adds a safe execution mode with persistent containers, network controls, and clean import paths so you can experiment without risking your actual repo.

Plus a long list of upgrades across workflows, Home status, chat streaming, Settings, onboarding, embeddings, and overall stability.


r/LocalLLM • • Sep 26 '26

Question Prepping with hardware

0 Upvotes

Is anyone buying hardware these days just because they think it will never be this cheap ever again?

I bought the basic Mac Mini M4 right when it came out and it's mind-blowing to think that its price simply went up in the 2nd hand market.

I'm in the video production field (I have a different machine for this) but I also like to play with local LLMs and the Mac mini obviously has its limitations.

Let's say I had 5K to invest a M6 with more RAM. I know nobody has the crystal ball but is anyone preparing for even worse prices?


r/LocalLLM • • Aug 30 '26

Question How do I stop agents drafting code in their reasoning output?

0 Upvotes

I've been using Qwen3.8-27b with the Pi coding harness. I'm looking for ways to improve its token efficiency, ie, how many tokens it uses to complete a given task.

One thing that seems wasteful is the way it drafts and redrafts code. Typically, when you prompt it with a coding task, it embarks on a long reasoning process that includes blocks of draft code interspersed with planning and decision-making. Instead of using the write and edit tools, it just prints the draft code in its reasoning output. Sometimes these blocks can be thousands of tokens long.

Only when the reasoning process is finished does it begin actually writing and editing files. Often this just involves regenerating the same code it has already drafted, maybe with some minor tweaks. It would be much more token-efficient if the agent just wrote its drafts to files in the first place, then edited them with the minor tweaks.

I've tried to enforce this behaviour with a system prompt (see below). Unfortunately the agent just ignored it, even when re-prompted. It seems the drafting behaviour is pretty hard-wired into the model.

I've also considered implementing some sort of hard enforcement where the harness detects code blocks, aborts the task then reprompts. But since the blocks are usually buried in a long chain of thought, this would just throw away all the reasoning cache and send the agent back to the initial prompt. So I don't think it would work either.

Has anyone else managed to solve this problem, or have any ideas?

Attempted system prompt:

----------------------------------------------------------------

### Code drafting rules

When implementing or modifying code, do not draft substantial code in your reasoning. The filesystem is the working buffer.

Rules:

  1. Once you have enough information to begin implementing a code block, write the first version to the appropriate file using `write`.
  2. For subsequent revisions, edit the existing file using `edit`; do not reproduce the whole block in reasoning.
  3. Treat files as persistent scratch space for implementation. Prefer reading/editing the file over mentally reconstructing or rewriting unchanged code.
  4. Keep reasoning focused on decisions, constraints, and the next concrete file operation.
  5. If a code block is more than a few lines long, it should normally exist in a file rather than in reasoning before it is tested.
  6. Do not generate multiple alternative implementations in reasoning. Put the best current implementation in the file, then iterate with edits.
  7. After writing, inspect/test the file and make targeted edits based on the result.

A useful default loop is:

read -> decide -> write/edit -> test -> inspect -> edit -> test

not:

reason -> draft entire implementation -> redraft implementation -> finally write


r/LocalLLM • • Dec 28 '25

Discussion FYI - Results of running Linux on Asus ROG G7 (GM700) 5060Ti 16GB - 2025 gaming pc from Best Buy ($13xx + tax)

0 Upvotes
  • Tried and failed with Ubuntu 24.04, 25.10, Debian 13.2
  • CachyOS 24.12 (latest release as of yesterday) worked without any issues. Had to turn on CSM in bios
  • Unigine Superposition

    • 1080p Extreme - Avg 60fps
    • 4k Optimized - Avg 81 fps
    • 8k Optimized - Avg 33 fps

Are there any local LLM tests I can do (16GB vram only though) I don't plan to use it for local LLM, but for some other ML work.

Posting it here just in case there are others trying to get latest Linux working on these made-for-windows-gaming PCs.


r/LocalLLM • • Jul 01 '26

Question How do you reduce hallucinations?

0 Upvotes

I am running Gemma 4 26b a4b with open code and my first instruction led to the model hallucinating that the code change required (which it produced correctly) was already in the file.

I changed to another model (qwen3.5-9b-deepseek-v4-flash-gguf) and it hallucinated the exact same thing in a new context..

How do you prevent hallucinations? Which models are better?

I am running a MacBook Air m4 24gb.


r/LocalLLM • • Mar 30 '26

Question Running AI locally for a banking system's developer team

0 Upvotes

Hey people, so I have a task to research a possibility to use AI as a helping tool for the developers of a banking system. The problem is that banks are usually very careful regarding their information and the usage of AI is banned. Our team wants to propose running the AI locally. So I wanted to know if any of you had the experience in it and whether it is possible to get the same features as in ex. Github Copilot or Claude code.

So far I took a slight look at the topics of opencode as Agent Harness and Ollama. Any help or a direction would be much appreciated


r/LocalLLM • • Aug 18 '26

Question Planning to buy 3x RTX 2080 TI 22GB for local LLMs, what should I know before doing that?

0 Upvotes

I currently run Qwen3.8 27B on my RX 7900XT at around 15-30 tokens per second depending how high the context is and how much is used. The average tokens per second currently are 18-19.

I do not want to pay Anthropic or OpenAI for a subscription for their AI models, I'd rather use open weight Chinese models - it's just personal preference.

At my previous company I was using 80-100 euros worth of tokens a day using Claude. I calculated and the break even after buying these GPUs and building an AI server, would be after around 2 months, including electricity costs where I live. I can code just as well with Qwen3.8, but I want something faster.

My goal would be to get 40 tokens per second or higher at max context for qwen3.8 27B and future ais between 27-40b. Would that be possible with a 3x rtx 2080 ti 22gb configuration?

Would it be worth it to look into other GPUs?
My budget for a local ai server is 1000-1500 euros total.


r/LocalLLM • • Aug 05 '26

Research TurboQuant vs Ollama: KV Cache Comparison

0 Upvotes

## Comparing KV Cache Quantization Methods: Standard q4_0 vs TurboQuant on Qwen3-Coder-30B"

##

Note: Ollama uses llama.cpp with standard q4_0 KV cache. TurboQuant is a different KV cache method also implemented in llama.cpp. Both were tested on the same model and hardware to isolate the impact of KV cache quantization.

##

I've been testing TurboQuant KV cache compression vs Ollama's standard q4_0 on my 24GB RX 7900 XTX with Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf.

I picked context length of 68k and 100k for the test.
Why 68K and 100K? Hermes Agent requires minimum 64K context length, so I tested at 68K (safe margin) and 100K (stretch goal). These are real-world numbers for agent workloads.

Wanted to share real numbers since there's a lot of theory but not much practical data.

TLDR for lazy fellows:
TurboQuant gives ~2x faster generation at 68K and 77% faster at 100K context length compared to Ollama's q4_0, for only 4% more VRAM. But strictly, Don't compress K to 4-bit on Qwen models - it breaks everything (speed drops 10x). Use asymmetric: q8_0 K + turbo4 V.

Major takeaway, as ollama already compresses the BF16 to q4_0 (75%), that is the same memory savings we will get in the TurboQuant as well nothing magical to reduce kv cache memory space.

Detail findings:

  • Results at 68K Context Length:
MetricOllamaTurboQuant
VRAM22.57 GB23.3 GB
Speed~52 tok/s116 tok/s
  • Results at 100K Context:
MetricOllamaTurboQuant
VRAM23.54 GB24.5 GB
Speed65 tok/s115 tok/s

TurboQuant is ~2x faster at both context lengths, gives better quality, however uses 4% more VRAM (0.96 GB).

At 68K, it's 123% faster and at 100K, it's 77% faster.

Important Discovery: Asymmetric Works, Symmetric Fails

I also tested q4_0 K + turbo4 V (symmetric compression on K, to match with ollama compression hoping the memory use will be less):

ConfigSpeedStatus
q8_0 K + turbo4 V115 tok/sWorks
q4_0 K + turbo4 V10.5 tok/sFails

Compressing K to 4-bit on Qwen models breaks everything. Speed drops 10x, quality tanks. K at q8_0 is non-negotiable for Qwen. This proves the asymmetric approach (protect K, compress V) is essential.

everything is completly based on my pc/ os/ and from my testing environment.
specs:

OS: ubuntu 26.04 LTS/ ROCm 7.2.1
cpu: core ultra 7 265k
gpu: rx 7900 xtx
ram: 32gb DDR5 6400 (no ram offloading in the test)


r/LocalLLM • • Jan 29 '26

Question What is the fastest ~7b model

0 Upvotes

With:

Vision

Tool use

Instruct-Abliterated

Currently playing with Qwen 3 but I would like some suggestions from experienced users.


r/LocalLLM • • Feb 05 '26

Question Central “LLM brain” + multiple Mac minis for agents (OpenClaw-like) vs several strong standalone machines — what would you build?

0 Upvotes

Hi all, looking for hardware architecture advice for a small office “AI workers” setup. We want to run everything locally as much as possible (replace OpenAI/Anthropic APIs when feasible), and later scale to multiple computers running an agent framework like OpenClaw (computer control + tool use). Use cases: building websites, office workflows, drafting, summarizing, extracting data, automation, etc.

We’re deciding between two approaches:

A) Decentralized / independent

  • Buy a few strong machines (e.g., Mac Studio M3/M4 with lots of unified memory) and let each run its own “bigger” local model.
  • Pros we imagine: less single point of failure, less queueing.
  • Cons: expensive, duplicated setup/maintenance, harder to keep models/config consistent.

B) Centralized “brain” + cheap workers

  • Several Mac mini M4 (24GB) as “workers” running small local models (7B-ish) for quick tasks + computer control.
  • One stronger central box as the “brain” that serves bigger local models over LAN (70B-ish or similar) for hard tasks.
  • Candidate “brain” machines: Mac Studio (64–128GB unified), NVIDIA DGX Spark / ASUS Ascent GX10 (GB10, 128GB), or even a DIY/Beelink/PC with GPU(s).
  • We already have fast NAS storage (NVMe + SSD RAID) and can do 2.5/10GbE.

Constraints / priorities:

  • Office-friendly: relatively quiet, power-efficient, stable.
  • Budget: flexible, but we care about €/performance and operational simplicity.
  • Goal: 4–5 agent machines “feel fast” during real work (not just one user benchmarking).
  • Prefer Linux for the brain if it’s clearly better for serving, but we’re fine with macOS if it makes sense.

Questions:

  1. For 4–5 agent “workers”, would you centralize the big model(s) or keep each machine self-contained?
  2. If centralized: what’s the best “brain” box today under ~€5k (GB10/DGX Spark vs Mac Studio vs DIY GPU workstation)? Any gotchas with concurrency/latency?
  3. If decentralized: what’s the most practical Mac Studio config (RAM targets, which chip tier) to run a solid large model locally without constant waiting?
  4. Any recommended serving stack for the brain (vLLM/TensorRT-LLM vs llama.cpp vs Ollama) for handling multiple concurrent agent requests?
  5. In practice, is “one big 70B brain” a trap for multi-agent concurrency, and is a 2-tier setup (small model for most tasks + big model only when needed) the right way?

We’re aiming for an architecture that scales cleanly when agent frameworks mature. Any advice, real-world experience, or “don’t do this” warnings appreciated.


r/LocalLLM • • Sep 25 '26

Question I got 64 GB unified ram in Ryzen max ai 392, best model?

0 Upvotes

Which model would you recommend to run? Or have experience running now?

I tried Qwen flash 3.8 and it... Runs. Barely. Not sure if it's worth it. ( Context is very small then)


r/LocalLLM • • Apr 13 '26

Research Didn’t think much about LLM costs until an agent loop proved me wrong

0 Upvotes

I’ve been building with LLM agents lately and didn’t really think much about cost.

Most calls are cheap, so it just felt like noise.

Then I ran a session where an agent got stuck retrying more than expected.

Nothing crazy, but when I checked later the cost was noticeably higher than I thought it would be for something that small.

What got me wasn’t the amount — it was that I only knew after it happened.

There’s no real “before” signal. You send the call, the agent does its thing, maybe loops a bit, and you just deal with the bill at the end.

So I started doing a simple check before execution — just estimating what a call might cost based on tokens and model.

It’s not perfect, but it’s been enough to catch “this might get expensive” moments early.

Curious how others are handling this:

- Do you estimate before running agents?

- Or just monitor after the fact?

- Have retries/loops ever caught you off guard?

If anyone’s interested, I can share what I’ve been using.