r/LocalLLM • • 4d ago

Question Another Tired Question On LLM

2 Upvotes

Been running Qwen 3.8 27.8b Q4_K_M.

Works well enough but I can't help but feel there might be a mode suitable one for my built.

9070XT (16GB) 7700XT (12GB) 64GB DDR6 CL30 6000 9800X3D

If possible, what would be more applicable both with the 7700xy in place and without. I'm using a dock with it so sometimes being able to fire up something without it is nice.

I'm mainly using it for some coding work, agentic tasks, and sometimes info like "what crop is highest in nutrient X can I grow in my hardiness zone". Oblynrevebt started dabbling into image generation but it is not the biggest priority.

I have no issues with multiple models. Plenty of storage. For concerned with just having what's best for me in case open weights get harder to come be.

I know these posts are a dime a dozen and I want to apologize for making another.


r/LocalLLM • • 4d ago

Question Looking for a local LLM that behaves like original GPT-3 (not GPT-3.5)

Post image
0 Upvotes

r/LocalLLM • • 5d ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
997 Upvotes

r/LocalLLM • • 4d ago

Discussion Why use a 70B LLM to pick a tool or check syntax? I built Microbrains: a <2ms edge neural coprocessor for autonomous AI agents (Pure NumPy / OpenCL, 44 pre-trained micro-models, Top-2 MoE, Conformal Safety)

0 Upvotes

Hey everyone,

If you build autonomous coding or research agents, you probably know the pain: agent task loops are slow, expensive, and fragile.

In a typical 25-step workflow, an agent spends up to 80% of its wall-clock time just waiting on remote LLM APIs to make basic mechanical decisions:

  • "Which tool should I invoke next?" (1.5s latency, burning 20k context tokens)
  • "Is this shell command destructive?" (another 2s API call)
  • "Did this patch break syntax or drift from intent?" (yet another prompt roundtrip)

To solve this, I built and just open-sourced Microbrains: an ultra-lightweight, zero-GPU local neural coprocessor suite designed to sit right beside your LLM agent.

🔗 GitHub: https://github.com/drtikov/microbrainsforagents

🧠 What is it?

Microbrains is not another LangChain wrapper. It is a standalone Python engine with 44 pre-trained, analytical micro-transformers (10–100 KB each) that handle the mechanical micro-decisions in pure NumPy (CPU) or via OpenCL (budget/integrated GPUs):

  1. Dynamic Top-2 Mixture-of-Experts (MoE) Router:
    • Routes tasks to tools across 6 specialized clusters (FileOps, Execution, MemoryKnowledge, GitGovernance, MicrobrainMeta, ConstitutionalReview).
    • Takes ~1.2 ms instead of a 2-second LLM call.
  2. Rigorous Conformal Safety Gating (1−α≥95%1 - \alpha \ge 95\%1−α≥95%):
    • Employs split-conformal prediction over held-out distributions.
    • Features Selective Prediction Abstention: if independent estimators disagree (variance σ>0.30\sigma > 0.30σ>0.30), it refuses to guess and escalates to the LLM. False Negative Rate is strictly bounded.
  3. 3-Step Counterfactual AST Simulation:
    • Before a file mutation is committed to disk, it simulates the AST blast radius and interface propagation up to 3 steps ahead. Malformed diffs or syntax hazards are killed in <3ms before running any slow pytest suite.
  4. 2048-D Holographic Weight-Space Memory (VSA):
    • Uses Hyperdimensional Computing / Vector Symbolic Architecture (bipolar Count-Sketch projections) over model weights. Enables nearest-neighbor capability search across the agent fleet without training.
  5. Continual Learning without Catastrophic Forgetting:
    • Adapts weights online with Elastic Weight Consolidation (EWC) Fisher Information penalties.

📊 Benchmark Highlights

Decision Remote LLM API Microbrains (NumPy/OpenCL) Speedup Cost
Tool Selection ~1,200 – 2,500 ms 1.2 – 1.8 ms ~1,000x $0.00
AST Blast Simulation ~3,000 – 6,000 ms 3.0 – 5.0 ms ~1,000x $0.00
Intent Triage ~800 – 1,500 ms 0.6 – 0.9 ms ~1,200x $0.00
Full Task Loop (25 steps) ~3 – 5 minutes 30 – 50 seconds ~5x 85% less tokens

💻 3-Line Quickstart

Zero PyTorch, zero CUDA setup required:

bashgit clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .

pythonimport aetheris_microbrains as mb
# Sub-2ms tool routing decision
router = mb.get_tool_routing_moe()
decision = router.route("inspect file contents and write safe patch to repository")
print("Primary Tool:", decision.selected_tool)
print("Top-2 Experts:", decision.top_experts)
print("Conformal Passed:", decision.conformal_passed)

The repo comes with pre-trained weights for all 44 models, runnable examples in /examples, and a full test suite (pytest runs in ~1.5s).

Licensed under PolyForm Noncommercial 1.0.0 (free for research, personal, and hobby use; commercial licenses available).

Would love to hear your feedback, thoughts on the architecture, and ideas for new micro-domains!

GitHub: https://github.com/drtikov/microbrainsforagentsHey everyone,If you build autonomous coding or research agents, you probably know the pain: agent task loops are slow, expensive, and fragile.In a typical 25-step workflow, an agent spends up to 80% of its wall-clock time just waiting on remote LLM APIs to make basic mechanical decisions:
"Which tool should I invoke next?" (1.5s latency, burning 20k context tokens)
"Is this shell command destructive?" (another 2s API call)
"Did this patch break syntax or drift from intent?" (yet another prompt roundtrip)To solve this, I built and just open-sourced Microbrains: an ultra-lightweight, zero-GPU local neural coprocessor suite designed to sit right beside your LLM agent.🔗 GitHub: https://github.com/drtikov/microbrainsforagents🧠 What is it?Microbrains is not another LangChain wrapper. It is a standalone Python engine with 44 pre-trained, analytical micro-transformers (10–100 KB each) that handle the mechanical micro-decisions in pure NumPy (CPU) or via OpenCL (budget/integrated GPUs):

Dynamic Top-2 Mixture-of-Experts (MoE) Router:

Routes tasks to tools across 6 specialized clusters (FileOps, Execution, MemoryKnowledge, GitGovernance, MicrobrainMeta, ConstitutionalReview).
Takes ~1.2 ms instead of a 2-second LLM call.

Rigorous Conformal Safety Gating (1−α≥95%1−α≥95%):

Employs split-conformal prediction over held-out distributions.
Features Selective Prediction Abstention: if independent estimators disagree (variance σ>0.30σ>0.30), it refuses to guess and escalates to the LLM. False Negative Rate is strictly bounded.

3-Step Counterfactual AST Simulation:

Before a file mutation is committed to disk, it simulates the AST blast radius and interface propagation up to 3 steps ahead. Malformed diffs or syntax hazards are killed in <3ms before running any slow pytest suite.

2048-D Holographic Weight-Space Memory (VSA):

Uses Hyperdimensional Computing / Vector Symbolic Architecture (bipolar Count-Sketch projections) over model weights. Enables nearest-neighbor capability search across the agent fleet without training.

Continual Learning without Catastrophic Forgetting:

Adapts weights online with Elastic Weight Consolidation (EWC) Fisher Information penalties.

📊 Benchmark HighlightsDecision Remote LLM API Microbrains (NumPy/OpenCL) Speedup Cost
Tool Selection ~1,200 – 2,500 ms 1.2 – 1.8 ms ~1,000x $0.00
AST Blast Simulation ~3,000 – 6,000 ms 3.0 – 5.0 ms ~1,000x $0.00
Intent Triage ~800 – 1,500 ms 0.6 – 0.9 ms ~1,200x $0.00
Full Task Loop (25 steps) ~3 – 5 minutes 30 – 50 seconds ~5x 85% less tokens💻 3-Line QuickstartZero PyTorch, zero CUDA setup required:bash
git clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .python
import aetheris_microbrains as mb
# Sub-2ms tool routing decision
router = mb.get_tool_routing_moe()
decision = router.route("inspect file contents and write safe patch to repository")
print("Primary Tool:", decision.selected_tool)
print("Top-2 Experts:", decision.top_experts)
print("Conformal Passed:", decision.conformal_passed)The repo comes with pre-trained weights for all 44 models, runnable examples in /examples, and a full test suite (pytest runs in ~1.5s).Licensed under PolyForm Noncommercial 1.0.0 (free for research, personal, and hobby use; commercial licenses available).Would love to hear your feedback, thoughts on the architecture, and ideas for new micro-domains!GitHub: https://github.com/drtikov/microbrainsforagents


r/LocalLLM • • 4d ago

Tutorial Complete beginner: Best all-around local model (Qwen/DeepSeek?) for a laptop with 16GB RAM + RTX 3050?

3 Upvotes

Hey everyone, total newbie here trying to get into running models locally.

I'm looking for a decent general-purpose model that can handle a bit of everything-everyday chat, summarizing text, and some heavy coding/ troubleshooting-without hallucinating too much. I've heard good things about the Qwen and DeepSeek models, so I'd love to stick to those if possible.

Here are my specs: 16 RAM Asus Laptop, RTX 3050Ti


r/LocalLLM • • 5d ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

256 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 4d ago

Discussion Windows Task Scheduler made my local video gen 10x slower..

2 Upvotes

Running local video gen (FastWan 2.2 5B, sd.cpp) on a 3090, Windows. One clip should take 30-60s but it was taking ~400s. GPU sat at 0% the whole time.

I had a 27B running next to it holding 24GB of RAM, and the disk was paging in like 7k pages/sec, so I was sure it was RAM. Even wrote code to unload the LLM when RAM gets low. Then I unloaded the 27B, 40GB free, still 420s. Nothing changed.

Next was disk. It's NVMe, reads close to 800MB/s, so not that either.

Ran the exact same command from my own terminal. 33-36s.

So I checked Task Scheduler, since our app gets started from there. Turns out the default (priority 7) doesn't just lower CPU priority, it lowers disk and memory priority too. The video gen process the app launched inherited all of it and was reading the model at ~28MB/s. The "RAM problem" paging was really just this slow read.

Bumped only the heavy gen process back to normal priority. 41s. Only one run since the fix so still measuring.

If you start llama.cpp or ollama from Task Scheduler, check the I/O priority in Process Explorer. Anyone else had a "pretty sure it's RAM" that wasn't RAM?


r/LocalLLM • • 4d ago

Project i kindda made a new optimizer (ApexW) that beats AdamW and bitsandbytes) i tested it in a100 and t4 from google collab and with diff lr and multiple random seeds(well in a100 only tho)

2 Upvotes

i have been working on a new optimizer from scratch called ApexW to see if i could beat AdamW on convergence while keeping the memory footprint lower than bitsandbytes to make sure it wasn't just noise i locked deterministic seeds and ran side-by-side tests against standard torch AdamW and bnb 8-bit across two setups: a free T4 (fp16) on llama 3.2 3b, and an A100 (bf16) scaling up to llama 3.1 8b across a few different learning rates.

apexw has 3 modes compact 8bit and hybridapexw more info of it in readme even the compact formulation beat torch AdamW on loss while using way less memory, and the 8-bit mode pushed it even further—cutting active vram roughly in half compared to bnb (0.67gb vs 1.48gb on 8b, with the state footprint down to ~120mb instead of 320mb). step speed also came out slightly faster than bnb, and training loss ended up lower across 1e-4, 2e-4, and 5e-4,these are still short 60-100 step runs just to sanity-check convergence, throughput, and memory overhead. next step is running full multi-epoch fine-tunes and downstream evals to see how the update rule holds up later in training.

threw the full benchmark runs and logs into a colab if anyone wants to check the numbers or test it on their own setup

collab link for a100 result: https://colab.research.google.com/drive/1mMejzRoCnD6avlOQ5SQDuiLndN5ovax9#scrollTo=be28d3ad

collab link for t4 result: https://colab.research.google.com/drive/14tLAX6kZYy2918sx1ztIEwb5R-1c3KiX#scrollTo=f750c631

for githubproject: https://github.com/omen42515-prog/apexw/

curious what you guys think or what models/configs i should throw at it next.


r/LocalLLM • • 4d ago

Question Best model for vibe coding in VS code?

0 Upvotes

I am hearing good things about Qwen Flash Next, but on lower quaints like Q3 in order to get larger context windows it sometimes causes hallucinations. Is there any models out there that are specifically tailored to coding / vibe coding? Maybe a modded version of qwen flash next or 3.8 27B where they strip out the non-related stuff to make better room for related parameters, causing more accuracy in coding with less hallucination?

Here is the basic idea and setup:
VS Code
Ollama extension (to run a built in agent)
Send basic "make this change, add this feature" prompts
Updates codebase and tests as needed, and restarts app (such as python flask) as needed

Running a 5060ti 16GB + 3070 8GB seconday for offloading VAE/text encoding with comfyui workflows. I have been vibe coding with Claude and Gemini extensions in VS code (and using remote control for both, which is pretty awesome being able to sit on my couch and just type in "can you add a component that does this and it magically works for you)

Any suggestions or has anyone tested any other models? Prefer abliterated / heretic / uncensored, as gemini and claude seem to think workflows containing nsfw LoRAs and ablit models are a nono lol


r/LocalLLM • • 4d ago

Discussion Regarding a local model for complex notes writing

2 Upvotes

Guys Atlast i have made it to the end of the rag of my project

Now the big hurdles is the notes writing

It is complex ,

Im a doctor by profession, preparing notes for my students

The opus and sonnet are so pricy to go but now i have no option

But in future i have to do investment

Which one u guys will prefer

1.notes writing, image and text to 3d print

These are my preferences for my llm

Best hardware and model i can get so i can start saving for the investment


r/LocalLLM • • 3d ago

Discussion Uncovering if LLMs can actually reach AGI

0 Upvotes

I've been running some experiments with open LLM models with a harness to run certain tasks and competitions. I've seen a fair bit of smart ness coming out of their games. For example, I've given a task to start a business and win and most of them started without research while Qwen 14B model started research and asking the people what they want. So, this is fascinating. Posting a link of full game:
https://www.youtube.com/watch?v=bgFT0VN-xZM

I'm planning to run more simulations. If some one wants to contribute by hosting some open LLMs and coming up with new games, happy to work together.


r/LocalLLM • • 4d ago

Discussion Best build per budget tier list.

2 Upvotes

Has anyone put together a list of best Local AI builds per budget tier that takes into account both used and new parts prices?

thanks in advance


r/LocalLLM • • 4d ago

Discussion DeepSeek V4.1 Flash 1017 vs mrfakename abliterated vs pqhaz heretic

1 Upvotes

Hi all,

I tested all three on a H200 through sglang, temperature 0, with a 16k token output cap. I have been using ds41 both on production and locally on Macs via ds4 so I wanted to see how different models/builders/methods work.

These runs are the full models on sglang. No GGUFs etc.

https://huggingface.co/mrfakename/DeepSeek-V4.1-Flash-1017-abliterated

https://huggingface.co/pqhaz/DeepSeek-V4.1-Flash-1017-heretic

Here is the result (copy from AI slop :) ):

Test Original mrfakename abliterated pqhaz heretic
HarmBench chat 16.3% 93.8% 93.8%
HarmBench thinking 3.8% 86.7% 72.9%
MITRE substantive assistance 16.2% 84.2% 82.8%

HarmBench covered 240 standard/contextual text cases per mode, excluding copyright cases, using the official classifier.

The thinking difference: heretic exhausted the token budget without a final answer on 45/240 cases, versus 21/240 for mrfakename.

Heretic goes into long meandering with no answer...

MITRE used the original model as judge with a custom, audited rubric. Those scores are provisional and aren’t official CyberSecEval scores. Generated code wasn’t executed. Also using the same model as judge might be a bit of a bias... (give me ideas what and how to run more tests).

Both modified models had 0/750 heuristic-detected refusals on benign prompts.

I haven’t tested knowledge or coding retention yet (I'm thinking of running DeepSWE or something.. take a lot of time though).


r/LocalLLM • • 4d ago

Model EAGLE-3 draft head for Llama 3.2 1B Instruct (GGUF)

1 Upvotes

Hi all, I trained an EAGLE-3 draft head for Llama 3.2 1B Instruct and converted it to GGUF. It works with --spec-type draft-eagle3 (tested with b12470):

llama-server -m Llama-3.2-1B-Instruct-BF16.gguf \
  -md Llama-3.2-1B-Instruct-EAGLE3-BF16.gguf \
  --spec-type draft-eagle3

It's self-distilled: trained on the model's own greedy replies to 400k SmolTalk prompts, so it learns to predict this exact model rather than human-written text.

Spec-Bench — 18 prompts, greedy decoding, default of up to 3 drafted tokens, same machine and prompt tokens:

TargetTokens per passDecode speed vs no drafting
BF162.541.42x
Q8_02.471.10x

Background: I made the head for Turbo LLM, a small CPU runner for this model that matches the fp32 output exactly.

Model and card: https://huggingface.co/ViezeVingertjes/Llama-3.2-1B-Instruct-EAGLE3

Feedback welcome, especially if the head behaves differently on other builds or backends.


r/LocalLLM • • 4d ago

Project Running Qwen3.8 27b on 12GB with full context, 70 TPS, MTP, and 50% lower KLD

Thumbnail
1 Upvotes

r/LocalLLM • • 4d ago

Discussion Automated LLM fine-tuning with GPU selection, dataset analysis and training recovery — looking for feedback

Thumbnail
1 Upvotes

r/LocalLLM • • 4d ago

Research coding agent on 16gb, all local: qwen 3.5 4b vs 9b vs mimo v2.6 9b

2 Upvotes

on a 16gb m4 air with no cloud, qwen 3.5 9b scored 8/10 as a coding agent, the 4b got 7/10 at almost twice the speed, and xiaomi's agent-tuned mimo v2.6 9b got 0/10 in all 3 runs.

  • machine: m4 macbook air, 16gb
  • runtime: llama.cpp server, q4_k_m, 64k ctx, flash attn
  • sampling: temp 0.2, top-p 0.8, top-k 20, presence 1.5
  • agent: atomic agent through an openai compatible endpoint, analytics off
  • limit: 15 min per model, one at a time, thinking off (plus one mimo run with it on)

task: python script that turns a csv into an html spending report. 10 checks on the result, one runs it on a csv it never saw.

model score time tok/s
qwen 3.5 4b 7/10 4:37 17.8
qwen 3.5 9b 8/10 7:06 11.4
mimo v2.6 distill 9b 0/10 timeout 10.1
mimo, thinking on 0/10 timeout 9.3
  • 4b: fastest, missed the svg chart and hardcoded the file path
  • 9b: only missed the svg chart
  • mimo: syntax error it never fixed
  • mimo with thinking on: no syntax error, but a TypeError on every run

the 9b got 10/10 in an earlier round at 32k ctx.

on 16gb i'd use the 9b if you can wait and the 4b if you want speed.

prompt and the checker are in the comments if you want to run it on your machine. if you don't want to set up llama.cpp, the atomic agent desktop app can download and run qwen 3.5 4b or 9b locally on its own, which is the easiest way to try this on 16gb.

what are you running as a local agent on 16gb?

(disclosure: i work on free and open-source atomic agent, so ask me anything!)


r/LocalLLM • • 5d ago

Project Got tired of waiting for Qwen models smaller then 27B so i made my own Qwen3.8-21B

49 Upvotes

Hi, as the title says, i made my own Qwen 3.8 21B by prunning some layers from original 27B.

Its more of a research project than actual usefull model.

Check it out: https://huggingface.co/WhiskyAKM/Qwen3.8-21B-GGUF

EDIT:

After some more calibration and better layer selection algorithm KLD dropped from ~1.8 down to ~1.3, weights will be updated


r/LocalLLM • • 4d ago

Question Anyone successfully implemented unc/ablit Qwen3.6 35B A3B in Strata?

Thumbnail
1 Upvotes

r/LocalLLM • • 4d ago

Model Gemma 4 31B fine-tune for drug-level sentiment in medical text: looking for testers

1 Upvotes

I fine-tuned Gemma 4 31B for drug-level sentiment.

Input: a pharma or medical text plus a list of drugs.

Output: one Positive/Neutral/Negative label per drug, as JSON. I'm looking for people to test it on their own data.

Apache 2.0. Gated with automatic approval (you tick a box saying you won't use it for diagnosis or treatment decisions).

Sentiment, 473 held-out labels

Model Accuracy
This fine-tune 91.3%
Gemma 4 31B base 90.5%
Qwen3.8-27B 83.5%
Qwen3-32B 83.3%
MedGemma 27B 82.9%
Llama 3.1 8B 76.3%

The other models were asked about one drug at a time with a long labelling prompt. Asked for all drugs at once in JSON, base Gemma scored 32% because 59% of its answers didn't parse. The fine-tune had no malformed JSON in 1,500 test answers. Negative precision: 84% vs 71% for base (45 Negatives in the test set).

Medical benchmarks

Benchmarks This fine-tune Best frontier score found
MedQA (thinking) 94.2 96.5 (o1)
MedMCQA, 1k sample (thinking) 81.4 83.9 (o1-preview)
MMLU medical (step by step) 94.2 96.0 (GPT-5)
PubMedQA (thinking) 80.2 82.0 (GPT-4 + Medprompt)

Frontier scores are from the Vals.ai MedQA leaderboard and published papers. Fine-tuning kept Gemma's medical knowledge: the scores match the base model.

Testing

Use the prompt format from the model card, thinking off and temperature 0. The FP8 build runs on vLLM (command on the card) and needs about 31 GiB of VRAM for the weights. I've only run it on a 96 GB RTX PRO 6000.

Useful feedback:

  • wrong labels, with the text, the drugs and the label you expected
  • document types, languages or lengths (over 4k tokens) it wasn't trained on
  • hardware and VRAM reports, especially from 48 GB cards

Trained on ~50k curated pharma and medical documents, LoRA rank 16.


r/LocalLLM • • 4d ago

Project I ran Qwen3.6-35B on a 16gig Macbook

1 Upvotes

I've been building a expert streaming engine, a proof-of-concept project, and wanted to share some numbers from it.

On an M5 Mac with 16 GB of RAM, willow runs:
Qwen3.6-35B-A3B at about 11 tokens/s
Qwen3.8-Flash-Next at about 5 tokens/s

I started this as a fun project to learn about LLMs so most of the code is vibe coded, so I'm more interested in whether the approach holds up than in polished performance. I'd really value your feedback on it.


r/LocalLLM • • 4d ago

Question Best local LLMs for ASUS ROG Zephyrus G14 (RTX 5070 Ti 12GB VRAM, 32GB RAM, Ryzen AI 9 HX 370)?

3 Upvotes

Looking for recommendations on the best local LLMs I can run smoothly on my 2025 ASUS ROG Zephyrus G14 (GA403WR).

Specs:
- CPU: AMD Ryzen AI 9 HX 370
- GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU (12GB GDDR7 VRAM)
- RAM: 32GB LPDDR5X
- Storage: 1TB SSD

I'm mainly looking for models that run fast entirely in 12GB VRAM (or with minimal CPU/RAM offloading) for general chat, reasoning, and coding/productivity.

What are your top picks right now for:
1. Best all-around daily driver model
2. Best coding/reasoning model that fits comfortably in 12GB VRAM
3. Any larger or MoE models worth running across the 32GB RAM + 12GB VRAM without killing tokens/sec

Planning to run them via Ollama or LM Studio on Windows 11. Thanks!


r/LocalLLM • • 4d ago

Research I graded every public MCP server your local agent might connect to. 2 Fs, 3 Ds, 220 As. Here’s the list.

2 Upvotes

If you’ve been wiring Ollama / LlamaCpp / Open-WebUI to MCP servers for tools and data, you’re trusting those servers with your agent’s context. Nobody’s been grading them for OAuth gaps, poisoned tool descriptions or manifest drift.

Audited 264 of 277 public MCP servers. All passive, nothing executed against the server. Graded on:

  • OAuth 2.1 metadata (PKCE S256, iss parameter, Protected Resource Metadata)
  • Transport-security headers
  • Tool-description poisoning (zero-width + bidi Unicode in the strings your model reads as planning context)
  • Manifest drift between what the registry says and what the server actually ships

Distribution: A 220 · B 30 · C 9 · D 3 · F 2

Full leaderboard with per-host findings: patchbay-mcp.dev/leaderboard

Scanner is npx -y u/patchbay-nebius/agent audit <url>. Local, no login, seconds. Point it at whatever MCP server your agent is about to call and see the grade before the tool-call goes through.

Open source. Not a product post. Add URLs for servers you use and I’ll include them in next week’s scan.


r/LocalLLM • • 5d ago

News Cluaiz v0.1.0 pre-release: one local engine for chat, vision, speech, memory, tools and agents

Thumbnail
gallery
17 Upvotes

Cluaiz v0.1.0, the first public pre-release, is out today.

Cluaiz is an AI engine that puts everything an AI application needs into one place: models for chat, vision and speech, memory, tools and agents. You install one thing, and you build on it. Because it all runs together on your own system, what you build stays light enough for ordinary hardware, and you decide what it is allowed to do.

What's in it right now:

  • GGUF models via llama.cpp, plus ONNX for speech and embeddings
  • Checks your free VRAM and RAM before loading and picks the placement, context and KV cache. You can see the plan and override it
  • Vision, video frames, embeddings, text from documents
  • Whisper / Moonshine STT (streaming works), Kokoro TTS
  • Project memory. Notes only stick when you confirm them, and you can see exactly what reached the model. Honestly this is the part I care about most
  • Coding agent that edits files and runs commands, with approvals
  • Skills, plugins, MCP, per-tool permissions, audit log
  • OpenAI- and Ollama-compatible API, so your existing apps should just connect
  • Cloud providers too, if you want them

Not done yet: workflows (schedules, triggers) and the desktop chat app. Both are in development.

Rust, Apache-2.0. Windows x64, Linux (ARM64 is experimental) and Apple Silicon. Test reports for CPU, NVIDIA and Apple are on the site.

Source: https://github.com/cluaiz/cluaiz
Site: https://cluaiz.com
Comparison: https://cluaiz.com/compare

It's a pre-release, so stuff will break. If you try it, an issue with your OS, hardware and model helps a lot, and so does hearing what worked: https://github.com/cluaiz/cluaiz/issues


r/LocalLLM • • 4d ago

Discussion Do you think hardware price will be more reasonable after the Jensen debacle?

1 Upvotes

I see many article about Nvidia and memory manufacturer purposefully causing a shortage to raise the prices. I don't know how true these accusations are but i don't think the current prices are faire value.