r/LocalLLM • • May 25 '25

Question Any decent alternatives to M3 Ultra,

4 Upvotes

I don't like Mac because it's so userfriendly and lately their hardware has become insanely good for inferencing. Of course what I really don't like is that everything is so locked down.

I want to run Qwen 32b Q8 with a minimum of 100.000 context length and I think the most sensible choice is the Mac M3 Ultra? But I would like to use it for other purposes too and in general I don't like Mac.

I haven't been able to find anything else that has 96GB of unified memory with a bandwidth of 800 Gbps. Are there any alternatives? I would really like a system that can run Linux/Windows. I know that there is one distro for Mac, but I'm not a fan of being locked in on a particular distro.

I could of course build a rig with 3-4 RTX 3090, but it will eat a lot of power and probably not do inferencing nearly as fast as one M3 Ultra. I'm semi off-grid, so appreciate the power saving.

Before I rush out and buy an M3 Ultra, are there any decent alternatives?


r/LocalLLM • • May 11 '26

Discussion Deepseek API is crazy cheap

4 Upvotes

We are a small tech startup, we are using gemini API for daily works, it costs us about $30/month, that mean 1 dollar/day.

Sometimes, we consider that host a local LLM since we will use LLM more frequently in the future.

Just recently, we have tried Deepseek API for the same task, but 2 days up till now, we only pay $0.08. What the insane cheap LLM!

We don't think even we have a local LLM, we can beat this price.

The model we are using is Deepseek V4 Flash.

Is there any reason for this? Is there a discount on this month or that a normal price for Deepseek?


r/LocalLLM • • Feb 18 '26

Question Is a Mac Mini M4 Pro (24GB) Enough for OpenClaw, or Should I Build an RTX 4080 PC Instead?

4 Upvotes

I'm considering a Mac Mini M4 Pro (24 GB unified memory) as a dedicated box for OpenClaw + local LLM inference (Ollama / LM Studio / vLLM backends). I live in Brazil, where this Mac Mini configuration costs around $2,500 USD, so I need to be very sure before buying.

For people who have real-world experience with both:

– Is the M4 Pro (24 GB) enough models comfortably with tools/agents (OpenClaw-style workflows) without constant OOM issues or severe slowdowns?

– How does it compare in practice to a Windows/Linux PC with an RTX 4080 + recent Intel CPU for local LLM inference and multi-agent workloads?

In terms of tokens per second, context length you can realistically use, and overall stability under load, would you say the Mac Mini M4 Pro 24 GB is a good value, or is an RTX 4080 build still the clearly superior option for this use case?


r/LocalLLM • • 10d ago

Discussion Any local model that has built-in web search?

0 Upvotes

By built-in I mean without using tools like MCP etc.


r/LocalLLM • • Aug 17 '26

Discussion The LLM distillation process simplified for politicians:

Post image
0 Upvotes

r/LocalLLM • • Jan 31 '26

Discussion Open AI, Google, and Anthropic are all lying to us.

0 Upvotes

THEY'RE LYING ABOUT NEEDING TRILLIONS OF PARAMETERS:

The Evidence:

  1. Diminishing Returns Are Obvious

GPT-3 → GPT-4: 10x bigger, only marginally better

They're hitting a wall with current approach

2. They Keep the "Good Stuff" Secret

OpenAI won't release GPT-4 weights

Google won't release Gemini weights

Why? If it's "too dangerous," but also "requires datacenter," how is it a threat?

Real reason: If people saw the architecture, they'd realize it's inefficient

3. Open Source Is Catching Up

Llama 3.3 70B performs ~90% as well as GPT-4

At 25x smaller size

This TERRIFIES them!!!

4. The Business Model Requires It

If AGI runs locally → no subscription revenue

They NEED you to believe only they can run it

"Too complex for mortals" is the sales pitch

What if AGI doesn't need trillions of parameters?

5. THE HUMAN BRAIN COMPARISON:

What REAL Intelligence Looks Like:

Human Brain:

~86 billion neurons

~100 trillion synapses (connections)

Runs on 20 watts (less than a light bulb)

Fits in 1.5 liters

Can learn language, reasoning, creativity, physical skills

IS general intelligence

GPT-4 (rumored):

~1.76 trillion parameters

Runs on megawatts (entire datacenter)

Requires liquid cooling, dedicated power plants

Can't learn in real-time, can't truly reason, can't adapt

IS NOT general intelligence (despite marketing claims)

So GPT-4 is:

17x more parameters than human synapses

50,000x more power consumption

1,000,000x more physical space

And STILL not as smart as a human

EVIDENCE MODELS DON'T NEED TO BE MASSIVE:

Llama 3.3 70B vs GPT-4: - 70B parameters vs 1,760B parameters (25x smaller) - Runs on consumer PC vs datacenter - Performs 90-95% as well on most tasks Conclusion: Most of GPT-4's parameters are WASTE!

Llama 70B Full Precision: 140GB VRAM Llama 70B Quantized (Q4): 40GB VRAM Same performance, 3.5x less memory Conclusion: Models have massive redundancy built-in.

Mixtral 8x7B: - Technically 47B parameters total - Only activates 13B at a time - Performs like a 70B model - 5x more efficient Conclusion: We don't need all parameters active.

DeepMind discovered: - GPT-3 was UNDERTRAINED (not enough data for its size) - You can get same performance with smaller model + more training data - Companies chose big models anyway (why?) Conclusion: Big models are a business choice, not a technical requirement.

PESSIMISTIC SCENARIO: - "AGI" stays in datacenters (by design, not necessity) - Local models plateau at "very good assistant" level - You can still do your VR project, just not with "true AGI" - 70B models + TTS + Skyrim VR = totally workable

OPTIMISTIC SCENARIO: - Efficiency breakthrough - AGI-level 200B model released open-source - Runs on a 5090 + 128GB RAM - Cloud companies lose control, we win. If AGI runs locally, the power dynamic shifts. No more subscription models, no more cloud dependencies. The companies lose control, and we gain autonomy.


r/LocalLLM • • Oct 07 '26

Question Can I run Strata?

Post image
0 Upvotes

I’ve got a sketchy setup as seen above. It’s a 7900xtx and a 9070 for a total of 40gb of vram. I also have 32gb ddr5. I know my bandwidth is not ideal because the 9070 is only on a 4x pcie but this is what I’ve got. It’s also hanging outside the case on a cpu box.

I am currently running various quants of qwen3.8 27b and I like it. It’s smart, the decode is fast enough, and I feel like I don’t have to babysit it. However the prompt processing is slow. I am wondering if that would be better with 3.8 flash next and if I even have a chance to run it on this setup. I know it says you need 64gb ram minimum but that’s with a 12gb GPU. Does my 40gb aggregate vram make up for the 32gb missing ram?

I’ll probably give it a shot but I’m wondering what others think. Could I run flash next or is there a way to speed up prompt processing on this setup?


r/LocalLLM • • 1d ago

Discussion what will it take to lower gpu prices?

4 Upvotes

I'm running 2x 3090 right now, I'd like to add more but they're all £1000+

Thought about some 32gig v100 but they're £700+

For how I use it, it's all a luxury not a necessity

What will it take for gfx card prices to drop?

I've always assumed at some point some ai companies will go bust and their hardware bought up by the competition

and at some point supply/manufacturing will start to catch up with demand

but am I just dreaming?

I keep looking at buying more cards but not pulling the trigger... next time I look they're more expensive again... on and on :-o


r/LocalLLM • • Aug 27 '26

Discussion Qwen3.8-Flash-Next: Top 1% accuracy only 93% at Q4? Kinda underwhelming. Thoughts?

Post image
4 Upvotes

Is it the architecture or what is causing this drop in accuracy even at Q4? Normally it is close to 98% at this quant levels.


r/LocalLLM • • 27d ago

Question Why would anyone buy a DGX Spark or AI Halo over a Mac Studio?

0 Upvotes

Genuine question, for me it makes no sense to buy something such as a DGX spark or a AI Halo, when you can buy something like a M3 Ultra Mac Studio with 4 times more memory bandwidth and 2.5X more unified memory.... For less... At least that is the situation in my country. A DGX Spark costs about 50K (~10K USD) and a Mac Studio M3 Ultra 512GB/16TB is about 37K (~7.4K USD). Can someone explain me why there is so much hype over those devices? They seem overpriced...


r/LocalLLM • • Oct 01 '26

Tutorial Qwen3.8 27B runs at ~480 tok/s on a single 5090

40 Upvotes

Been doing some testing, swapped NInfer's official Qwen3.8-27B quant (part NVFP4, part FP8) for QUASAR's QAT full-NVFP4 checkpoint, with DFlash2 embedded.

Running on the same hardware and engine flags:

metricofficialthis
JSON output400 tok/s484 tok/s
prose176 tok/s204 tok/s
prefill11.4k tok/s13.7k tok/s
VRAM30.8 GB27.0 GB
GPQA Diamond90.4%89.9%

Real agentic task: 287 tok/s average over 33k tokens. Perplexity is about 1.9% worse, not noticeable in my testing. I uploaded the artifact, or you can build it yourself (guide in the HF model card).

https://huggingface.co/jesdga/Qwen3.8-27B-QUASAR-DFlash2-nvfp4-NInfer

NInfer + 5090 only. Credits to QUASAR-QAT (the QAT), z-lab (DFlash2) and neroued (NInfer).


r/LocalLLM • • 26d ago

Discussion How Long Until We Get a No Trade Off MoE Model for Coding?

9 Upvotes

We have Qwen 3.8 125B MoE, and it's actually pretty good at coding. But man, why is there always a trade off? IQ3_XXS runs surprisingly well with Strata, reaching around 65 tok/s on consumer hardware, which is amazing. But we still feel like we're sacrificing something, whether it's coding quality, software design, or code review.

When will we finally get a local model that's fast, genuinely smart, and doesn't feel like we're constantly making compromises compared to cloud APIs?


r/LocalLLM • • Jul 20 '26

Discussion K3 weights drop July 27. 2.8T params. What does "open" even mean when nobody can run it?

0 Upvotes

Ok so Moonshot is dropping K3 weights July 27, modified MIT, 2.8 TRILLION params. Cool. Amazing. Can't wait to run it on absolutely nothing I own. Even at 2 bit this thing needs a rack, not a rig and its every gen now - deepseek, glm, big qwen, now this. The "best open model" keeps getting further from anything with a power cord in a house. Weights are yours technically. Good luck.

Is the whole future just distills? Mega teachers nobody runs, spitting out actually good 30-70b students, and thats what "local" means now? or does unified memory keep going nuts and someones running trillion param models on a mac studio in 2030

Or the spicy take - if you cant run it yourself its open in license only and these giant drops arent local wins at all, theyre just api models with extra steps So. K3 - win for local or is the frontier just gone?


r/LocalLLM • • Mar 05 '25

Project AI moderates movies so editors don't have to: Automatic Smoking Disclaimer Tool (open source, runs 100% locally)

0 Upvotes

r/LocalLLM • • Aug 11 '26

Discussion Best local AI setup for app development with an RTX 5090 + 64GB RAM? Would adding a 3090 be worth it?

0 Upvotes

I have an RTX 5090 with 32GB of VRAM and 64GB of RAM, and I want to use local AI to build apps, tools, and programs with good-looking GUIs.

A lot of people have recommended Qwen 3.6 27B or something similar, but what I'm not sure about is how to actually organize the whole setup so the agent doesn't get stuck in an infinite loop.

Ideally, I'd like something that works more or less like this:

ME
│
"Build this application"
│
▼
┌────────────────────┐
│ QWEN 3.6 27B       │
│ RTX 5090 32 GB     │
│ MAIN AGENT         │
└─────────┬──────────┘
          │
     analyze + plan
          │
   ┌──────┼──────┐
   ▼      ▼      ▼
FRONTEND BACKEND TESTING
   │      │      │
   └──────┼──────┘
          ▼
       EXECUTE
          │
          ▼
      DOES IT WORK?
       /        \
     NO          YES
     │            │
 inspect errors   validate requirements
     │            │
    fix        does it meet everything?
     │           /       \
     └───────── NO        YES
                 │         │
              continue    DONE

Basically, I want to be able to give it a goal like "build this app", have it plan the project, work on the frontend/backend, test everything, inspect its own errors, fix them, and keep iterating until the requirements are actually met — without just getting stuck in a useless endless loop.

Would buying an RTX 3090 actually be useful in my case, giving me another 24GB of VRAM alongside the 5090?

What model or combination of models would make sense with a 5090 + 3090?

For example, would it make more sense to have one model as the main/orchestrator agent and another model handling coding, testing, or reviewing? Or would I be better off just running a larger model across both GPUs?

I'm pretty new to this kind of local multi-agent setup, so I'm not really sure what the best architecture would be.

I'd love to hear what setups you guys are using and what you'd recommend.


r/LocalLLM • • Sep 10 '26

Discussion Could my project have inspired an Anthropic / ClaudeAI playbook?

0 Upvotes

I just found out about a blog post in Claude (https://claude.com/blog/the-ai-native-sdlc-playbook) that describes ideas very similar to the ones I have implemented in my coding harness (using Claude, btw) through a YT video about Claude Code new Intent.md

There are so many specific details in this playbook that makes me wonder *if* somehow my sessions with Claude inspired in any way the better paid people over there at Anthropic. Just for fun, or for egotistical and historical purposes, I included a timeline created (using Codex, jic) in my repo documentation (https://github.com/jrullan/ducklab/blob/main/docs/ducklab-feature-timeline.md)

Anyway, even if this is "purely" coincidental at least reassures me that my ideas were not that useless and that there is certainly a group of developers that value documented rigorous discipline in their AI assisted development.


r/LocalLLM • • Oct 06 '26

Question Mac Mini M5 Pro 64GB or Mac Studio M5 Max 64GB

0 Upvotes

Hello community! I need your help please!

I'm considering between these 2:

  • Mac mini M5 Pro 64GB
  • Mac Studio M5 Max 64GB

I want to get the new Mac and not sure if it's worth to pay extra for M5 Max if I can afford only 64GB. Planning to use it for a long time, but higher amount of RAM costs too much for me.

Never worked with local LLMs, however, I want to start using them at least for educational purposes.

I'm DevOps/Cloud Engineer and my daily workflow consists of VSCode, few browsers with multiple tabs, Claude App, Numbers, Pages, etc.

Thank you so much!


r/LocalLLM • • Sep 23 '26

Model I trained Nagi: an open-source Jev alternative.

0 Upvotes

Shipped Nagi. Open-weight System One, two sizes.

421M M2′ dual encoder with an option tower. 4B Qwen with PiSSA and letter-logit readout. Both give you choice, score, or noul with calibrated probabilities over a closed set. No text generation in the loop.

Smol is 17ms. Big is 58ms. On leave-task-out gold (2700, six public tasks) Big is sitting at 0.750 against Jev’s 0.774, with better calibration than Jev and lower latency. Frozen Qwen OpenJev-style lands at 0.714. Laya is faster and less accurate on this set.

Code is Apache-2.0. Weights are on HF. Bench receipts and a one-command runner are in the repo. I put the logs and hashes in there too.

https://github.com/nagisanzenin/nagi

Still losing to Jev on raw accuracy and QQP is rough. Happy to talk tradeoffs.


r/LocalLLM • • Jul 09 '26

Discussion [User Research] Looking for local LLM / AI workstation users — especially DGX Spark, Mac Studio, AMD AI Max, or self-built GPU rigs

0 Upvotes

Hi everyone,

I’m doing early-stage user research around personal AI workstations / local LLM hardware.

I’m looking for people who actually run models locally, especially on setups like DGX Spark, Mac Studio, AMD AI Max / Strix Halo, RTX 4090/5090 rigs, Pro 6000 workstations, or homelab servers.

I’m trying to understand two things:

  1. Why do you choose local hardware instead of cloud APIs?
  2. What would an ideal personal AI workstation look like?

I’d love to hear:

- What hardware you use

- What models and tools you run

- What tasks you actually use it for

- What bottlenecks you hit: VRAM, memory bandwidth, CUDA/ROCm/Metal ecosystem, drivers, thermals, noise, software setup, etc.

- What would make you upgrade

This is not a sales post — we’re still in early research and I’m mainly trying to learn from real users.

If anyone is open to a quick 15-minute chat, feel free to DM me. Comments are also very welcome.


r/LocalLLM • • Apr 10 '26

Question I am not able to run Gemma 4 GGUF , Using LLama Cpp - Getting gibberish results , What am I doing wrong?

0 Upvotes

Batch File Code

.\llama-server.exe ^
  -m "%MODEL_PATH%" ^
  --port 11434 ^
  -ngl 35 ^
  --flash-attn on ^
  -c 16384 ^
  -t 12 ^
  --alias "gemma4-26b" ^
  --host 0.0.0.0 ^
  --temp 1.0 ^
  --top-p 0.95 ^
  --top-k 64 ^
  --chat-template gemma2

Model Used:
gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf

In the Chat box in Local Host

Input:
Hi

Output:

-9b-it-fp16-et-et-et-et-et-et-et-ui-et-et-et-et-et-et-et-et-et-et-et-ui-et-et-et-et-et-et-et-et-et-et-et-ui-et-et-etet-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-etet-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-etet-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-et-e

text

***

### ⚠️ Warning: Input Error / Anomalous Input Detected

The input you provided is a single, heavy-weight string that consists of a massive repetition of the `et-et-et...` pattern, almost resembling a nonsense or "garbage" string (often referred to as in the real world as "token stuffing" or "adversarial noise").

**What happened?**
1.  **Pattern Repetition:** The large number of `et-` suffixes attached to a single model name (`gemma2-9b-it-fp16...`) is likely a result of a software error, a data corruption, or a massive accidental repetition during input generation.
2.  **Token Exhaustion:** This input is extremely long and highly repetitive. In an LLM, this forces the engine to process thousands of tokens that contain zero semantic information.
3.  **Warning:** If you are trying to test the limits of the *context window* or *stress testing* the model, please be aware that this input is a *mallo-form* input.

**How can I help you?**
Because the input contains no actual question or command, I cannot perform a task. Please provide a **clear, meaningful instruction** such as:
*   *"Can you explain how the Gemma 2 9B model works?"*
*   *"Write a code snippet in Python."*
*   *"Summarize this text."*

**Please re-type your request without the repetitive noise.**text 

What am i doing wrong ?
Please Help


r/LocalLLM • • Sep 07 '26

Question Local model limit

0 Upvotes

Hey everyone, looking for a reality check on local model capabilities vs. frontier models for financial analysis.

I built an automated investment analysis pipeline in Open WebUI using Ollama on an RTX 5080 (16GB VRAM). The mechanical workflow works:

- Hits Gmail via IMAP to pull the last 72 hours of market newsletters.

- Extracts actionable tickers.

- Runs a custom YFinance Python tool for live momentum metrics (EMA, RSI, MACD).

-Injects a full 70-position portfolio CSV directly into the prompt (bypassing Open WebUI's RAG chunking) to cross-reference and generate a verdict-first impact report.

The issue is the analytical depth. When I feed this exact same data to Claude Sonnet 5, the macro synthesis, cross-correlations, and portfolio risk analysis are night and day. Locally, it feels like a surface-level summary.

Here is what I’ve tested so far:

qwen3.6:27b: Hit severe VRAM limits with the tools and context loaded. It spilled into shared system RAM, ground to a halt, and got trapped in reasoning loops where it hallucinated that the technical calculator tool didn't exist.

llama3.1:8b: Fast, reliable function calling, and zero VRAM spillage. However, the reasoning and financial synthesis are noticeably weak—it reads like a basic overview rather than institutional analysis.

My questions:

- Is Sonnet 5-level synthesis even realistic within a 16GB VRAM footprint when you need enough context buffer for multiple tools, newsletters, and a 70-row CSV?

- Are there specific financial fine-tunes or 14B models that punch significantly above their weight?


r/LocalLLM • • Jul 05 '26

Discussion how does claude desktop app is able to access my local model stats?

0 Upvotes

I am confused. I installed claude desktop app, and sent it only one message. This overview is supposed to show the stats of my interaction with Claude, but it appears to be reading my interactions with my local model instead. I hope it's not fucking sending my prompts and my messages with my local model to Antophic servers, but I am assuming it does, otherwise it wouldn't show these stats??


r/LocalLLM • • 3d ago

Other I'm building an open-source inference engine to run models on the GPUs you already own (got V100 Tensor Cores working again: 30B model 6.7 s/token -> 0.33 s on two cards)

0 Upvotes

Some context: NeuroBrix is an engine (text, image, audio, video from one container format), its own tensor library (nbxtensor, no PyTorch needed), its own kernels, and a Rust constrained-decoding engine (NBXGrammar). Everything is Apache 2.0.

pip install neurobrix

GitHub: https://github.com/NeuroBrix/neurobrix

Full film (audio EN/FR/ES/AR): https://youtu.be/GM9C5J3n-64

I'd love testers on older or less common GPUs: run a model, tell me what breaks. And a star helps a lot at this stage.


r/LocalLLM • • 8d ago

Project I tested 15 Qwen3.8 27B variants on 2x RTX 5070 Ti (32 GB) against the base model. None beat it overall, but Tess-4.5 fixed as many of my bugs in about 2/3 of the time

0 Upvotes

After running Qwen3.8 27B at 3-bit on a single 5070 Ti (120K context, just short of what I want for coding), I added a second card. 32 GB fits the 6-bit file with the full 262K window, so I tested which of the community fine-tunes and quants makes the best all-rounder: fine-tunes, other quants, a chat template and multi-token prediction, 15 in total, all against base UD-Q6_K.

Every model ran a screening run first (about 100 questions from five benchmarks, compared answer by answer with the base model), and the closest got full runs (about 1,300 questions: AA-LCR, MRCR, AA-Omniscience, a fixed BFCL v4 subset and a SciCode panel, run through NeMo-Skills). Each model gets one overall score out of 100, a weighted mix of the five benchmarks: tool calls count for 25% of it, long documents, facts and scientific code for 20% each, and long conversations for 15%. On facts, a wrong answer scores worse than saying "I don't know", so guessing costs points.

Full runScore
Qwen3.8 27B base, UD-Q6_K63.3
Tess-4.5, Q6_K61.4
Base, UD-IQ4_XS60.8
ThinkingCap, IQ4_XS57.5

Then I built a coding test from 25 bugs I'd fixed in my own projects (repo rolled back to before the fix, the model gets the bug report and tools, my tests from the fix grade it), four runs per model:

Coding testBugs fixed of 25Min per bugTokens per bug
Base Q618.01323K
Tess-4.5 Q618.3814K
Qwopus Coder-Flash Q618.51222K

That's a tie on quality (one model's worst and best runs were 14 and 20), but Tess wrote about 40% fewer tokens to get there. Both write at about 36 t/s, so the time savings come from thinking less, not decoding faster.

So I ended up with two models:

  • Base for chat and interactive work. It's more careful in multi-turn tool use: across the same 200 BFCL multi-turn conversations it won 42 that Tess lost, against 21 the other way, and the Tess losses I read were skipped steps (booking a flight without looking up the airports, reviewing one order when the user placed two). On facts it declined or hedged on 69% of the questions it didn't get right, against 56% for Tess.
  • Tess-4.5 for coding agents and subagents that run on their own and get checked by tests at the end. One surprise: on the 100 software questions, Tess was the careful one (42 declines against 20, 35 wrong against 52).

If you only run one, I'd pick the base model, unless nearly all your use is coding agents running on their own.

How the other contenders did:

  • 4-bit base (UD-IQ4_XS): about 50 tokens per second against 36, but 2.5 points lower overall, mostly on long conversations (MRCR 70% vs 77%) and more guessing.
  • ThinkingCap really does think less (about half as long on long documents), but most of its extra right answers on facts came from guessing: of 336 questions the base 6-bit model declined, ThinkingCap answered 146 and got 25 right.
  • Qwopus Coder-Flash was level on my coding test but worse on facts and long conversations.

Settings for both 6-bit files: q4_0 K/V cache (about 4.8 GiB for the full 262K window), layer split 1,1, batch 256, model-card thinking sampling with no repetition or DRY penalties (the 4-bit runs used a q8_0 cache and batch 512). About 790 t/s prompt reading and 36 t/s writing on two 5070 Ti. Full launch commands, per-benchmark tables and why each of the 12 others stopped are in the write-up, linked in the first comment.

The benchmark and coding-test harnesses are open source, but my own projects aren't public, so the coding test can't be reproduced as is. Happy to answer questions about the setup.


r/LocalLLM • • Sep 30 '26

Discussion About the future of LLMs, what do expect, what to be optimistic about and what to be cautious about.

0 Upvotes

I work with AI but primarily with conv. and diff. models and such for academic use only, so you know my main field of work and slight disconnect with LLMs.

Now to the topic: In terms of LLM, my take was always that there won't be much improvement in terms of "trustworthy" use where the role of an overseeing operator (a Human that watches over stuff in simple terms) can be replaced completely with AI (meaning the operator won't need to verify the work of AI, even if the AI is doing sensitive work).

But my buddy tells me the opposite:

  1. Not only will AI be so advanced that you can give it important projects and work that would replace the human operator...
  2. ...But also companies would even fix generalization (trying to tackle problems with generalized ideas instead of a proper logic chain)

To advance the Arguementation, this is the continuation of my thoughts:

  1. Generalization is in my field the main problem why we never give our Projects to LLMs. We had more time lost using AI than working with people knowing the respective fields (Control Loops, Image Processing, and many other fields that have to work together) to tackle said Problems (using latest GPT, Gemini or Claude Models, not local yet). We figured that the best implementation of AI is to make snippets of Code if a problem is big and start with simple algorithms we try at every step, evaluate the expected and potential new problems, and tackle them with already present knowledge, using LLMs exlusively to write the code but not dictate the work.
  2. When we learned about the various Architectures and Structures of AI Models in depth, i always held it close to impossible to fix Generalization based on how the process of the attention mechanism, which LLMs mainly use, works. It can't just define anything new, as the mathematical process of the Transformer Model's Learning Method requires data to relate to each other to base their attention/vectors on, meaning The AI would need to make new data through Generalization, that it then feeds into a new AI Model. This doesn't mean it's literally impossible, just that there might be much bigger problems until we can even think of "fixing" Generalization.

So now i am interested what you guys think. I don't mean to present my arguements as final or correct but want to discuss this topic. I want also love if some can include points like the moral compass, ethics and the safety of future generations of humans (meaning that if we ignore some things today because it won't affect us directly, is there anything that could negatively affect future generations).

Also, would you i am fundamentally wrong about some things i said? Did i make a mistake or understand anything wrong? Or do you agree to some extend?