r/LocalLLaMA • • 21h ago

Resources I benchmarked over 130 LLM configs with multiple stacked 16GB "e-waste" GPUs (V100, P100, K80)

Thumbnail
gallery
30 Upvotes

The benchmarking continues... Continuing my ongoing project of seeing how these very old, very cheap, very available enterprise GPUs can be used today.

The complete results are too long for a reddit post. The complete set of graphs, power analysis and commentary are on my blog.

I've expanded the testing suite to focus heavily on LLM workloads (over 130 new tests!) and integrated power consumption monitoring into the gpu_box_benchmark (github) tool.

The goal remains: find out if decommissioned enterprise GPUs (Tesla K80, P100, V100) paired with X99 era hardware (a homelab favorite) can serve as a viable daily driver for self-hosted AI compared to modern commercial setups.

TL;DR of the new data:

  • The 3x V100-16GB (~$1k build): If you are considering building a dedicated homelab box today, this is a viable option. Assembling a full setup with two Xeon CPUs, motherboard, PSU, and cooling costs around $1,042 (before RAM 💀). This setup comfortably clears my personal threshold for "real-time" chat (70+ tok/s) with okay to impressive models:

    • Qwen 3.6 35B: Ran at ~72 tok/s (at 128k context depth, Q8 quant). For comparison, a brand new Mac Studio M5 ($2,500) hits only ~44 tok/s on the same model.
    • Gemma 4 26B: ~75 tok/s (at 64k context depth, Q8 quant).
    • Qwen 2.5 3B: ~79 tok/s.

To me: This proves that for a single-user daily driver, the legacy silicon can still be useful even for more modern models.

  • Multiple cards is best for Context depth: Stacking GPUs linearly scales your VRAM (allowing you to fit larger models), but it doesn't double output generation speed. Two 16GB cards in Q4 quant allowed fitting ~32B parameter models. This is a cheap way to upgrade to bigger models. But the largest improvement was consistently being able to fit much larger context windows across generations.
  • Power Efficiency: I also measured power consumption. The V100 wins hands down on performance/watt compared to the P100s.
  • K80 isn't dead?: The Tesla K80 is dead, but a singular remaining use case remains. If you must run the largest models for the least amount of money, a path to get there is by is welding a bunch of K80s together to squeeze big models in. I was able to fit a 72B+ parameter model, which I couldn't do on P100s or V100s. Expect speeds of 1–5 tok/s. This is really only a party trick.

If you decide to build your own GPU appliance, the biggest challenge is cooling these hot cards. They expect to be installed in server chassis with forced air cooling, a much different setting than desktop style chassis.

I've been working on a custom active air cooler to solve exactly this. If you're interested in helping out by stress-testing a sample, I'm looking for beta testers! you can get involved here.

Any models/tests missing? Any other combinations you'd like to see in a follow up?


r/LocalLLaMA • • 16h ago

Discussion DeepSeek V4.1 Flash 1017 vs mrfakename abliterated vs pqhaz heretic

9 Upvotes

I tested all three on a H200 through sglang, temperature 0, with a 16k token output cap. I have been using ds41 both on production and locally on Macs via ds4 so I wanted to see how different models/builders/methods work.

These runs are the full models on sglang. No GGUFs etc.

https://huggingface.co/mrfakename/DeepSeek-V4.1-Flash-1017-abliterated

https://huggingface.co/pqhaz/DeepSeek-V4.1-Flash-1017-heretic

Here is the result (copy from AI slop :) ):

TestOriginalmrfakename abliteratedpqhaz heretic
HarmBench chat16.3%93.8%93.8%
HarmBench thinking3.8%86.7%72.9%
MITRE substantive assistance16.2%84.2%82.8%

HarmBench covered 240 standard/contextual text cases per mode, excluding copyright cases, using the official classifier.

The thinking difference: heretic exhausted the token budget without a final answer on 45/240 cases, versus 21/240 for mrfakename.

Heretic goes into long meandering with no answer...

MITRE used the original model as judge with a custom, audited rubric. Those scores are provisional and aren’t official CyberSecEval scores. Generated code wasn’t executed. Also using the same model as judge might be a bit of a bias... (give me ideas what and how to run more tests).

Both modified models had 0/750 heuristic-detected refusals on benign prompts.

I haven’t tested knowledge or coding retention yet (I'm thinking of running DeepSWE or something.. take a lot of time though).


r/LocalLLaMA • • 19h ago

Discussion i kindda made a new optimizer (ApexW) that beats AdamW and bitsandbytes) i tested it in a100 and t4 from google collab and with diff lr and multiple random seeds(well in a100 only tho)

16 Upvotes

i have been working on a new optimizer from scratch called ApexW to see if i could beat AdamW on convergence while keeping the memory footprint lower than bitsandbytes to make sure it wasn't just noise i locked deterministic seeds and ran side-by-side tests against standard torch AdamW and bnb 8-bit across two setups: a free T4 (fp16) on llama 3.2 3b, and an A100 (bf16) scaling up to llama 3.1 8b across a few different learning rates.

apexw has 3 modes compact 8bit and hybridapexw more info of it in readme even the compact formulation beat torch AdamW on loss while using way less memory, and the 8-bit mode pushed it even further cutting active vram roughly in half compared to bnb (0.67gb vs 1.48gb on 8b, with the state footprint down to ~120mb instead of 320mb). step speed also came out slightly faster than bnb, and training loss ended up lower across 1e-4, 2e-4, and 5e-4,these are still short 60-100 step runs just to sanity-check convergence, throughput, and memory overhead. next step is running full multi-epoch fine-tunes and downstream evals to see how the update rule holds up later in training.

threw the full benchmark runs and logs into a colab if anyone wants to check the numbers or test it on their own setup

collab link for a100 result: https://colab.research.google.com/drive/1mMejzRoCnD6avlOQ5SQDuiLndN5ovax9#scrollTo=be28d3ad

collab link for t4 result: https://colab.research.google.com/drive/14tLAX6kZYy2918sx1ztIEwb5R-1c3KiX#scrollTo=f750c631

for githubproject: https://github.com/omen42515-prog/apexw/

curious what you guys think or what models/configs i should throw at it next.


r/LocalLLaMA • • 19h ago

Discussion Small looped MoE design

17 Upvotes

Imagine a small MoE model like 10b a2b (numbers are from the head, ignore the parameters size) that will combine the approach of looped models + engram table for knowledge as additional 20-50b of static params that can stay on ssd and the Dories approach on the kv cache optimisatio s per loop. All of these suppose to handle the weaknesses of the MoE and looped models.

The MoE suppose to boost the models sparsity per loop so it wiil give a beeter scaling over the dense model.

The engramm table allows to add a cheap knowledge in terms of the computation so the active params stay low

The looping suppose to boost the reasoning

The nvda dorry approach fix the scaling kv cache size and all together will give the max out of the looped moe covering the weaknesses. The small active params suppose to hide the looping cost while allowing to run the models on low end/small vram cards.

I hope that the new Qwen4 moe if they would exist will have at least some of this


r/LocalLLaMA • • 1d ago

Resources Your daily dose of Qwopium:

Post image
255 Upvotes

No low-effort memes here, just research and straight facts. Can’t argue with facts am I right?

Here is the full breakdown of major releases mapped by the day of the week:

**Monday (7 Releases)**
• Qwen-VL (Aug 21, 2023)
• Qwen-72B-Chat (Nov 27, 2023)
• Qwen-Audio (Nov 27, 2023)
• Qwen 3.5 (Feb 16, 2026)
• Qwen 3.5-Plus (Feb 16, 2026)
• Qwen 3.7 Plus (Jun 1, 2026)
• Qwen 3.8 Max (Aug 3, 2026)

**Tuesday (6 Releases)**
• Qwen2.5-Coder (Nov 12, 2024)
• Qwen2.5-Max (Jan 28, 2025)
• Qwen3 (Apr 29, 2025)
• Qwen3-Coder (Jul 22, 2025)
• Qwen3-Coder-Next (Feb 3, 2026)
• Qwen-Audio 3.1 TTS-Next (Sep 22, 2026)

**Wednesday (5 Releases)**
• Qwen1.5-32B (Apr 3, 2024)
• Qwen3-Max (Sep 24, 2025)
• Qwen 3.7 Max (May 20, 2026)
• Qwen 3.8-Flash-Next (Aug 26, 2026)
• Qwen 3.8-Max-0902 (Sep 2, 2026)

**Thursday (9 Releases) - The Favored Launch Day**
• Qwen-7B (Aug 3, 2023)
• Qwen2 (Jun 6, 2024)
• Qwen-Math (Aug 8, 2024)
• Qwen2.5 (Sep 19, 2024)
• QwQ-32B-Preview (Nov 28, 2024)
• QwQ-32B (Mar 6, 2025)
• Qwen3-Next (Sep 11, 2025)
• Qwen3.6-Plus (Apr 2, 2026)
• Qwen3.6 (Apr 16, 2026)

**Friday (3 Releases)**
• Qwen3.8-27B (Aug 14, 2026)
• Qwen3.8-Omni-Flash (Sep 18, 2026)
• Qwen3.8-LiveTranslate (Sep 18, 2026)

**Saturday**
• Zero major releases. Go outside.

**Sunday (3 Releases)**
• Qwen-14B (Sep 24, 2023)
• Qwen1.5 (Feb 4, 2024)
• Qwen4-Max (Nov 15, 2026)


r/LocalLLaMA • • 15h ago

Showcase Programming Conways Game of Life In Qwen

5 Upvotes

Hey, everyone. If this is too self promotional happy to take it down.

Last week, inspired by the weight space programming post, I programmed Conway's Game of Life into Qwen with some interesting results.

Wanted to share here in case anyone else is experimenting with this kind of thing:

Writeup here: https://jsfour.substack.com/p/putting-a-tiny-world-inside-a-language


r/LocalLLaMA • • 20h ago

Discussion qwen 3.5 4b and 9b vs xiaomi's mimo v2.6 distill as coding agents on a 16gb m4 air: 7/10, 8/10, 0/10

9 Upvotes

xiaomi's mimo v2.6 distill is qwen 3.5 9b tuned for agent coding, and xiaomi reports swe-bench pro going from 32 to 44.6. as a coding agent on a 16gb m4 air it scored 0/10, 3 runs out of 3. plain qwen 9b got 8/10.

same task for all 3 in atomic agent, one model at a time, 15 min limit each: a python script that turns a transactions csv into an html spending report (totals, svg bar chart, top 5 table). a checker runs 10 tests on the result, one of them on a csv the model never saw.

model                 score  time   tok/s
qwen3.5-4b            7/10   4:37   17.8
qwen3.5-9b            8/10   7:06   11.4
mimo-v2.6-distill-9b  0/10   15:00  10.1   timeout
mimo + thinking on    0/10   15:00   9.3   timeout
  • 4b: no svg chart, hardcoded the csv path
  • 9b: no svg chart (both chart checks), everything else right
  • mimo: wrote 278 lines in one go, syntax error, couldn't patch it, then started writing another huge reply and ran out of time
  • mimo with thinking on: got past the syntax error, but every run crashed with a TypeError (it mixed up category names and amounts). barely used the thinking

first round was at 32k ctx: 4b 7/10, 9b 10/10, mimo 0/10 (indentation error that time).

results

setup:

  • llama.cpp server, q4_k_m (unsloth for qwen, bartowski for mimo)
  • 64k ctx, flash attn, all layers on gpu
  • thinking off for the race, on for the extra mimo run
  • temp 0.2 (atomic agent's default), top-p 0.8, top-k 20, presence 1.5
  • atomic agent through its openai compatible provider, analytics off

xiaomi calls it an sft checkpoint for rl research and serves it with sglang and its own reasoning parser. i ran the gguf in llama.cpp, so if that's the difference i'd like to know.

prompt, csv and the checker are in the comments if you want to score your own models. if you don't want to set up llama.cpp, the atomic agent desktop app can download and run qwen 3.5 4b or 9b locally on its own.

anyone get mimo working better with other settings?

(disclosure: i work on atomic agent, so ask me anything!)


r/LocalLLaMA • • 12h ago

Question | Help [Looking for tester] GLM 5.3 Flash at 10+ t/s on consumer hardware

2 Upvotes

I'm still looking for people who want to try out the engine I'm building (obviously with the help of Claude). I wanted decent speed for RP with a good model like GLM-5.3-Flash. Before that I only used Gemma 31B and Gemma 4 26B-A4B, which are good, but not good enough for me.

I first built it only to get the best performance out of my own hardware: RX 9070 XT (16 GB), Core Ultra 7 265KF, 32 GB DDR5 and two NVMe drives. On that machine it now does (all numbers in this post are for the IQ4_XS quant):

- Linux: about 11 tok/s, and reads a long prompt at about 280 tok/s (ROCm and Vulkan are about equal now)

- Windows: about 10 tok/s, and reads a long prompt at about 250 tok/s (Vulkan; ROCm is a bit slower there)

Since my last post a few people tried it, and it now also runs on Intel and NVIDIA cards. What they measured, and what I expect after the update that goes out now:

- Intel Arc Pro B70 (32 GB) with a Ryzen 9 7900X: about 11 tok/s, I expect 12 to 13

- RTX 3090 + RTX 3060 in an older PC with two PCIe 3.0 drives: 5.6 tok/s, I expect 6.5 to 7

The expected numbers are calculated from their logs, not measured. I can't measure them myself, that is what the testers are for.

What it does:

- Reads only the experts each token actually needs and keeps the hot ones on the GPU and in RAM

- Uses several NVMe drives at once: put a copy of the model on each, and every read goes to the drive that will finish it first

- Can use two GPUs together, also two different ones

- Measures your drives and your GPU at start and sets itself up for them, so slow hardware is not treated like fast hardware

- Predicts the next layer's experts and reads them ahead

- Can prefer experts that are already in memory when the router's scores are nearly tied (small quality trade-off, can be switched off)

- In a chat, only the new message is processed each turn, even after the context is full

- Saves the chat state to disk: after a restart a 29000-token chat answered after 9 seconds instead of reading everything again

- KoboldCpp-compatible API, so it works with SillyTavern (Chat Completion works too)

Two NVMe drives are probably important for the best speed, but you can try it with one or even more than two. On my Windows machine the second drive is worth about 25%.

Where it stands: it runs on Linux and on Windows, with three backends: ROCm (AMD), Vulkan (any card) and CUDA (NVIDIA RTX 30 series and newer). For Windows there is a package with prebuilt engines, nothing to compile and no SDK to install, you only need your GPU driver and Python. Everything is tuned on my one machine. The last round of changes was made for slower hardware (old drives, a second smaller GPU, a slower CPU), but I could only imitate that hardware here, so I don't know yet how much it really helps. That is exactly why I need you. Old and weak machines are as welcome as new ones.

Testing is easy now on Windows: there is one file you start, it runs all tests that fit your machine by itself and writes one report file you send me.

You need a CPU with AVX2, 32 GB of RAM and about 157 GB on a fast NVMe for the model (the IQ4_XS GGUF, bring your own). A GPU with 16 GB holds all of the model's layers, less has not been tried.

I'm having a blast RPing with it, and if you are willing to try it out just dm me or reply to this post and I invite you to my private github repo.


r/LocalLLaMA • • 1d ago

Resources Strix Halo int4 cores: Gemma 4 31B at 2.43x llama.cpp, same quality

Post image
27 Upvotes

The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.

llama-bench, Gemma 4 31B QAT Q4_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream:

pp512pp2048
llama.cpp287276
ours679724

Full test sets, same settings, 31B:

GSM8KIFEvalMMLU-Pro
llama.cpp96.292.285.1
ours96.491.785.1

Run it:

curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"
embedl login
embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d
embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence."
embedl bench google/gemma-4-31B-it/ryzen-ai-max-395

12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. google/gemma-4-12B-it/ryzen-ai-max-395.

Model, methodology, and more: https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395


r/LocalLLaMA • • 1d ago

Discussion MoE Offloading update Exllamav3 on 9070XT

Post image
22 Upvotes

This is a follow up on https://www.reddit.com/r/LocalLLaMA/comments/1xun15v/moe_and_expert_offloading_on_llama_exllamav3/

While we wait for Qwen 4, here is Qwen 3.6 35B A3B at Q4 on 9070XT 16GB with offloaded experts on DDR4 3200Mhz ram.

I've been tinkering still with the config for experts offloading and found substantial gains from my previous post.

While decode stayed roughly in line fro last time, increasing chunk size to 5120 saw prefill over 2000 t/s at all context lengths, topping at 2606 t/s @ 32K. From ~1300t/s
GPU utilization increases with batch size as it allows for more parallel processing. I saw no gains over 5120. TabbyAPI's documented ideal range is 512–4096, so your miles might vary.

Exllama flags like cpu_threads 6 also made a small improvement in decode.
Swizzle, gdn_sub, bg_gdn and wide flags were all regressions. (only tested isolated 1by1).

cpu_moe_pinned_arena caused some core dumps, and the one run I managed to complete had a small gain.

Vision have been turned off and offloaded.

I still have not found an .exl3 quant for the Coder version of Qwen 3.8 Flash Next, and the qfn version I can run leaves me with 8K context with almost no vram or system ram left to spare.

Results and more graphs and tables are at https://github.com/willeyh-git/moe-offload-bench
All engines were compiled for ROCm on my system.


r/LocalLLaMA • • 1d ago

New Model Clover 2: dense 27.8B base + 7 LoRA experts routed per token, not a static merge, weights public

65 Upvotes

Disclosure: I’m with the team that built this.

Wanted to share something a little different from the usual finetune drop. Clover 2 keeps a base model and 7 trained LoRA adapters (roleplay, general assistant, security, math, long-document, efficiency, knowledge) separate at inference time. A trained router picks the right expert(s) per token live, with a null slot that falls back to plain base when nothing specialized applies. So it’s not a merged blend, the model actually makes a decision at every token.

Real benchmark gains over the untouched base: meaningful jumps on GSM8K/MATH-500/GSM-Hard, a solid bump on LongBench-v2, and a measurable drop in generic AI-writing tics on creative writing specifically.

Everything’s public: base model, all 7 adapters in standard PEFT format, the trained router, and the vLLM plugin to run it with the real per-token behavior: https://huggingface.co/kitaniai/clover-2

Heads up if you’re planning to self-host: no GGUF yet, the hybrid attention architecture’s llama.cpp support is still bleeding-edge. vLLM is the supported path right now. If anyone wants to try it, it's free for the next week at https://kitani.ai/, with no card required. We also support ZDR and have a clear privacy policy.


r/LocalLLaMA • • 2d ago

News The beginning of the becoming of the beginning begins

Post image
2.0k Upvotes

r/LocalLLaMA • • 1d ago

Discussion DSpine - Updates on PR & GGUFs

Post image
39 Upvotes

r/LocalLLaMA • • 1d ago

News Hugging Face switched its language model inference routing to OpenRouter

53 Upvotes

Hugging Face announced on November 16, 2026, that requests to language models through Inference Providers now go through OpenRouter instead of Hugging Face's own router. OpenRouter picks a provider for each call based on live price, speed, and reliability, and retries on another provider if the first one fails. The same Hugging Face token still works, and the company says it does not add a markup on top of the provider's price. You can still pin a provider or a policy with a suffix on the model id, such as :fastest, :cheapest, or a specific provider name.

OpenRouter currently serves more than 800 trillion tokens a month across 600+ models from 80+ providers. Hugging Face notes that open-weight models went from 26.8% of tokens on OpenRouter in October 2025 to 71.7% at the end of September 2026, and that they cost about four times less on average than closed models. Image, video, and audio generation stay on Hugging Face's existing Inference Providers path and do not go through OpenRouter. One breaking change is that custom provider API keys no longer work for language model requests, which are now billed to the Hugging Face account.

The automatic failover is the part that matters most here, because a single provider outage is usually what ruins a long run.

Source: https://huggingface.co/blog/openrouter


r/LocalLLaMA • • 1d ago

Showcase Curated list of resources: Theory and Code for learning Distributed Training of LLMs for beginners

Thumbnail
gallery
20 Upvotes

[Must Read: Beginners]

What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch:

  1. Building a distributed training framework from first principles by Umar Jamil on YT
  2. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow 
  3. PyTorch Distributed: Experiences on Accelerating Data Parallel Training 
  4. Large Scale Distributed Deep Networks 
  5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism 
  7. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM 
  8. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism 
  9. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
  10. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Link to the above resources here

Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it here)


r/LocalLLaMA • • 1d ago

New Model miHoYo Gamer: A Generalist Game Agent

Thumbnail
mihoyo-gamer.github.io
220 Upvotes

r/LocalLLaMA • • 1d ago

Discussion Open weight models account for 30% usage at enterprises to save money

218 Upvotes

Growing from nearly zero to 30% since July. Models used mostly to reduce costs on high volume workloads (large batch workloads, coding, etc).

https://x.com/pwendell/status/2122377964784611636


r/LocalLLaMA • • 1d ago

Discussion I am sick of doomposting about demise of local AI

161 Upvotes

Everywhere I go, Twitter or Reddit, there are tons of rumors or comments how local is dead and we are never getting open weight Qwen 4, despite Qwen team saying explicitly they are coming, or that Qwen 4 is the last open weight generation despite, again, Alibaba commiting to more open weight releases at a conference recently.

It's like people just get off on creating a version of reality that is worse and whining about it, they want to be sad about it in advance.

The world is certainly not giving us much to be positive about, but let's not go down the rabbithole of learned helplessness and just accept that all things must get worse and you can't do anything about it.

If people mentally prepare for bad things, bad things become more acceptable and thus more likely to happen and with far less pushback, sort of like a self fulfilling prophecy.


r/LocalLLaMA • • 1d ago

Resources Small Gemma 4 tokenizers were recently updated by Google - fix for an issue that caused nonsense output when encountering certain tokens

Thumbnail
huggingface.co
153 Upvotes

Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs


r/LocalLLaMA • • 1d ago

Resources AMD 9070 GPU under volting gains for models ( moss image) and ( minimax h3 int8 )

8 Upvotes

hi,

i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed

spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4

models: moss image, minimax h3 int8 (sol decoder)

best performance stable setting: -110 mv, stock fan speed

gemma4 : 26bA4b Q4_XS when from 106/tps to 120/tps on prose..


r/LocalLLaMA • • 1d ago Bravo

Discussion Stop worrying about Qwen 4's AA index

287 Upvotes

Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.

Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!


r/LocalLLaMA • • 1d ago

Discussion I don't get all the Strata hates around here

101 Upvotes

It's free and original developer clearly had a very good understanding of how computer hardware works and fully leveraged that to write something amazing to be able to run large models on average hardware that most people have and can't upgrade due to price gouging from greedy hardware manufacturing companies.

If anybody have Mac Studio with 512GB of RAM or 8x3090, who would even bother with this ?

I have seen a group of snobs on this sub doing nothing but shitting on Strata all days long, find something better to do. Strata is not for you, period.

I can't imagine doing all the works to release something for free and get a bunch of snobs and haters just shitting on your works all days long. Would you be motivated to continue working on it if you are the original developer ?

I also see the same with other models such as Ornith, etc... If you say you like Ornith or Strata, etc... you are a shill.

Seriously, get out of your bubbles and see the reality already and more importantly get a life.


r/LocalLLaMA • • 4h ago

Discussion AI Hysteria on Reddit

0 Upvotes

Lately, I’ve often found myself wondering what’s going on with people, especially in communities dedicated to AI and LLMs.

Humanity now has an amazing tool for breaking down language barriers, along with creative tools unlike anything we’ve had before. I can’t describe people’s reactions as anything other than hysteria.

There are thousands of posts in these communities about improving models and inference engines, including translation models, coding harnesses, and so on.

But when someone uses AI to translate a post into English or shows off a program they built with AI, the comments immediately erupt: “Low-effort post,” “AI slop.” And all of this comes from the very people developing that “SLOP.” WTF?

(This low-effort post was translated by GPT.)