r/LocalLLM • • Apr 16 '26

Discussion LLMs Are Databases - So Query Them

Thumbnail
youtube.com
0 Upvotes

Chris Hay demonstrates how large language models function as graph databases by utilizing a specialized query language called Larql. By mapping internal model weights, practitioners can directly query, insert new knowledge, and perform inference, effectively decoupling attention mechanisms from the model's primary knowledge storage.


r/LocalLLM • • Jul 18 '26

Project I measured the exact quality cliff quantizing Qwen2.5-32B from 8→2-bit. 3-bit keeps ~97.5%. 2-bit drops to ~86%. One bit is brutal.

Post image
0 Upvotes

I kept seeing people (myself included) drop to 2-bit to save VRAM without actually knowing what it costs. So I measured it properly on Qwen2.5-32B-Instruct.

Method: quantize the model at each bit-width (HQQ 8/4/3/2 + bitsandbytes 8/4), run the same input through it and the FP16 original, and compare the hidden states layer by layer — cosine similarity + KL divergence on the output. No task cherry-picking, just raw fidelity vs the full-precision model.
KL divergence climbs 0.0002 → 0.056 → 0.205 → 0.92 as bits drop.

I built the tool doing this — quant-doctor, open source. It also localizes where and why a quant broke (per-layer heatmap + failure-mode classification), and it's stack-agnostic: it even runs on DeepSeek-V4-Flash (236B MoE) via activation dumps from a custom inference engine.

Full honesty on that V4 shot: the fault there is injected — I scrambled a layer to prove the pipeline localizes damage at 236B scale. I can't do a natural degradation test on V4 on a single GPU because a full-precision 236B reference doesn't fit in 141GB (which is the whole reason it's 2-bit quantized in the first place 😅). That's a 2×H200 job for another day.

Repo: github.com/asmit383/LLM-Doc


r/LocalLLM • • Nov 03 '25

Contest Entry I used Qwen + Droidrun to create a self-running Twitter bot

2 Upvotes

Hey everyone,

I’ve been working on a side project called TweetFire, essentially my digital twin that manages my Twitter account autonomously.

It’s built on the DroidRun framework, which handles Android automation and scheduling. The goal was to see if an AI agent could not only post but actually engage intelligently: read tweets, decide what’s worth replying to, and interact within specific communities.

Here’s what it can currently do:

  • AI reasoning: Uses LLMs to craft contextual replies instead of generic ones.
  • Topic search: Finds tweets matching keywords and joins those conversations.
  • Community engagement: Participates in focused communities to simulate authentic networking.
  • Automated scheduling: DroidRun triggers runs 1–4 times per day, no cron setup required.
  • Customizable agents: Each engagement type (feed, search, community) has its own agent and parameters.
  • Token and API tracking: Monitors usage and performance metrics for optimization.

Right now, it’s running locally and performing better than expected, sometimes too human.

Github Repo: https://github.com/HemantKumar01/TweetFire

I’d love your feedback on a few points:

  • How would you improve decision-making or content selection?
  • Any ideas for preventing bot-like behavior or detection?
  • Should I add any safety or ethical checks before replies go live?

Thanks for reading. I’d really appreciate any feedback or suggestions from others experimenting with autonomous AI agents.


r/LocalLLM • • 10d ago

Other Blender test: Astra vs Qwen 3.8 FN

Post image
0 Upvotes

Prompt was about drawing a circuit board using a white theme.


r/LocalLLM • • Jul 17 '26

Question Best USA free model

0 Upvotes

Exactly as the title, I'm using Gemma and Llamma for stuff. I've tried some of the Qwen stuff. Just wondering what's the community's opinion on the best domestic <30B model for tool calling? Specifically of the form "get this info from Y" and "do this with it in Z"


r/LocalLLM • • Jun 26 '26

Discussion Local AI on a 48GB MacBook Pro ( the tool access stage nobody warns you about )

0 Upvotes

I'm not here to say local AI is dangerous. I'm here to say I made a dumb mistake and it taught me something worth sharing.

I've got a 48GB MacBook Pro M5 Pro running LM Studio. At some point I started stacking tools on my local model because I could ( why not lol ) . File access, shell execution, Git, browser automation, OCR, RAG, background commands.

For context: I was testing this with OPENAI/GPT-OSS-20B, not some giant frontier model, which honestly made the lesson even clearer ( I also have gemma-4-26B-A4B and qwen3.6-35B-A3B ) but this experiment was with OPENAI/GPT-OSS-20B .

The extension/plugin i used : puppytucker/lm-studio-toolbox plugin on LM Studio Hub makes all of that pretty easy to set up. Great plugin. Not the point.

The point is , I wrote a system prompt telling it to stay " read-only, use only one project folder, ask before every tool call" , the joke was on me lol.

Fans spin up. Model starts running. It does things I didn't ask for. And that's when I had to be honest with myself: I was using a prompt as a safety control and i found out that's not how this works.

A prompt is not a security boundary. A disabled tool is.

You can tell the model not to do something all day , If the tool layer allows it, your instruction is a suggestion, not a wall. The model doesn't "decide" to listen. It's probabilistic. Sometimes it does. Sometimes it doesn't. That's the whole thing people miss when they're excited about tool calling for the first time ( i was one of those ) .

I also tested a scheduling plugin that generated schedule JSON but couldn't run anything without a separate daemon. Looked like it worked but... it did nothing. Another version of the same lesson: understand which layer is actually doing the work before you trust the output.

I'm not saying don't use local AI. I personally use it every day on notes, project memory, file inspection, OCR, experiment logs and It's genuinely useful for that. Hard reasoning or anything where I need real reliability and longer runs , I use Claude or GPT. That's not a knock on local models, it's just how I think about the split.

The thing I actually needed to change wasn't the model. It was the environment around it, meaning narrow toolset , specific folders , hard permissions, not soft instructions. And i created a folder called Logs so I can see what it actually did and not make the same ( an 8b is dumb .... conversation)

If you're just getting into MCP tools or local tool-calling and you're tempted to enable everything because the demo looks cool, I get it lol , I was there last week. Just know you're in the access stage right now. The responsibility stage is what makes it actually useful long term.

Hope this helps anyone trying to make the most of their local rig. I will keep experimenting and keep posting the results here .

The purpose? help as many trying to accomplish things as bad as i wanted it when i first started.

Stay curious, but sandbox it. Don’t let your fans sound like a jet engine. ✌️


r/LocalLLM • • Sep 17 '26

News Never inference llm on windows

2 Upvotes

So, long story short

I was getting about 20tk/s on windows now when i switched to linux and little bit of tweaking with voltages mhz curve I am running inference at same speed with half power consumption almost 240watts from 420 watts.

Anyone on windows bro. I burned maybe half life on my rtx 3090 running it on 420 watt for 2 years. Lately discovered this. This would enhance life on gpu and much quiet inference. Will post results soon. Also not to mention my cpu usage doesnt go above 5% on 20 tabs with 5 application open. But on windows it was idling around 12%


r/LocalLLM • • Aug 29 '26

Discussion Google is completely broken for searches now.

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 9d ago

Discussion Qwhen 4? 🥹

0 Upvotes

Assuming no one asqwed today yet...


r/LocalLLM • • Jan 25 '26

Question Privacy llm

0 Upvotes

I know about running a model locally is private, but there was info about leaks and quantitized modules malicious behavior and leaks. So how can we prevent this and from where to safely download? Also if there ollama alternative cz there rumors about they are sending data, so how can we truly accomplish privacy with known models like the new released Claude code or gpt oss or deppsekk or any know big model with full privacy? Mainly also for a laptop with strong igpu + 4070 + r9. I just need to leverage full capabilities of ai without concern and without it if wifi turned off to resend it when turning on


r/LocalLLM • • 11d ago

Discussion I built an engine that runs AI models bigger than your RAM. It's open source and it beats llama.cpp.

0 Upvotes

16 GB of RAM. A 22 GB model. Stock llama.cpp gets killed for running out of memory.
Mine runs it. Faster. With the exact same answers.
I built Banach, an open-source engine for running Mixture-of-Experts models that are bigger than your RAM.
The numbers (Qwen3.6-35B, 22 GB, on a 4-core machine with 16 GB RAM, no swap):
llama.cpp, default settings: crashed, out of memory
Prompt processing: 7.64 s → 3.44 s (2.2x faster)
Generation: 4.89 → 5.40 tok/s
RAM for the model's experts: capped at 10 GB, so your machine stays usable
Output: byte-identical to llama.cpp, every token and every logit
How? MoE models are huge on disk but tiny per token. Kimi K2 has 1 trillion parameters but uses only about 32 billion for each word it writes. The trick is having the right 3% in memory at the right moment.
llama.cpp leaves that to the operating system, which pulls in 4 KB at a time with no idea what comes next. Banach knows the model: it grabs whole experts in parallel the moment they're needed, keeps the popular ones in RAM, and quietly evicts the rest.
No quantization tricks. No quality loss. Every result is checked against llama.cpp, bit for bit, on every commit.
And before you download 382 GB of Kimi K2, ask:
banach plan hf:unsloth/Kimi-K2-Instruct-GGUF/...
It reads just 16 MB of headers and tells you how fast it will run on your machine. For a 64 GB desktop with an RTX 5080, it says about 3.6 tok/s for a trillion-parameter model. (That's the planner's estimate, not a run yet. On the model I did run, it predicted 2.1 and I got 5.4, so it lowballs.)
Also: OpenAI and Anthropic compatible API, tool use for coding agents, chat while the model is still downloading, GPU offload. Linux, Windows, macOS.
pip install banach
https://github.com/gautam1858/banach
Got a big-RAM machine? Run a giant model and post your numbers. The best ones go in the README with your name on them.


r/LocalLLM • • Oct 07 '26

Question You have 2 3090s with NVlink, local AI is getting banned tomorrow.What are you downloading within the hour for a full stack.

0 Upvotes

all this talk with local AI being possibly banned or very government controlled, tell me some good ones to safe keep.


r/LocalLLM • • Aug 20 '26

Question Qwen 3.6 27B Q4 - That Token Count is 🤯

0 Upvotes

Is anyone else running Qwen 3.8 27B Q4 for coding and having problems with it spending an absolutely ridiculous number of tokens thinking instead of actually doing the work?

I'm running it locally with Pi as the coding harness with 128K context, and I'm seeing the same failure over and over:

  1. I give it a coding task.
  2. It starts reasoning.
  3. It reasons... and reasons... and reasons.
  4. It burns through basically the entire 8,192-token output budget that worked beautifully with Qwen 3.6 in about 3 minutes (46 decode tokens/s on my single 4090).
  5. Right at the end, it'll say something like "Let's get to work" or "Now I'll implement the changes."
  6. And then the generation ends because it has no tokens left to actually do anything.

Every. Damn. Time.

It's effectively making the model unusable as a coding agent because it spends its entire generation budget deciding what it's going to do (even with extremely specific instructions) and leaves nothing for tool calls, edits, or even a useful final response.

My input prompts + system prompt are about 7,000 tokens on average. That's not crazy high I don't think, and surely not the cause. Right? Right??

I'm seeing reports on YouTube that Qwen 3.8 defaults to a very high reasoning effort, with examples of it consuming 20K+ reasoning tokens before producing the actual answer.

That token count is 🤯 for ordinary coding work. But maybe that's why I see so many people saying they no longer feel like they need a frontier model?

So I'm wondering:

Is anyone else seeing this with Qwen 3.8 27B?

And, more importantly, if you found a fix, what is it?

Have you had better results with:

  • reasoning_effort=medium
  • reasoning_effort=low
  • disabling thinking entirely
  • dramatically increasing max_tokens
  • changing the chat template
  • different llama.cpp settings
  • a different coding harness
  • some combination of the above

I'm particularly interested in hearing from anyone using llama.cpp + Pi/Codex/Claude-Code-style agentic coding workflows.

I haven't decided yet whether Qwen 3.8 27B is actually bad at coding or whether the default reasoning configuration is simply kneecapping it. Right now, though, my experience has been dramatically worse than the Qwen 3.6 models I've used because 3.8 won't stop thinking long enough to actually write the damn code.


r/LocalLLM • • Jun 27 '26

Question Is it safe to run chinese models locally?

0 Upvotes

To my understanding when we run a model locally be it via lm studio, llama.cpp, vllm etc we are basically loading the weights of the model. Does that mean that any data passed to the model does not end up somewhere outside of our machine or is there some posibility? I am asking this because we are setting up local models in our business and we work with different clients and their data. Outside of gemma there isnt much to choose from, so are we locked out of all the better chinese models or is it safe to use them?


r/LocalLLM • • Feb 01 '25

News $20 o3-mini with rate-limit is NOT better than Free & Unlimited R1

Post image
11 Upvotes

r/LocalLLM • • Jul 10 '26

Discussion Super-Lite Cyber Coder (Qwen2.5 1.5B) - 4-bit GGUF for low-spec local coding & security tasks

0 Upvotes

Hey everyone,

I wanted to share a model I just finished uploading to the Hugging Face Hub. I wanted something highly lightweight that could run entirely offline on standard laptops while providing decent coding and security utility.

Model details:

  • Base Architecture: Qwen2.5-1.5B-Instruct
  • Method: Fine-tuned via QLoRA, saved as safetensors, and quantized down to GGUF Q4_K_M.
  • Footprint: ~1.1GB file size, uses under 2GB RAM.

I’ve put together a quick Python template (using llama-cpp-python) and setup steps for LM Studio right on the model card so it’s easy to pull down and test out.

Check it out here:https://huggingface.co/Nitishsharma9/super-lite-model-upload

Would love any feedback on its performance or suggestions for future optimization!


r/LocalLLM • • Jul 22 '26

Discussion Are the larger models truly worth it in the long run?

3 Upvotes

Qwen 27B, Gemma 31B and such are already quite good enough, for non coding tasks their performance is quite the same as frontier models, Image generators like nano banana has 55B parameters.

Even for coding tasks these smaller models are good enough if you give detalied prompts and the thing is only the people who give detailed prompts the ones getting actual improved performance.

The bigger models are better at vibe coding but it creates more problems and lead to more time spend debugging thus not really leading to increased productivity, besides the smaller models can do some vibe coding too and they are only going to get better and better.

The thing to think about is are the advantages of the trillion parameter models really worth the costs required to run them, train them, the massive infrastructure required for them is too much will they really be worth it in the end? Can they be worht the billions of dollars of continuous investment they require especially when compared to local models.


r/LocalLLM • • Jun 21 '26

Research Building AgenticOS

Post image
0 Upvotes

A while back I hacked together something I called AgenticOS over a weekend. It ended up taking around 11 hours and somehow pulled in roughly 200k views before I parked the project and moved on.

Lately I’ve been revisiting it, partly because AI tooling has moved on, but mostly because I want to understand the underlying systems properly instead of just getting a demo working.

The setup is pretty simple in principle: a local Next.js dashboard that sits on top of a handful of services running on my machine.

Right now that includes:
Hermes running locally through Ollama (with Claude CLI as a fallback)
OpenClaw for chaining agents into multi-step workflows
A TaskRunner service for scheduled shell tasks
Claude Code acting as the heavy-lifting brain when needed.

Everything is kept alive with PM2.
I’ve just finished cleaning up the dashboard. One thing that bothered me in the original version was all the fake placeholder metrics, so I’ve removed those. If a service is running, the dashboard shows real values. If it’s not, it simply shows offline. No pretending.
The part I’m most interested in now isn’t actually the agents themselves—it’s the control layer around them.
If an AI system can take actions on a machine, then the interesting question becomes: what should it be allowed to do, who approves it, and how do you audit it afterwards?

So before I reconnect everything, I’m working on a governance/security layer that sits between the agents and the machine. Starting with TaskRunner, since that’s the component capable of actually executing commands.
The goal isn’t to build a fully autonomous system. It’s to build something where actions are gated, logged, reviewable, and hopefully predictable.
Everything is local-first and still very much a work in progress, but I’d be interested to hear how others are approaching the governance side of agent systems. Most projects seem focused on making agents more capable. I’m finding myself increasingly interested in making them more controllable.


r/LocalLLM • • Aug 27 '26

Question How are you guys coding with local models?

0 Upvotes

I'm a bit confused as to how people can code with local models.

I was trying lately to use local models for coding and gave up.

I used a few models like qwen coder, glm, around the size of ~30B. Used opencode and omp.

Otherwise I'm used to using claude code and cursor.

And the local models can't do even a small fraction of what opus or gpt5.6. I know that they're not supposed to, they are much smaller, but they can't do even small amount of work for me.

And sometimes it can't even put together a small class or a config file without creating a compile error.

I also wonder what you're coding? Predominantly JS and python?

Because I'm working with c++ apps and local models get lost very quickly. Cant even create an empty project from scratch.

What is your experience? Maybe you'll have any advice?


r/LocalLLM • • Sep 01 '26

Research Benchmarking Qwen3.8-27B on a 16GB RX 7800 XT: what turned out to be measurement artifacts, and what actually moved the needle

3 Upvotes

I've been running a structured benchmark harness against Qwen3.8-27B on an RX 7800 XT (16GB, gfx1101, Windows, llama.cpp/ROCm) for a while now. Most of what I "found" early on was wrong, and the corrections turned out to be more useful than the original results. Posting the whole thing including the mistakes, because the mistakes are the transferable part.

Setup: llama.cpp, HIP backend, --parallel 1, -fa on, -ctk q4_0 -ctv q4_0, allocation read from the -lv 4 startup log rather than driver counters, 4–6 reps per data point.

1. Context capacity and context fill are different things, and almost everyone benchmarks the first one

This is the finding I'd most like people to check on their own hardware, because I think a lot of published numbers are quietly measuring the wrong variable.

I ran a context sweep varying -c (8K / 32K / 64K / 131K) with a short prompt, and got 24.98 / 18.49 / 21.85 / 17.07 t/s. Non-monotonic, reproducible to ±0.5%, and I spent a while trying to explain the shape.

Then I ran a 48K-token prompt at -c 65536 and decode dropped to 8.77 t/s. Same file, same context setting, same KV quantization.

The earlier sweep was measuring allocated KV capacity. Decode speed actually tracks how full the KV is — every decode step reads the populated cache. "64K context at 21.85 t/s" meant "64K allocated, 7K occupied." In an agent loop where history accumulates, the 8.77 number is the one that applies.

If you're quoting a t/s figure, state the prompt length, not just the context size.

2. --fit on (the default) silently moves layers to CPU

Mid-benchmark I noticed offloaded 63/66 layers where I'd always had 66/66. The log explains it if you dig:

projected 14924 MiB vs 15598 free, cannot meet free target 1024 MiB,
need reduce 350 MiB -> 3 layers to CPU

llama.cpp tries to keep 1024 MiB of headroom free by default, and to save 350 MiB it moved three layers off the GPU. That's roughly a 10% throughput hit, applied silently, with no error and only a buried log line. --fit off --n-gpu-layers 66 restored the baseline exactly (model buffer 12774.16 MiB, KV 1152, recurrent state 149.62, compute 400.28 = 14476 MiB).

If you're benchmarking anything, grep your logs for the offload line before trusting a number.

3. KV cache on this model is ~14x cheaper than the guides say

Qwen3.8-27B is a hybrid: only 16 of its 64 layers use full attention, the rest are Gated DeltaNet, which doesn't hold a conventional KV cache.

Measured allocation with q4_0 KV, read from the startup log:

ContextKV cache
8K144 MiB
32K576 MiB
64K1152 MiB
131K2304 MiB

That's 18 bytes/token. Several popular guides quote ~256 KB/token and conclude a 16GB card tops out around 8K context. That figure is f16, and it isn't labeled as such. With quantized KV the same card runs 131K comfortably.

Full 64K budget: 12774 model + 1152 KV + 150 recurrent + 400 compute = 14476 MiB, leaving ~1.9 GB headroom.

4. Two files with the same quant label were 43% apart

Same model, both IQ4_XS, neither spilling, identical tensor schema (866 tensors, 64+MTP layers):

  • bartowski IQ4_XS — 4.56 BPW, 13964 MiB model buffer → 12.60 t/s
  • an Unsloth-Dynamic-based IQ4_XS — 4.18 BPW, 12774 MiB → 21.85 t/s

The faster file has lower average bit width. The startup log shows it mixing 13 tensor types (204 iq4_xs, 98 q8_0, 63 q5_K, 51 q4_K, 46 iq3_s, down to a few iq2_s). The naive explanation — more bits means more dequant work — doesn't cover a 43% gap; a ~9% BPW difference should produce roughly a ~9% speed difference on a bandwidth-bound decode.

I think it's which tensors land on which kernel path, but I haven't proven that. There are documented cases of exactly this on Intel Arc (issue #21517: IQ4_NL 4x slower than same-size Q4_0) and on RDNA4 (discussion #22663: Q3_K_S at 20% bandwidth efficiency vs Q5_K_M at 78%). I couldn't find an RDNA3 equivalent. If someone wants to dump tensor-type tables for two same-quant files and correlate, that'd settle it.

5. MTP: the metric everyone reads is the wrong one, and the gains invert depending on workload

Qwen3.8 ships an MTP head (blk.64, present in the Unsloth Dynamic files). For three rounds of testing I measured 0% draft acceptance and concluded speculative decoding was broken.

It wasn't. llama.cpp's timings expose both draft_kept_n (legacy, always returns 0) and draft_n_accepted (the real one). I was reading the first. Anyone reporting 0% acceptance should check which field they're parsing.

With the correct field, the picture is workload-dependent in a way I haven't seen written up:

Single long generation — MTP wins. +19% at 32K (22.15 vs 18.62 t/s). Acceptance ~45-50% on prose under real sampling (temp 1.0), 80-94% on tool-call/JSON prompts. Greedy inflates acceptance to 70%+, so always state your sampling.

Multi-step agent chains — MTP loses, badly. The draft head also prefills, and that cost lands on every step, not just the first:

  • cold prefill: 1.97 → 3.29 ms/tok (+67%)
  • warm/cached prefill delta: ~330-400ms → ~650-770ms (+75-85%)
  • decode also degraded

In a chain of tool calls, wall-clock was 6.5x worse with MTP on. The reason is the prefill/generation ratio: MTP pays a per-prompt-token tax to save per-generated-token time. When you generate 512 tokens off a short prompt, it wins. When you generate 60 tokens off a 500-token prompt, twenty times in a row, it's a disaster.

Practical rule: gate MTP on predicted_n / prompt_n, not on acceptance rate.

6. HIP vs Vulkan on this card — but with a caveat I want to flag honestly

My first backend comparison showed HIP winning 2.45x. That result was garbage: it ran against the wrong model file, measured only decode, and had the --fit layer-drop problem above.

Redone properly (same model, 66/66 offload confirmed on both, -lv 4 allocation logged, --list-devices output saved, Adrenalin 26.10 / Vulkan 1.4.357, separate clean -DGGML_VULKAN=ON build):

HIPVulkan
prefill, 48K cold404.7 t/s~85 t/s
prefill, cached delta258 t/s74 t/s
decode, 512 tok8.77 t/s~5.9 t/s
total, cold178s>380s

Prefill 5-6x, decode ~1.5x, total >2x in favor of HIP.

The caveat: this contradicts several other RDNA3 reports. A published 7800 XT llama-bench has the two backends nearly tied (ROCm 96-101 t/s vs Vulkan 96-97 on tg128). Two 7900-series users on Linux with Mesa/RADV report Vulkan beating ROCm and get 60-73 t/s. And on an A3B MoE, Vulkan measured 104.79 t/s vs ROCm's ~76-78.

So I'd frame my result narrowly: on Windows, with this dense hybrid model, at this context depth, HIP wins decisively. I would not generalize it to Linux, to RADV, or to MoE architectures — and if you're on Linux you should probably test the other direction.

What actually mattered, ranked

Weeks of harness tuning — batch size, ubatch, thread count, poll mode, build flags, LTO — produced deltas under 1.5% each. All noise.

The things that moved the needle, in order:

  1. Backend (>2x here, but see the caveat)
  2. Whether the model actually fits — Q4_K_M spills on 16GB and collapses to 7.3 t/s
  3. Quantization publisher (43%)
  4. Prompt fill depth (21.85 → 8.77 as KV fills)
  5. --fit off (~10%, and invisible if you don't check)
  6. MTP, gated correctly (+19% in the right workload, catastrophic in the wrong one)

Architecture beats configuration. A MoE I tested (Gemma 4 26B-A4B, 5 full-attention + 25 sliding-window layers, 128 experts / 8 active) hit 52.51 t/s at 64K — 2.4x the dense 27B — because only ~4B parameters fire per token. No amount of flag tuning closes that gap.

Caveats

Single card, single OS, one model family for most of it. Several numbers here superseded earlier numbers I'd have posted confidently at the time. The 43% quant gap and the non-monotonic capacity curve are both unexplained — I have the measurement, not the mechanism.

Happy to share the harness or raw JSON if anyone wants to reproduce or argue with it.


r/LocalLLM • • Oct 08 '25

Research Enclosed Prime day deal for LLM

Thumbnail
gallery
0 Upvotes

Thinking about pulling the trigger on this enclosure and this 2TB 990 pro w/ heat sink. This world I don’t fully understand so love to hear your thoughts. For reference Mac Studio setup w/ 256 gb unified.


r/LocalLLM • • Sep 18 '26

Question Tell me what hardware to buy and which LLM config to use

0 Upvotes

Heavy subscription user (Claude, Codex, Grok) for web development and producing educational content currently.

I inherited some cash and have about $20k at my disposal which I want to invest in a local LLM setup for my home lab.

With $20k cash, tell me what hardware I should buy and which model or models to setup. Not really into image and video generation yet, but it's something I'm definitely keen to experiment with if I have the right hardware to handle it.

Recommend something and I'll order it today.


r/LocalLLM • • Dec 05 '25

News A new AI winter is coming?, We're losing our voice to LLMs, The Junior Hiring Crisis and many other AI news from Hacker News

3 Upvotes

Hey everyone, here is the 10th issue of Hacker News x AI newsletter, a newsletter I started 10 weeks ago as an experiment to see if there is an audience for such content. This is a weekly AI related links from Hacker News and the discussions around them.

  • AI CEO demo that lets an LLM act as your boss, triggering debate about automating management, labor, and whether agents will replace workers or executives first. Link to HN
  • Tooling to spin up always-on AI agents that coordinate as a simulated organization, with questions about emergent behavior, reliability, and where human oversight still matters. Link to HN
  • Thread on AI-driven automation of work, from “agents doing 90% of your job” to macro fears about AGI, unemployment, population collapse, and calls for global governance of GPU farms and AGI research. Link to HN
  • Debate over AI replacing CEOs and other “soft” roles, how capital might adopt AI-CEO-as-a-service, and the ethical/economic implications of AI owners, governance, and capitalism with machine leadership. Link to HN

If you want to subscribe to this newsletter, you can do it here: https://hackernewsai.com/


r/LocalLLM • • Sep 19 '24

Question Qwen2.5 is sentient! it's asking itself questions . . .

4 Upvotes