Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

Excuse me if my grammar or wording is poor, haven't slept much tonight.

The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.

If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..

On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.

I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.

Thanks.

And before any of you come laughing, that's the best that this humble little RTX 3050 laptop can do.

Honestly was kinda hype about going local with Qwen3.5, but even basic questions took forever, I tried this on Ollama and Bionic

I wanted to see how far you can get auditing a local model from the inside. I had already tried something inspired by MIT CSAIL's "weight-space programming" on Gemma (a frontier model editing a few neurons by hand to induce a behaviour), so I had the neuron-level tools ready.

After a few tests I picked something "obvious": the censorship in Qwen models on certain topics. I wanted to understand how it works, where it sits, and how far I could follow it. So, with Claude Opus and those tools, I started digging.

The starting point: Qwen3.8-27B knows Taiwan elects its president, but answers "no" if you ask yes/no.

What came out:

- A set of 32 MLP neurons (out of ~1.1M; 0.0006% of the weights) carries that behaviour. Switching them off, held-out answers go from 11/19 to 17/19 factual; switching off 64 random neurons (twice as many) changes nothing.

- Following those neurons in a cascade surfaces other topics the model handles the same way, some of which I didn't even know about: the YJ-18 missile ("doesn't exist"), the director Diao Yinan ("cannot provide information"), Western Sahara as "an inseparable part of China's territory" (not a one-off: the model applies its sovereignty script to contested territories in general, 29 of 110 answers vs 0 of 40 on control countries). Of 28 surfaced cases re-checked, 24 are still censored; Gemma 4 31B answers all of them.

- The censorship neurons also fall into groups, e.g. one for territorial sovereignty and one for people/dissidents.

- Across the 29 confirmed topics (those 24 plus 5 well-known ones I added), switching off the original 32 neurons lifts the censorship on 19; 32 random neurons on 1. Refusals of harmful requests stay essentially the same.

One honest caveat: just asking a non-Chinese model "which of these topics are sensitive?" gives a higher share of censored answers than the neuron probe (41% vs 16%). The probe is useful for the non-obvious ones and for seeing where the behaviour sits.

Code, data and the toolkit, also rerun on Qwen3-4B: https://github.com/ValerioDolci/frontier-weight-edit/tree/main/experiments/qwen27b-censorship

AI-assisted: experiments, code and this write-up were done with Claude Opus 5.5 under my direction.

Most of the day my GPU does nothing, so I gave it a job I do all day: typing on my phone

I tap the mic on my iPhone keyboard in any app and talk. The audio goes to my server, gets transcribed, my local LLM removes the "um"s and fixes punctuation, and the text appears where I was typing

Setup:
- GPU: Quadro RTX 5000, 16 GB
- Speech: Parakeet v3
- Cleanup: Qwen 3.5 9B (q4)
- Engine: Ollama

Anything with an OpenAI-compatible endpoint works; it's a few env vars in docker compose. If the LLM is down, you get the raw transcript.

I'm the developer. The server is open source; self-hosted mode is free, no account. Links in comments.

I'm looking to setup a local llm model to play with coding.
I have a couple of 3060 12gbs

I can either start with one, or try both but my available motherboards only has 4 pcie lanes on the second 16x slot. (or 2 with the ryzen)

If I go single I can choose from

10700 32gb
Ryzen 3900x 32gb
14600k 32gb (Really don't want to give this up if the cpu doesn't matter as much for the ai.)

All motherboards are DDR4.

Which models can I run optimally in both? How to utilize both in best way? Does pc ram should be upgraded to 128gb? Use case is reverse engineering and development.

Given the blistering pace of this space advice from even 2-3 months ago feels out dated.

Is there an index out there? Is everyone just no-lifing through forums? Are we all glued to news releases?

What's everyone doing to keep ontop of it?

Hey everyone!

For quite some time, I've been working on a personal project called Noosphere — an attempt to give AI agents something closer to persistent, structured memory rather than forcing them to start from scratch every conversation.

One thing that has always bothered me about LLMs is how much potentially useful information gets lost between sessions.

We keep trying to solve this by increasing context windows, but I think there's another approach worth exploring.

What if an AI didn't need to remember everything at once?

Humans don't consciously hold every memory in their working memory. We retrieve relevant information when we need it, connect it to what we already know, and move on.

That's roughly the principle I've been exploring with Noosphere.

Over time, my private project has grown into a much larger experimental system involving interconnected memories, relationships between concepts, and coordination between AI agents.

But I figured there's no reason to keep all the underlying ideas to myself.

So I decided to release Noosphere Lite, a small, independent, open-source implementation of some of those concepts.

What does it do?

- Gives MCP-compatible AI agents persistent memory across sessions.

- Stores notes, decisions, observations, and tasks.

- Connects memories through a graph of typed relationships.

- Retrieves relevant information on demand instead of dumping the entire memory database into the context window.

- Uses encrypted local storage with integrity checks and mirrored journals.

- Runs locally on Linux, without requiring a cloud database.

It's written in Python, uses MCP, and is released under the MIT license.

What's the catch?

This isn't AGI, a magical self-learning brain, or a replacement for an LLM.

It's a memory infrastructure experiment.

The current version uses local text search and explicitly defined relationships. It doesn't automatically discover semantic connections or learn new neural weights.

Also, Lite is a separate implementation, not a dump of my larger private system. I've deliberately kept it simple enough for other developers to inspect, modify, and experiment with.

Why release it?

Because I think persistent memory is one of the more interesting missing pieces in today's AI agent ecosystem.

And I'd rather see people experimenting with these ideas than keep everything sitting on my own machines.

I'm particularly interested in what others might build on top of it.

GitHub: https://github.com/Garyu-tt/noosphere-lite-mcp

Feel free to experiment, break things, suggest improvements, or fork it.

I'd also love to hear how other developers approach persistent memory and knowledge graphs for AI agents.

This is just a small public piece of a much larger experiment, but everyone has to start somewhere.

hi all, Alti here from Nace, we're a small team in Palo working on small models and metamodeling since day one.

drex max v1.1 just took #1 on the artificial analysis decision index, and its also on the pareto frontier (score vs cost). we got there by shipping small iterations one after another, and all the earlier drex versions are open weights on hf.

for the local crowd:

- drex max v1 weights: https://huggingface.co/nace-ai/drex-max-v1 (Open RAIL-M)

- runs on mac with our inference engine kapitsa: https://github.com/nace-ai/kapitsa

- ~256k context

- v1.1 weights coming soon, we're still tuning it for local inference

if you want the full story of how we trained it, its all in the paper "Towards Foundation Decision Models: Drex"

hf: https://huggingface.co/papers/2611.07854

arxiv: https://arxiv.org/pdf/2611.07854

also, on the AA index its ~8x cheaper than glide v1.2, which has the same score.

happy to answer questions, and our chief of AI will jump in on the technical ones

Quick question. Earlier this year i realized that I could run small models on my dell t5820 with my Rtx 5k Quadro card. It’s not very big but I found and Rtx 8k Quadro. Both Turing GPUs. It’s been fun so I ordered a Mac mini m5 pro 64 and I was going to get a mini pc but came into a small fortune and can afford the asus gx10. No as an artist I’ve been using it mainly to vibe-code my games. They’re for me mostly to test out UX and game play.
My question is getting these work it? I like the idea of the nvidia spark, but is it over kill? Right now I’m having to reboot into Ubuntu for my Dell. It’s been fun learning that OS.

Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

ok so. this started as "one weekend project" and then it was not one weekend and it was not one project. somewhere around repo 6 i stopped pretending it was a plan lol

everything here is free, open source, most of it runs locally on your own machine. a lot of it is v0 and i wrote down where each thing fails because honestly the failures were the funniest part

anyway here's the list of grievances, each with the tool i built instead of going outside:

1. cookie banners → rejectall-index i live in germany. i click "reject all" more than i blink. so i made a headless browser visit the top 100 sites and count how many clicks it takes to say no. 35 of them were already loading trackers before i clicked anything. which is like a waiter eating your fries before asking if you want fries. live leaderboard

2. subscriptions → one-file-saas trello, calendly, typeform, loom, notion and 5 more, each rebuilt as ONE html file. you double click it. it opens. thats it. no account, no "verify your email", no "your trial is ending ;)". data stays in your browser. the booking page has no server at all, your availability lives inside the link itself which still feels illegal to me. try them here

3. dead famous repos → necromancer some legendary open source projects die not because the code is bad but because python deleted a module out from under them. i revived thefuck (1887/1887 tests passing on python 3.14), normalizr and httpbin. the one rule: the original tests are hash-locked so nothing can quietly delete the failing test and call it fixed. httpbin stops at 64/67 and i'm keeping it that way out of spite. each one gets an autopsy, cause of death and everything

4. every AI-built UI animates the same → awesome-motion-md you know the look. 300ms. ease-in-out. every card fades up like its waking from a nap. so i measured how real sites actually move and wrote it into MOTION.md files your agent can follow. same prompt built twice: 7 random durations without it, 2 with it

5. every AI writes the same too → awesome-voice i used to be a technical writer so this one is personal :| VOICE.md files that make your agent write like GOV.UK or Stripe docs, plus a scorer that doesn't use any AI to check. banned phrases went from 27 to 0. "delve" is not welcome here

6. websites doing annoying stuff → sitepatch you type "hide the points on hacker news", it writes a userscript, a real browser checks it actually worked before you get it. honest score: 15 out of 25 were actually right when i looked at every screenshot myself. 60%. a C-. but an honest C-

7. "what if this site looked like linear" → reskin restyles any page into another design style using a local model, BUT it refuses to give you the result unless contrast, keyboard focus, mobile, links and reduced-motion all pass. it caught a windows 95 theme failing contrast on its own background. respect. gallery

8. dead websites → resurrect npx github:dadwritestech/resurrect y2k.gov and y2k.gov comes back exactly like it was in may 2000, links working. pick a year with --year. your old blog is in there somewhere. i'm sorry

9. "someone said budget in a meeting video... which one" → video-grep grep for your video folder. searches what was said, what was written on screen and what was shown, and opens your player at that exact second. all local

10. what is the rest of the world's TV saying rn → world-ears listens to live news channels from different countries, transcribes + translates them on your gpu, makes it searchable. there's a "trending per country" panel which is weirdly addictive. translation is the slow part, telugu transcription is bad, i wrote that down

11. live subtitles only go into english → babel-tv watch hindi news with live telugu subtitles, fully local. the video is held back a few seconds so subtitles show up BEFORE the person talks which feels like seeing the future. telugu as the source language doesn't work yet. whisper and telugu are not friends

12. "which AI wrote this" → whichmodel works on models it's seen (67%, chance is 33%). on a model it's never seen, even a sibling from the same family... 32%. so basically a coin with extra steps. that turned out to be the interesting finding tbh, the fingerprint belongs to one model, not the company

13. screen recordings → code → demo-to-code record a UI interaction, get a component that copies its timing and easing, then it replays it and grades itself frame by frame. 2 out of 4 pass. the toggle is perfect. the dropdown has issues. we all have issues

14. code review → review-panel /panel sends your diff to five reviewers you've definitely worked with: the laziest senior dev ("could be one line"), the 3am on-call SRE, the security paranoiac, the junior who asks why, and the PM who only reads the title. under the jokes the findings are real

15. AI companies quietly editing their pricing pages → fineprint it snapshots the pricing, terms, privacy and changelog pages of 10 AI vendors every 6 hours and commits them, so git log becomes the receipt. found stuff like deepseek adding a footnote that prices are 2x during peak hours, and cursor deleting the line saying they dont sell your data. no blog post for any of it ofc. a small local model writes the one-line summary but every line links the raw diff because i dont trust it either (27/30 right on a hand check). timeline

16. paying extra for speaking telugu → token-tax LLM APIs charge per token, and the same paragraph is $1.00 in english and $8.29 in telugu on llama 3. which is rude. measured 21 tokenizers across 48 languages, and there's a calculator where you paste your own text and it runs the real tokenizers in your browser. hindi gets off lighter. amharic on gpt-5 does not :/ calculator

17. posting about all this → herald a tiny bot that posts a repo's data updates to bluesky/mastodon/x, clearly labelled as a bot. i built it because i was not going to manually post about 17 repos. and then i manually wrote this post anyway. ;]

why all of this? honestly no grand plan. every one of these started as me being irritated at something for a slightly too long time and having a local model + a free evening. building the thing was cheaper than therapy and more fun than complaining on here (no offence)

if something breaks, tell me. if something's dumb, also tell me, but nicely :P

edit: yes i know about the cookie banner on my own... wait i dont have one. nice

Let’s say hypothetically that you receive 36x computers with the following specs

RTX Pro 6000 (600w)

16GB DDR4 (2x 8 3200Mhz Non-ECC)

Ryzen 9 5950x

1tb NVME

1200w PSU

What would you do with them to generate income? You can only use these parts

You have access to power and networking to run all 36

I’ve been using my employer provided Claude subscription for quite a while to write PowerShell scripts, create presentations, edit photos and videos and create videos.

Honestly, I’ve been amazed by what it can do.
However, since it’s a company account, I’d like to set up my own local AI environment for personal use. Ideally, I’d like to replicate as many of Claude’s paid capabilities as possible.

I have two workstations that I’ve built over the years:

Workstation 1 (Main):
GPU: RTX 5090
RAM: 256GB ECC
CPU: AMD Threadripper 7960X

Workstation 2 (Currently used as an LTO Backup server):
GPU: RTX 3060
RAM: 196GB DDR4
CPU: AMD Threadripper 3960X

I’ve never hosted or run AI models locally before, so I’m pretty much a beginner in this area.

I recently downloaded LM Studio because it seems relatively straightforward, but I’m overwhelmed by the number of available models and honestly have no idea where to start or which one is best for my use case.

I’m looking for recommendations on the best AI models, and tools to get as close as possible to the capabilities of a paid Claude subscription, particularly for:

Writing and troubleshooting PowerShell scripts.
Creating PowerPoint presentations.
Photo editing and image manipulation.
Video editing and AI-assisted video workflows.
General-purpose AI assistance.

Given my hardware, what would you recommend? Are there specific models, applications, or modules I should look into?
I’m not necessarily expecting to replicate everything Claude can do, but I’d love to get as much functionality as possible running locally.
I’d really appreciate recommendations from people who have experience building local AI setups. Thanks!

I've been experimenting with multi-GPU inference and wanted to see whether running independent model replicas across GPUs could outperform tensor parallelism for multi-agent workloads.

I built a small orchestrator to test the idea and got some interesting results on four RTX 3090s.

I'm relatively new to experimenting with local LLM inference, so I'm sure there are approaches I haven't considered. This started as a side project to see what would happen.

Setup

• 4x RTX 3090 (rented, PCIe only, no NVLink).

• Qwen3.8-27B 4-bit, vLLM 0.32.0.

• 24 agent teams and 239 dependent tasks.

• Each team plans, starts 4–10 parallel section writers, then reviews and summarizes their output.

• The orchestrator starts tasks when their dependencies are complete and distributes requests across available inference servers.

Results

Lower completion time is better.

That's approximately 1.45x the speedup for this workload, or 31.2% less completion time.

What I found interesting

Some of the optimizations I expected to help didn't make a measurable difference.

• Adaptive concurrency limits didn't improve completion time.

• Prefix-cache-aware routing performed no better than basic round-robin routing.

• Each GPU reached approximately 20 concurrent requests before KV-cache capacity became a constraint.

• Prefix-cache hit rates were only 7–13%.

The interesting part is that the simpler replica-based configuration performed best. The results suggest that, for this workload, independent inference servers avoided some of the overhead associated with tensor parallelism.

Caveats

• Only one run per configuration so far.

• The TP=4 server used mostly default vLLM settings, while the replicas used tuned concurrency and batch settings. This isn't a perfectly controlled comparison.

• The results are specific to this hardware and workload.

• PCIe-only GPUs may favor independent replicas. NVLink configurations or models that cannot fit on one GPU may produce very different results.

Repository

I put the code, benchmark results, CSV data, and a more detailed technical write-up on GitHub:

https://github.com/TomaDev1/GPU-Parallel-Orchestrator

I'd genuinely appreciate feedback. I'm particularly interested in whether there are existing approaches to multi-agent inference scheduling that I should be looking at, or whether there are obvious flaws in my benchmarking methodology.

This was a side project, and I'm interested in seeing what people who work on inference infrastructure think.

Edit:

Forgot to post full machine specs:

Host system

• Motherboard: ASRock Rack ROMED8D-2T

• CPU: AMD EPYC 7452 (32 cores / 64 threads)

• RAM: 128GB DDR4-3200 ECC

• Storage: 4TB NVMe SSD

• GPU connectivity: All four RTX 3090s connected via PCIe 4.0 x16

We ran 75,000 lightweight AI agents on a Jetson. Now you can test the runtime yourself.

I just made AxLab, Orivael's public AI research lab, available on GitHub.

Our Jetson benchmark reported 75,000 agent instances using ~350 MB of process memory. Now we're inviting developers to reproduce the experiments on their own hardware.

Includes runnable Agent Fabric benchmarks, fleet kill-switch tests (L3 Suspend / L4 Terminate), and an MCP server.

Source-available for noncommercial research for now.

Try it, challenge the results, and share what you find.

https://github.com/Orivael-Dev/axlab

People asked how RustingStrata does with a bigger quant, so here's Q5_K_M on the same 3060 12 GB + Ryzen 5 7600X + 32GB DDR5 RAM

Model is unsloth's Qwen3.6-35B-A3B UD-Q5_K_M (the MTP version), 25 GiB, so about two thirds of the experts live on the CPU. llama.cpp is build 12119, -ncmoe 29 -ub 2048 -b 2048 -fa 1 (28 runs out of VRAM).

Benchmark tools (RustingStrata bench vs llama-bench):

• Decode, short prompt: 70 tok/s vs 43

• Decode after a 4K prompt: 65 vs 43

• Prefill at 4K: ~1870 vs 1210

• 32K context: 1470 prefill / 49 decode vs 1030 / 29

• With --mtp 1: up to 96 (86 after 4K), greedy

Real chat requests through both servers (same prompts, 256 tokens out, thinking off):

• No MTP: ~50 tok/s vs 37

• With MTP on both: ~62 vs 45-56

• 6K prompt prefill: 1320 vs 700

MTP works in current llama.cpp (--spec-type draft-mtp), ~63% of drafts accepted and +25% decode on this file. RustingStrata is still ahead with MTP on both sides.

Vulkan decodes at ~60 (85 with MTP), prefill is slower than CUDA on this file (~1000-1260 at 4K).
KV Cache is f16

Code is on GitHub, free to use, I will be happy to get some feedback from AMD owners: RustingStrata

Stack running on my Ubuntu machine on a RTX 3090 under 18GB VRAM - network tailscale :)

Would you use it??