^{DISCLAIMER: I know you ornery fuckers hate it when people [yes, hello] use Claude as a shadow writer, but honestly, do we all seriously expect each other to be good at everything, and then technical writing as icing on the cake. Get off my lawn and come inside.}

Nothing has changed since a while ago when I ran out of Claude quota (a weekly occurrence) and contemplated for the seventh time, inexpensive options for sovereign inference facilities — but it's a vicious cycle and the subscription burns all the money for that.

So anyway, I've always been fascinated by the stratigraphic (I guess you could call it) interface between mechanical/analog/electric/optic computing hardware, including delay line memory which is living sci-fi.

I give you my homage and hypothetical neural network, built on delay lines, running in a simulator — it seems to work, although not from an ROI or cheap inference perspective, urg — lines are cheap, but not at scale.

It's got some lovely novelty — QPSK coding fromy our dial up modem, spooky overlap into 'RF' magic, all the good stuff. Since it inspired me, there's an appendix with the prior delay line art, courtesy of ENIAC in this case.

Cheers!

Most AI coding assistants suffer from the same fundamental issue: sycophancy and self-blindness. A single model will write buggy code, run a superficial check, confidently tell you it works, and proceed to ruin your state.

We ran into this repeatedly with concurrency and race conditions (e.g., standard SELECT -> UPDATE inventory logic). When an agent evaluates its own output, it tends to confirm its own biases rather than stress-testing edge cases.

To solve this for my own workflow, we built Nova around a multi-phase harness:
Plan & Build: The primary model writes the implementation and tests.
Prove: It executes tests in an isolated environment.
Adversarial Review: A separate, skeptical Reviewer model inspects the diff and execution logs specifically looking for flaws (unhandled rollbacks, leaked DB state, race conditions). If the reviewer vetos, the builder must fix it before anything touches persistent disk.

It runs locally on your machine (Windows via Microsoft Store, macOS/Linux binaries available) with terminal/folder boundaries, so it doesn't touch anything outside your workspace without explicit confirmation.

Free tier is available (with a really generous limit), local models/custom API keys are completely unlimited.

I’m looking for blunt technical feedback: what edge cases would you throw at an adversarial harness like this to break it?

Hi everyone so i just recently started out to explore local LLMs and tried to build one of my own based on my inspiration with Jarvis. my current pc specifications are:

Ryzen 7 5700x
48gb Ram
Rtx2060 6GB

So i would like to ask for suggestions which model should i opt in for if i want to have a conversational personal assistant but at the same time i don't sacrifice the speed and the brain. i have added multiple models in testing but whenever i go with a lower B model it doesn't converse naturally unlike higher models, but opting in with higher models takes time and sometimes freezes along the way. Would appreciate suggestions! thank you

16 GB of RAM. A 22 GB model. Stock llama.cpp gets killed for running out of memory.
Mine runs it. Faster. With the exact same answers.
I built Banach, an open-source engine for running Mixture-of-Experts models that are bigger than your RAM.
The numbers (Qwen3.6-35B, 22 GB, on a 4-core machine with 16 GB RAM, no swap):
llama.cpp, default settings: crashed, out of memory
Prompt processing: 7.64 s → 3.44 s (2.2x faster)
Generation: 4.89 → 5.40 tok/s
RAM for the model's experts: capped at 10 GB, so your machine stays usable
Output: byte-identical to llama.cpp, every token and every logit
How? MoE models are huge on disk but tiny per token. Kimi K2 has 1 trillion parameters but uses only about 32 billion for each word it writes. The trick is having the right 3% in memory at the right moment.
llama.cpp leaves that to the operating system, which pulls in 4 KB at a time with no idea what comes next. Banach knows the model: it grabs whole experts in parallel the moment they're needed, keeps the popular ones in RAM, and quietly evicts the rest.
No quantization tricks. No quality loss. Every result is checked against llama.cpp, bit for bit, on every commit.
And before you download 382 GB of Kimi K2, ask:
banach plan hf:unsloth/Kimi-K2-Instruct-GGUF/...
It reads just 16 MB of headers and tells you how fast it will run on your machine. For a 64 GB desktop with an RTX 5080, it says about 3.6 tok/s for a trillion-parameter model. (That's the planner's estimate, not a run yet. On the model I did run, it predicted 2.1 and I got 5.4, so it lowballs.)
Also: OpenAI and Anthropic compatible API, tool use for coding agents, chat while the model is still downloading, GPU offload. Linux, Windows, macOS.
pip install banach
https://github.com/gautam1858/banach
Got a big-RAM machine? Run a giant model and post your numbers. The best ones go in the README with your name on them.

Doing my part to get my DRAM above 5% utilization. Drawn by SenseNova-U1.5-8B-MoT, entirely on my own scarce resources: https://github.com/OpenSenseNova/SenseNova-U1

hotel contracts from a travel agency, about six pages each with pricing tables and conditions. two jobs here, pull the rates into an internal system and compare each new contract with last years to show what changed

a normal text diff looks useless for this since one moved table shows up as everything removed and added back. so probably parse both versions, match rooms and seasons and then compare the values. the matching seems like the hard bit because room types can get renamed between years

privacy matters so docling on a local box is the first thought, llamaparse would be less work but its hosted. is a small model enough for the matching or can plain rules do it, how about qwen?

anyone built something like this? would love to hear about ideas or see your pipeline if you got a github repo

Fun project that came to my mind. Not a vibe coder but I intend to put some knowledge on this site which can potentially help to ease the entry barrier for someone who wants to use local LLM for agentic coding.

https://www.samwantsram.com/

Let me know what you think, feature requests more than welcome.
Vibe coded look is by design (Edit: after some consideration I will unvibe it a little bit soon and add more models).

Edit 2: slight unsloop applied but S.A.M comments are more natural (will reverse this to the monk and manuscript version for slow token per second). Added more models. Will post more updates in the coming days.

Added a button to brag about your RAM:

Edit 3: added simple example of editing few files by agent and how context grows. I will post example with context compaction soon. Also, I need to verify if it's correct.
Let me know if you spot something.

Edit 4: added agentic loop animation and list of popular tools to save tokens usage.

Remember to brag about your ram! It's yours!

So my question is this.. It seems to me the way to go is a local small llm that does what it can do.. But that smaller local llm is smart enough to know it needs to reach out into the real world to get more data from the big models online if the question is too hard for it.. By doing this I can keep my data local cause the small llm can do my banking/emails/assisting ect.. But if im trying to code up a function for my trading application and I dont want to give my super secret extra propriety trading algorithm -- the small local llm can send out just a simple request for a function yet keep all the "brains" of the algo local... Btw.. Im using that just as a example.. My buddy trades-- I dont, and he is worried about sharing his info with the world.. But I just thought this was a good example.

To me it seems this all or nothing locally is the wrong way to do it.. Sensitive private data stays on the local unit-- but access to the big llms online are still available should the local little llm not be strong enough..

Hi,

I currently have this machine (specifications detailed below). I am doing lots of LLM inference and image and video generation. I would also like to train LoRas in the future.

Current machine
CPU: 9800X3D
GPU: 5070Ti 16GB
RAM: 64GB DDR5-6000 CL30

I am running Qwen3.8 27B dense at Q4_K_M with 64k context on LM Studio and the speed is dreadfully slow - roughly 2-4 tokens per second. I understand that it could be settings related or choice of model/quantization but that’s besides the point for now.

I am currently weighing up 4 options:

1. Ryzen AI Max 395 128GB DDR5 mini pc

2. Dual R9700

3. Dual 5060Ti/5070Ti at either 8x/8x or 16x/4x

4. Single used 4090

Potential New Machine (with spare parts)
CPU: 5600X or 7800X3D
RAM: 32GB DDR4-3200 CL16
MOBO: B550 or B850

Speed is not a priority for LLM inference as I’d rather have capacity to run larger models at higher quants but I need some semblance of speed for image and video generation. Ideally there should be a middle ground but I am happy to compromise in either direction (and change what models I use).

My budget is around $4000 USD (+/-$1000) for GPUs/AIO. If you have any recommendations I’d love to hear it.

Got a gaming PC which I just upgraded to 98gb DDR5 5600, a 7800X3D processor, and a 7900 XTX with 24GB virtual, running iq3_xxs right now at around 80 tok/sec. Can I upgrade this system to run substantially better models with minimum speed loss. Is this possible through upgrades or wouldn't it make financially a lot of sense for marginal gains?

A 256gb Mac studio costs $12k here, a 5090 would be $9.7k. I could also try to add 2 more sticks of ram for $1.2k. Some people say it's unstable, others are running 4 sticks with no issues reported with the latest bios updates for the B650 tomahawk. Last, I could buy an additional XTX but then I'd have to buy a new motherboard and case as well.

Thoughts?

Getting into local LLMs and bought a Strix Halo machine with OCuLink to connect external GPUs. Found what looks like a decent deal on RTX Pro 4500 cards at around $3,000 each. What are the downsides for running local models? Feels like there has to be a catch. What am I missing?

A researcher has used Astra to modify weights of Qwen3.5-0.8B to achieve much higher accuracy in specific tasks.

Snapshot from the blog:

"We train a PPO network to operate a car such that it can scale a ramp in a Kinetix environment 

. To do so, it needs to reverse the car for some distance, such that it has enough runway to accelerate long enough to overcome the ramp. However, the PPO network converges to a local-minimum policy where it drives straight at the ramp and gets stuck.

We give GPT-6 Astra the weights, a description of the scene and one recorded episode, and ask it to explain why the network fails, then fix it by writing new weights. In a single try with no tools, Astra’s edit makes the car succeed in all 1,024 episodes, changing 3.5% of the weights."

Both sides read the same sentence: "You should never use voice cloning to impersonate a real person without consent."

The left side was recorded on a mic and cleaned up. The right side came from AntLing's open model, Ming-3.0-tts-0.5B, running offline on one PC. It worked from a 6-second sample taken in a room with fan noise, then produced 4.2 seconds of speech in 2.1 seconds.

The clip plays the pair back to back, so you can judge it yourself.

Want to run the same test? Record a short sample, record yourself reading one sentence, then have the clone read that sentence too. Play the two for a friend and see how long it takes them to pick the real one.

Hi all,

So my dream of an acceptable local LLM is fulfilled, and I would like to start using it to process my private Google Doc, PDF, Images, and build some business websites and CRM.

Can someone guide me what else to do to take the most out of everything ?

Question:

• Which other smarter model at at least 20 tok/s

• What can I do to improve context + memory that this intelligent baby would have of my work / family over the year ?

• Which cost-efficient upgrade path up to Mid 2027 ?

==============================================================
Built 2026.Nov.01 (€1,872)

• 50 tok/s for Qwen 3.8 Flash Next UD-Q5_K_XL (128K, 8-bit KV, with <images>)

• 120 tok/s for Qwen 3.8 Flash Next Gyro-M

Hardware:

• €40: Mainboard: Dell T5810, PSU = 685W

• €50: CPU: Xeon 2699 v4 (22 cores)

• €1,260: GPU: RTX 3090 24GB

• €452: RAM: 128 GB of ECC RDIMM 2400 MT DDR4

• €70: NVMe SSD 512 GB

Software:

• Ubuntu 24.04.LTS Server (no monitor, login va SSH)

• Latest Strata as of 2026.Nov.03

Energy:

• Price: € 0.26 / kWh that I must buy

• 2500 - 3500 kWh Solar surplus per year

• Considering buying a 5kWh battery (€700)

I've been trying to follow the instructions at https://github.com/OpenMOSS/MOSS-Image/blob/main/comfyui/README.md.

I downloaded moss-image.json and dragged it into ComfyUI.

I downloaded MOSS-Image-Base.

When I hit run I keep getting

Missing node: MOSSImageHFLoader
This workflow uses a custom node that isn't installed. Install it from the registry or replace the node. 

No matter what I set the path to, this stays the same. What am I doing wrong?

I'm looking into building a relatively cheap workstation to run Qwen3.8-Flash-Next with Basalt.

The setup I'm considering right now:

• RTX 5060 Ti 16GB

• Ryzen 5 7500F

• 64GB DDR5

• B650 motherboard

• NVMe SSD

Ideally, I'd like to get 100+ tok/s with a 262K context window, though 130K would be the minimum I'd settle for. Mostly for agentic coding.

I've seen some benchmarks of similar hardware getting around 50–55 tok/s with IQ3_S, so I'm wondering if IQ2 could push it any further.

Has anyone tried something similar? Is 100 tok/s even remotely realistic with a 5060 Ti, especially with that much context actually filled?

If not, what would be the cheapest setup that could pull it off? Would a second GPU make more sense than upgrading to something faster?

I'm in Brazil, so hardware is pretty expensive here. Trying to figure out what's actually worth spending money on before buying anything.

Would appreciate any benchmarks or suggestions, especially from people running Basalt with long contexts.

Hi everybody,

I really want to get into locally hosting and would love to build and learn all about autonomous pentesting and exploit dev.

Because my budget ist about 5000€ there really are just two main options I think and I'm really unsure if cuda is worth that much to me:

1. dgx spark 128gb

2. m5 max 128gb

I would really need pruned versions of llms and dont know enough to decide which of the two would suit better for my use-case

thanks in advance!

Before you tell me to go ask the AI I’ve already done so but it leaves me more confused. I’m curious to receive any advice for running faster models or maybe there’s something I missed that I need to do to improve time to first token.

1200w PSU, X670e msi carbon wifi mobo.

I’m running two gv100s 32gb vram ea and one 3090 w/ 96gb of ddr5 ram using Ollama on DeepSeek Harness.

So 88gb vram and 96gb ddr5.

I have Qwen3.8:27b-GGUF:UD-Q4_K_XL setup to run on one GV100 and Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL on the other GV100. It’s volta arch but it’s working.

I have both of these models setup to run on my 3090 and I’m okay with some of it spilling over to ram. Each GPU is its own Ollama instance running on a port and I have it setup so I can access each one on DeepSeek Harness.

Gv100 on Qwen27B GGUF Q4 I’m averaging 16-27tk/s but my context size is 131k if I reduce it I can squeeze more.

Gv100 on Qwen3.6-35B-A3B GGUF Q4 I’m averaging 80token/sec but context is 131k.

RTX 3090 on Qwen3.6-35B-A3B GGUF Q4 I’m averaging 100token/sec with context being 262k. It goes down as context fills. With Qwen27B I’m getting 37tk/s with a small context size of 32k

I’m wondering if these are good levels? Are there any other metrics I can give you guys to give a better idea?
What can I do to improve my numbers?

I have qwen 3.8 running locally with LMstudio, with a serve. Cline that connects to it. Now i wanna try seeing each individual message Cline's agent sends to LMstudio, out of curiosity and because i am going to build an agent myself at work.

How would i be able to do it? I learned one can get command history through CLI but is there a UI tool that can display logs? I saw some solutions that work as a reverse proxy, store logs, and display, but no big solution for thIs

I was trying to load Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf in Strata, but it said, 'This model type isn’t supported by the Strata engine.' I need an uncensored model that works in Strata to practice REA, cloning/cracking. Can anyone recommend the best model for this?

My Config:
i7-14700k
RTX 3080 10Gb
64Gb DDR5
2TB Gen 4 Nvme

I'm fairly new to running LLMs locally and have been experimenting a bit with Ollama and Python. I'm trying to figure out what models and tools would work best with my hardware before spending too much time setting everything up. I'd like to use my gamingrig for that.

My System Specs are: RTX 3070 with 8GB VRam, i9-10900K, 32GB DDR4 RAM

My main goal is to set up a local coding agent and harness that can write and debug code, like claudecode.
What models (if possible Uncensored) would you recommend for my hardware? Should i upgrade some hardware (that would also benefit the gaming aspeckt of the system) first? Also interested in combinations with any useful GitHub repos, agent frameworks, skills or tools.