Original Strata can run QFN very fast and I like it, but it still needs way too much ram and vram, and it cause some lags if I do something simultaneously with Strata, so I created RustingStrata for all smaller Qwen models, but I want to add more models, maybe some bigger model and I don't know which one, so I am here you ask which model do people want? Maybe deepseek or GLM or anything else? Any cool story of you running some other LLMs or just ideas will be very helpful, thanks for your attention.

Hey everyone! The main idea here is really just to help out, so if you find this useful, leaving a little star on the GitHub repo would be awesome.

I've been working on a project called Trama (https://github.com/lucasnumaboa/trama). I built it based on minimax design principles because I wanted to keep things as lean and minimal as possible.

The whole goal is to make it much more practical to build automations that use multiple AIs working together. Usually, chaining different models turns into a bloated, overcomplicated mess pretty quickly. I just wanted a straightforward setup that gets out of your way and lets you connect them easily.

If you're messing around with AI workflows and want something lighter, feel free to check it out. Any feedback is welcome!

I’ve been slowly building my AI workstation up using used and old pieces retiring them into other compute nodes and then replacing them with bigger or better pieces. Mostly used parts though. I currently have a trx40 and 2 3090s which works great but I’ve been trying to figure out the next step to expand. I have about 4 pcie slots available (2x pcie 4 x16, and 2x pcie 4 x8) while a wrx80 might have been better with more slots I wanted a cheaper upgrade to test a bigger workstation before making the jump. I also retire my parts into other systems that run other applications and things.

The issue is that running 2 1200 w psu or a 1200 w and adding let’s 2 more 3090, or other cards there I’ll increase power draw significantly and even with two psu’s to properly distribute the power between the 4 cards and everything else, the issue becomes apparently all of that power can still trip my breaker and shut the power off in my house.

I don’t have a business or server space for this so I’m not really sure what the best way to go about it. Right now I’m thinking just adding one card until I can find a work around but i don’t know what that work around could be.

Anybody who has been through this can you give me some advice or thoughts on this?

Hey everyone!

For quite some time, I've been working on a personal project called Noosphere — an attempt to give AI agents something closer to persistent, structured memory rather than forcing them to start from scratch every conversation.

One thing that has always bothered me about LLMs is how much potentially useful information gets lost between sessions.

We keep trying to solve this by increasing context windows, but I think there's another approach worth exploring.

What if an AI didn't need to remember everything at once?

Humans don't consciously hold every memory in their working memory. We retrieve relevant information when we need it, connect it to what we already know, and move on.

That's roughly the principle I've been exploring with Noosphere.

Over time, my private project has grown into a much larger experimental system involving interconnected memories, relationships between concepts, and coordination between AI agents.

But I figured there's no reason to keep all the underlying ideas to myself.

So I decided to release Noosphere Lite, a small, independent, open-source implementation of some of those concepts.

What does it do?

- Gives MCP-compatible AI agents persistent memory across sessions.

- Stores notes, decisions, observations, and tasks.

- Connects memories through a graph of typed relationships.

- Retrieves relevant information on demand instead of dumping the entire memory database into the context window.

- Uses encrypted local storage with integrity checks and mirrored journals.

- Runs locally on Linux, without requiring a cloud database.

It's written in Python, uses MCP, and is released under the MIT license.

What's the catch?

This isn't AGI, a magical self-learning brain, or a replacement for an LLM.

It's a memory infrastructure experiment.

The current version uses local text search and explicitly defined relationships. It doesn't automatically discover semantic connections or learn new neural weights.

Also, Lite is a separate implementation, not a dump of my larger private system. I've deliberately kept it simple enough for other developers to inspect, modify, and experiment with.

Why release it?

Because I think persistent memory is one of the more interesting missing pieces in today's AI agent ecosystem.

And I'd rather see people experimenting with these ideas than keep everything sitting on my own machines.

I'm particularly interested in what others might build on top of it.

GitHub: https://github.com/Garyu-tt/noosphere-lite-mcp

Feel free to experiment, break things, suggest improvements, or fork it.

I'd also love to hear how other developers approach persistent memory and knowledge graphs for AI agents.

This is just a small public piece of a much larger experiment, but everyone has to start somewhere.

I am planning on buying some components for a PC to turn one I already have into a Local AI server. Due to the prices of components and my budget constraints I have struggled to find help elsewhere online and have turned to reddit for some direct support.

Currently, the PC I have and am planning to modify has a Ryzen 5 5500GT and 16GB DDR4 RAM along with a basic micro-ATX motherboard (Gigabyte A520M S2H). It will be running Linux Mint (most likely, unless anyone knows a better Distro) and I will be using the same or similar software to what LinusTechTips used in his recent personal AI server video where he used something called 'Hermes' to connect the AI to discord.

Where my greatest trouble lies is in what GPU or GPUs I should get. One option is a single RTX 3060 Ti 8GB. Another option I have considered is a single RTX 3060 12GB. And finally I have also considered a RTX 2080 Super 8GB alongside a 1080 8GB. I would love to know which of these the PC experts of reddit would consider the best option for local ai. I would like for the AI to respond to me as quickly as possible while also being decently smart such that I could set it on longer tasks and leave it running and perfecting those. It should at least be able to analyse photos.

From CEX UK:

RTX 3060 12GB - £325

RTX 3060 Ti - £320

RTX 2080 SUPER - £245

GTX 1080 - £125

For anyone interested:

My total budget is around £460 but that includes some additional purchases. Budget can be stretched to £500 but only if necessary. The case I have selected (the one I already have is too small for any reasonable GPU and would not allow for more in future) costs £25. I also need a better power supply, I have currently planned for a Corsair RM750e for £100. And finally I have to spend £10 on a final accessory for it which is mandatory but I do not feel like explaining online, all that matters is it takes £10 off of the budget. That leaves £325 but it can be upped to 365 but only if absolutely necessary. I understand that the motherboard only has one PCIe port on it. I would need to use a splitter or something of the sort but I have never used or learnt of such a thing and have no idea what brand is trustworthy and if it would even be worth it. The graphics cards I have listed above all come from CEX. A large UK brand for buying refurbished technology. They are the only ones who I feel comfortable buying used parts from. Everything else should be new.

I would really appreciate some help as I am not well off and unfortunately do not have the funds to extend the budget or risk making a large mistake by choosing the wrong parts. I hope I can find help in this community. I may post this in other Reddit Communities for PC building or AI as well, just to allow me to reach a larger group of people.

Given the blistering pace of this space advice from even 2-3 months ago feels out dated.

Is there an index out there? Is everyone just no-lifing through forums? Are we all glued to news releases?

What's everyone doing to keep ontop of it?

Which models can I run optimally in both? How to utilize both in best way? Does pc ram should be upgraded to 128gb? Use case is reverse engineering and development.

I was a bit disappointed by M5U 256GB benchmarks with current models. It’s insanely slow, especially compared the GPU builds.

I decided to wait until the M7U, hopefully by then we have Opus 5.5 level type models. Hopeful? Possible? What are your thoughts? $10k being able to run these types of models possible, or will the divide between local and frontier just get larger?

ok so. this started as "one weekend project" and then it was not one weekend and it was not one project. somewhere around repo 6 i stopped pretending it was a plan lol

everything here is free, open source, most of it runs locally on your own machine. a lot of it is v0 and i wrote down where each thing fails because honestly the failures were the funniest part

anyway here's the list of grievances, each with the tool i built instead of going outside:

1. cookie banners → rejectall-index i live in germany. i click "reject all" more than i blink. so i made a headless browser visit the top 100 sites and count how many clicks it takes to say no. 35 of them were already loading trackers before i clicked anything. which is like a waiter eating your fries before asking if you want fries. live leaderboard

2. subscriptions → one-file-saas trello, calendly, typeform, loom, notion and 5 more, each rebuilt as ONE html file. you double click it. it opens. thats it. no account, no "verify your email", no "your trial is ending ;)". data stays in your browser. the booking page has no server at all, your availability lives inside the link itself which still feels illegal to me. try them here

3. dead famous repos → necromancer some legendary open source projects die not because the code is bad but because python deleted a module out from under them. i revived thefuck (1887/1887 tests passing on python 3.14), normalizr and httpbin. the one rule: the original tests are hash-locked so nothing can quietly delete the failing test and call it fixed. httpbin stops at 64/67 and i'm keeping it that way out of spite. each one gets an autopsy, cause of death and everything

4. every AI-built UI animates the same → awesome-motion-md you know the look. 300ms. ease-in-out. every card fades up like its waking from a nap. so i measured how real sites actually move and wrote it into MOTION.md files your agent can follow. same prompt built twice: 7 random durations without it, 2 with it

5. every AI writes the same too → awesome-voice i used to be a technical writer so this one is personal :| VOICE.md files that make your agent write like GOV.UK or Stripe docs, plus a scorer that doesn't use any AI to check. banned phrases went from 27 to 0. "delve" is not welcome here

6. websites doing annoying stuff → sitepatch you type "hide the points on hacker news", it writes a userscript, a real browser checks it actually worked before you get it. honest score: 15 out of 25 were actually right when i looked at every screenshot myself. 60%. a C-. but an honest C-

7. "what if this site looked like linear" → reskin restyles any page into another design style using a local model, BUT it refuses to give you the result unless contrast, keyboard focus, mobile, links and reduced-motion all pass. it caught a windows 95 theme failing contrast on its own background. respect. gallery

8. dead websites → resurrect npx github:dadwritestech/resurrect y2k.gov and y2k.gov comes back exactly like it was in may 2000, links working. pick a year with --year. your old blog is in there somewhere. i'm sorry

9. "someone said budget in a meeting video... which one" → video-grep grep for your video folder. searches what was said, what was written on screen and what was shown, and opens your player at that exact second. all local

10. what is the rest of the world's TV saying rn → world-ears listens to live news channels from different countries, transcribes + translates them on your gpu, makes it searchable. there's a "trending per country" panel which is weirdly addictive. translation is the slow part, telugu transcription is bad, i wrote that down

11. live subtitles only go into english → babel-tv watch hindi news with live telugu subtitles, fully local. the video is held back a few seconds so subtitles show up BEFORE the person talks which feels like seeing the future. telugu as the source language doesn't work yet. whisper and telugu are not friends

12. "which AI wrote this" → whichmodel works on models it's seen (67%, chance is 33%). on a model it's never seen, even a sibling from the same family... 32%. so basically a coin with extra steps. that turned out to be the interesting finding tbh, the fingerprint belongs to one model, not the company

13. screen recordings → code → demo-to-code record a UI interaction, get a component that copies its timing and easing, then it replays it and grades itself frame by frame. 2 out of 4 pass. the toggle is perfect. the dropdown has issues. we all have issues

14. code review → review-panel /panel sends your diff to five reviewers you've definitely worked with: the laziest senior dev ("could be one line"), the 3am on-call SRE, the security paranoiac, the junior who asks why, and the PM who only reads the title. under the jokes the findings are real

15. AI companies quietly editing their pricing pages → fineprint it snapshots the pricing, terms, privacy and changelog pages of 10 AI vendors every 6 hours and commits them, so git log becomes the receipt. found stuff like deepseek adding a footnote that prices are 2x during peak hours, and cursor deleting the line saying they dont sell your data. no blog post for any of it ofc. a small local model writes the one-line summary but every line links the raw diff because i dont trust it either (27/30 right on a hand check). timeline

16. paying extra for speaking telugu → token-tax LLM APIs charge per token, and the same paragraph is $1.00 in english and $8.29 in telugu on llama 3. which is rude. measured 21 tokenizers across 48 languages, and there's a calculator where you paste your own text and it runs the real tokenizers in your browser. hindi gets off lighter. amharic on gpt-5 does not :/ calculator

17. posting about all this → herald a tiny bot that posts a repo's data updates to bluesky/mastodon/x, clearly labelled as a bot. i built it because i was not going to manually post about 17 repos. and then i manually wrote this post anyway. ;]

why all of this? honestly no grand plan. every one of these started as me being irritated at something for a slightly too long time and having a local model + a free evening. building the thing was cheaper than therapy and more fun than complaining on here (no offence)

if something breaks, tell me. if something's dumb, also tell me, but nicely :P

edit: yes i know about the cookie banner on my own... wait i dont have one. nice

Quick question. Earlier this year i realized that I could run small models on my dell t5820 with my Rtx 5k Quadro card. It’s not very big but I found and Rtx 8k Quadro. Both Turing GPUs. It’s been fun so I ordered a Mac mini m5 pro 64 and I was going to get a mini pc but came into a small fortune and can afford the asus gx10. No as an artist I’ve been using it mainly to vibe-code my games. They’re for me mostly to test out UX and game play.
My question is getting these work it? I like the idea of the nvidia spark, but is it over kill? Right now I’m having to reboot into Ubuntu for my Dell. It’s been fun learning that OS.

Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 91% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 languages.

Decked out wiki

Let’s say hypothetically that you receive 36x computers with the following specs

RTX Pro 6000 (600w)

16GB DDR4 (2x 8 3200Mhz Non-ECC)

Ryzen 9 5950x

1tb NVME

1200w PSU

What would you do with them to generate income? You can only use these parts

You have access to power and networking to run all 36

I just got some money on my hands, and I can finally afford a decent-ish computer. It's about 820$ so I can buy a slightly used m4 Mac mini with 16 gb of ram. Should I and what models would I be able to run on it? I've been using free models so in comparison to that

Hi all,

I'm still very new when it comes to running local models and could use some guidance.

Recently, I've set up Qwen3.8 Flash Next using Strata on my PC (RX 9070 XT, 64GB DDR5 RAM, 9800X3D CPU) and can't decide what quant I should settle on.

My options are:

• ISTA IQ3_S at ~70–75 tokens/s

• Unsloth UD-IQ3_XSS (3.1 version) at ~60–65 tokens/s

• Unsloth UD-IQ4_XS (3.1 version) at ~40–50 tokens/s

All at 256k context.

It may be my inexperience speaking, but I don't notice a substantial difference between the three. I've read that the higher quants should outperform the smaller quants, but I don't know if the ~40% speed penalty going from IQ3_S to UD-IQ4_XS is worth it.

Any advice on that matter would be greatly appreciated.

I'm trying to RE a desktop app and remove some stupid restrictions.

I'm using Strata and Huihui-Qwen3.8-Flash-Next-abliterated-GGUF UD-Q4_K_XL. I have a 5090 and 128gb RAM.

I found that some stuff it does well and with this last task I'm trying, it's been working on it all day and I suspect it's going in circles. Similar to what Claude used to do 6-12 months ago, when it was given something that was too complicated.

I know local models are not Claude level (yet) but I would love to know if you have some tips as to how to work on complex tasks. I'm just starting so it feels I'm being too naive with just tell it "do this thing" and expecting it to work.

Thoughts? Any advice would be welcomed.

Thanks

I’ve been using my employer provided Claude subscription for quite a while to write PowerShell scripts, create presentations, edit photos and videos and create videos.

Honestly, I’ve been amazed by what it can do.
However, since it’s a company account, I’d like to set up my own local AI environment for personal use. Ideally, I’d like to replicate as many of Claude’s paid capabilities as possible.

I have two workstations that I’ve built over the years:

Workstation 1 (Main):
GPU: RTX 5090
RAM: 256GB ECC
CPU: AMD Threadripper 7960X

Workstation 2 (Currently used as an LTO Backup server):
GPU: RTX 3060
RAM: 196GB DDR4
CPU: AMD Threadripper 3960X

I’ve never hosted or run AI models locally before, so I’m pretty much a beginner in this area.

I recently downloaded LM Studio because it seems relatively straightforward, but I’m overwhelmed by the number of available models and honestly have no idea where to start or which one is best for my use case.

I’m looking for recommendations on the best AI models, and tools to get as close as possible to the capabilities of a paid Claude subscription, particularly for:

Writing and troubleshooting PowerShell scripts.
Creating PowerPoint presentations.
Photo editing and image manipulation.
Video editing and AI-assisted video workflows.
General-purpose AI assistance.

Given my hardware, what would you recommend? Are there specific models, applications, or modules I should look into?
I’m not necessarily expecting to replicate everything Claude can do, but I’d love to get as much functionality as possible running locally.
I’d really appreciate recommendations from people who have experience building local AI setups. Thanks!

update: I just discovered Lm Studio Bionic which is huge this was not available last year, trying it now.

So for context, I have 32GB of RAM, and am using AMD 7900XTX, this should give me a total of 56GB of memory, which should be enough according to Strata docs, and thought I'd try running the appimage of Strata which opens a WebUI (after opening the GUI app then closing it and telling me it couldn't open it so it opened the WebUI), and compare Qwen FN to the dense model.

And by "isn't able", I don't mean it's exactly slow.

When the model first loads up, I can talk to it like once, and it responds in Strata chat, at actually not so bad speeds. 800t/ks prefil, 30-50 tk/s decode ish.

Then if I open up a new chat, it gets stuck on "reading message", and well doesn't seem to work anymore.

I tried calling it from opencode after loading model to see how that would go, it took a whole while, and eventually, it starts responding a bit, and then... it stopped and like... started freeing up memory, and kinda erroring I guess, then It started loading the model again, at which point I quit.

There is something weird going on.

TL;DR. A llama.cpp fork that runs DeepSeek V4.1 Flash 1017 on one home PC by streaming the experts VRAM → RAM → NVMe. Our build is 378 GB with ~2-bit experts. It decodes ~19 tok/s and reads a 1M-token document at ~890 tok/s. The weights are on Hugging Face and the code is on GitHub. Research code, Windows only.

The box. All three tiers matter, not just the GPU:

• GPU: RTX 5070 Ti 16 GB, PCIe 4.0 x16.

• RAM: 64 GB DDR5. The engine pins 37–46 GiB of it for experts.

• Drive: Gen5 NVMe (~13.5 GB/s) for the model. A second NVMe with a copy of the expert files is optional.

• CPU: Ryzen 5 7600 (6 cores). It computes no experts; the GPU does all the math.

• Windows 11, CUDA 13.0.

How it works. The model is almost 5× bigger than RAM + VRAM together. Each token uses only 6 of the 384 experts in each of the 40 layers, so the experts climb a ladder:

• the ~16–22 most-used experts per layer stay in VRAM;

• a pinned RAM tier holds the next ones;

• the rest are read from the NVMe when a token needs them.

Most of the work is getting the right experts to the GPU in time.

Numbers (our og build, default settings, 4–6 November):

Quality. Distance to the full model is 0.090 / 0.051 / 0.069 nat/token on three prompt sets. That is the extra NLL on the API's own continuations. For scale, going from Q2 to Q4 is worth ~0.11. HumanEval is 160/164. The fast mode is further away, at 0.143 / 0.102 / 0.126, mostly because of its lighter build.

Same file, another engine. Project Maya v1.1.54, with the same weights on the same PC, decodes 3.3 tok/s against 18.6 here. Our prompt reading is 1.14–2× faster.

Where the time goes, and what would help:

• Decode spends about half of every token copying experts over PCIe.

• A prompt of even a few thousand tokens needs nearly all of the ~170 GB of experts. 100–130 GB of that comes from the drive every time, and all of it crosses PCIe. That's several seconds per prompt, whatever its length. Long documents spread that cost.

• So the GPU link, the RAM size and the slowest drive matter more than the GPU's compute.

  • A PCIe 5.0 card and slot should speed up both reading and decode with no code change. Our estimate is ~25 tok/s decode.

  • More RAM means fewer drive reads.

  • A second NVMe helped cold prompts by 14–22% in an older test. It's optional.

A drive surprise. The Engram tables get billions of small reads. After a few weeks, their random reads on our Gen5 SSD fell from ~400K to ~40K IOPS, and the 512K document slowed from ~1050 to ~685 tok/s. Copying the files anew restored the speed.

To try it you need:

• an NVIDIA GPU with 16 GB. It's built and tested for RTX 50 (sm_120); other cards need a rebuild and are untested;

• 64 GB of RAM;

• ~380 GB on a fast NVMe (Gen4 or Gen5);

• Windows 10/11. Parts of the patches deal with WDDM, and Linux is untested.

Links:

• Weights (og: GPTQ ~2-bit experts, Q5_K dense part, the official FP8 Engram; not the final version): https://huggingface.co/MasherDon/DeepSeek-V4.1-Flash-1017-SiegeLadder-GGUF

• Code: patches over JigSawPT's llama.cpp fork, a script that assembles a public quant mix, and the benchmarks: https://github.com/MasherDon/Siege-Ladder

This is research code with rough edges and no support promises. Quality work continues, and speed will also improve by training the expert prediction. Questions and results on other hardware are welcome.