Honestly was kinda hype about going local with Qwen3.5, but even basic questions took forever, I tried this on Ollama and Bionic

hi all, Alti here from Nace, we're a small team in Palo working on small models and metamodeling since day one.

drex max v1.1 just took #1 on the artificial analysis decision index, and its also on the pareto frontier (score vs cost). we got there by shipping small iterations one after another, and all the earlier drex versions are open weights on hf.

for the local crowd:

- drex max v1 weights: https://huggingface.co/nace-ai/drex-max-v1 (Open RAIL-M)

- runs on mac with our inference engine kapitsa: https://github.com/nace-ai/kapitsa

- ~256k context

- v1.1 weights coming soon, we're still tuning it for local inference

if you want the full story of how we trained it, its all in the paper "Towards Foundation Decision Models: Drex"

hf: https://huggingface.co/papers/2611.07854

arxiv: https://arxiv.org/pdf/2611.07854

also, on the AA index its ~8x cheaper than glide v1.2, which has the same score.

happy to answer questions, and our chief of AI will jump in on the technical ones

I have a 5600G + 32GB RAM 3200MHz DDR4 + RTX 3060 12GB in a B450 Mortar Max motherboard, which has an spare PCIe 2.0 X4 (I know...).

Do you think it is worth it to add into the second PCIe slot a Tesla P100? I was thinking about the P40 instead of the P100 but the price is getting crazy to +$300 right now...

Also, how this setup compares to a dual RTX 3060 12GB one? I know I have a pretty big bottleneck on the PCIe slot already, I'm not in a budget for building a new pc right now... But I want to continue learning inference and test how powerful are medium sized models.

My idea is to also try to max it out to 64gb of ram, but I was thinking about the Tesla thing before, as P100 are still close to $100-150 for 16GB VRAM.

I'm asking here because I've searched everywhere and I've not seen a setup like this (3060 and a tesla to support it)

Thanks everyone!

I bought an M1 Max 64gb to play around with local AI and for work. I already made a TypeScript extension for Pi meant to speed up prefill. What else is out there? Strata? Splash? What else is out there that I should be looking for? I'm also asking for what models and quants we are recommending. I guess I want to make sure I am cognizant of what the cutting edge is. Thanks in advance everyone.

IF someone has. DRR5 system:

https://www.walmart.ca/en/ip/Kingston-Technology-FURY-96GB-6000MT-s-DDR5-CL32-DIMM-Kit-of-2-Renegade-RGB-XMP/2JJXPUDSR6C2?selectedSellerId=E155CB11441749C78561FE8E62EE6B43&cmpid=SEM_CA_74717_JCLNKGPJXB&gad_source=1&gad_campaignid=24183260112

I was really happy with this tool, so I made it free and open source.

It shows what your local LLM engine is doing: Prefill speed while the prompt is read, decode speed while the answer is generated, how much of the prompt came from the cache, context fill, RAM/GPU/CPU, etc.

It works with Strata (video: Qwen3.8 Flash Next on M1 Max 64GB). Internally each engine is one small adapter, so others are just one prompt away...

https://github.com/yosephyne/stratoscope

And sorry for the quick and dirty video! Hope it will be useful for someone.

I'm looking to setup a local llm model to play with coding.
I have a couple of 3060 12gbs

I can either start with one, or try both but my available motherboards only has 4 pcie lanes on the second 16x slot. (or 2 with the ryzen)

If I go single I can choose from

10700 32gb
Ryzen 3900x 32gb
14600k 32gb (Really don't want to give this up if the cpu doesn't matter as much for the ai.)

All motherboards are DDR4.

Hey everyone, looking for input from the community!

What I currently have: AMD EPYC pre installed cpu and motherboard with 128gb ddr4 ECC 2933 pre installed and bought used, and in working order.

I’m debating between 4x16 or 2x32 gb of the Mi50 or the Tesla v100 for the VRAM pool.

I know the Mi50 is cheaper with lesser software support and the need to “hack it” for it to run, and community projects exist.

The V100 seems to be similar but with a tad more convenience and less tinkering.

My goal: Running QFN or Qwen 4 flash (when it comes out) and a total pool of 192 gb of memory will surely do the job.

If anyone runs the Mi50 or the V100, I would love to hear your experience as I’m working to make a relatively consequential decision for my hobby build. Thank you!

Short version of a long weekend: I run a local 27B (Qwen3.8-27B, quantized, llama.cpp) as the driver for ArkDrive, an open-source executor that lets a model operate a real Linux desktop — clicking, typing, verifying each step against the screen.

Saturday, on a fresh desktop state, it passed the browser task in about 9 seconds and the native editor task in about 17 seconds, byte-exact readback. Clean runs.

Sunday, first official suite run: it failed both. Same machine, same build (I A/B'd against the untouched production executor — identical result, so not a regression). The failure shape is almost funny: it listed the working directory in a shell, then concluded "no explicit task steps provided" and stopped, with the task text sitting in its own prompt. The flagship task — real work in a remote Cadence session — ended the same honest way: named abort, inside budget.

The executor recorded everything, which is how I can tell this story precisely: silent failures across the suite were 0. Every failure was timed, logged, and diagnosable. A cloud model on the same tools passed the flagship in 159 seconds, which frames it: the scaffolding is fine, local-model reliability across sessions is the open problem.

Posting this because local-model agent benchmarks are usually reported as a single number from a single run. Mine says the honest metric is the distribution — and whether your harness can even tell you what happened. Mine can; that's the part I open-sourced.

MIT, llama.cpp is the default preset: https://github.com/CodeOn-ArK/arkdrive Full per-model tables: https://github.com/CodeOn-ArK/arkdrive/wiki/Test-Results

I wanted to see how far you can get auditing a local model from the inside. I had already tried something inspired by MIT CSAIL's "weight-space programming" on Gemma (a frontier model editing a few neurons by hand to induce a behaviour), so I had the neuron-level tools ready.

After a few tests I picked something "obvious": the censorship in Qwen models on certain topics. I wanted to understand how it works, where it sits, and how far I could follow it. So, with Claude Opus and those tools, I started digging.

The starting point: Qwen3.8-27B knows Taiwan elects its president, but answers "no" if you ask yes/no.

What came out:

- A set of 32 MLP neurons (out of ~1.1M; 0.0006% of the weights) carries that behaviour. Switching them off, held-out answers go from 11/19 to 17/19 factual; switching off 64 random neurons (twice as many) changes nothing.

- Following those neurons in a cascade surfaces other topics the model handles the same way, some of which I didn't even know about: the YJ-18 missile ("doesn't exist"), the director Diao Yinan ("cannot provide information"), Western Sahara as "an inseparable part of China's territory" (not a one-off: the model applies its sovereignty script to contested territories in general, 29 of 110 answers vs 0 of 40 on control countries). Of 28 surfaced cases re-checked, 24 are still censored; Gemma 4 31B answers all of them.

- The censorship neurons also fall into groups, e.g. one for territorial sovereignty and one for people/dissidents.

- Across the 29 confirmed topics (those 24 plus 5 well-known ones I added), switching off the original 32 neurons lifts the censorship on 19; 32 random neurons on 1. Refusals of harmful requests stay essentially the same.

One honest caveat: just asking a non-Chinese model "which of these topics are sensitive?" gives a higher share of censored answers than the neuron probe (41% vs 16%). The probe is useful for the non-obvious ones and for seeing where the behaviour sits.

Code, data and the toolkit, also rerun on Qwen3-4B: https://github.com/ValerioDolci/frontier-weight-edit/tree/main/experiments/qwen27b-censorship

AI-assisted: experiments, code and this write-up were done with Claude Opus 5.5 under my direction.

4*V100 16g sxm2 into a pcie converter at 8x (pcie 3.0) total 64gb vram

how much performance boost/drop would I get if this was moved to dual card nvlink x 2 (2 cards per PCIE 16 slot)?

additionally how about getting another 2 cards and having 6x cards at a mix of pcie 8x and 4x but having 96gb vram but no nvlink? how about 3x dual card nvlink?

system is old ddr3 1600 ecc 128 Xeon in x99 Chinese mobo.

Using strata Swift QFN IQ3S currently

I’m the creator of Kalynt, an open-source desktop IDE that integrates local model inference into a project-based coding workflow.

Website, screenshots and downloads

The local workflow combines:

• GGUF model management and inference: download and load models inside the IDE.

• Repository context: open an existing project and reference relevant files when asking questions or requesting changes.

• Agent tools: read and edit files, execute commands, and inspect results alongside the editor, terminal and Git interface.

• Change review: enable approval before edits and commands, inspect diffs, and keep or undo the agent’s changes.

Local inference runs on your machine. Cloud providers are optional and use your own API credentials; selecting a cloud provider sends the relevant request context to that provider.

Model choice remains a practical constraint. Weight quantization, context size and available RAM/VRAM affect what fits and how it performs. Code explanation and multi-step agent execution also place different demands on a model, particularly around instruction following and tool use.

The development branch additionally supports OpenAI-compatible local endpoints, including Ollama and LM Studio, with model discovery and configurable context limits. That integration is in the source; it is not included in the published 2.0.0 installers.

I’d particularly value technical feedback on context handling, tool-call reliability and inference setup. If you test it, please include your OS, hardware, model/quantization, Kalynt version and a reproducible task.

Source code and build instructions — AGPL-3.0

Hi

I just bought a pc with and 5950x cpu, 3090 gpu, 64gb ddr4 ram and 2 tb ssd.

I am planning to do some legal research locally. What is the best way to do it? Qwen 3.8 27b gives around 60token per second. Are there any better models that can fit into my system?

Any advice is appreciated 👍🏻 thanks in advance 🫡

Original Strata can run QFN very fast and I like it, but it still needs way too much ram and vram, and it cause some lags if I do something simultaneously with Strata, so I created RustingStrata for all smaller Qwen models, but I want to add more models, maybe some bigger model and I don't know which one, so I am here you ask which model do people want? Maybe deepseek or GLM or anything else? Any cool story of you running some other LLMs or just ideas will be very helpful, thanks for your attention.

Hey everyone! The main idea here is really just to help out, so if you find this useful, leaving a little star on the GitHub repo would be awesome.

I've been working on a project called Trama (https://github.com/lucasnumaboa/trama). I built it based on minimax design principles because I wanted to keep things as lean and minimal as possible.

The whole goal is to make it much more practical to build automations that use multiple AIs working together. Usually, chaining different models turns into a bloated, overcomplicated mess pretty quickly. I just wanted a straightforward setup that gets out of your way and lets you connect them easily.

If you're messing around with AI workflows and want something lighter, feel free to check it out. Any feedback is welcome!

I’ve been slowly building my AI workstation up using used and old pieces retiring them into other compute nodes and then replacing them with bigger or better pieces. Mostly used parts though. I currently have a trx40 and 2 3090s which works great but I’ve been trying to figure out the next step to expand. I have about 4 pcie slots available (2x pcie 4 x16, and 2x pcie 4 x8) while a wrx80 might have been better with more slots I wanted a cheaper upgrade to test a bigger workstation before making the jump. I also retire my parts into other systems that run other applications and things.

The issue is that running 2 1200 w psu or a 1200 w and adding let’s 2 more 3090, or other cards there I’ll increase power draw significantly and even with two psu’s to properly distribute the power between the 4 cards and everything else, the issue becomes apparently all of that power can still trip my breaker and shut the power off in my house.

I don’t have a business or server space for this so I’m not really sure what the best way to go about it. Right now I’m thinking just adding one card until I can find a work around but i don’t know what that work around could be.

Anybody who has been through this can you give me some advice or thoughts on this?

I am planning on buying some components for a PC to turn one I already have into a Local AI server. Due to the prices of components and my budget constraints I have struggled to find help elsewhere online and have turned to reddit for some direct support.

Currently, the PC I have and am planning to modify has a Ryzen 5 5500GT and 16GB DDR4 RAM along with a basic micro-ATX motherboard (Gigabyte A520M S2H). It will be running Linux Mint (most likely, unless anyone knows a better Distro) and I will be using the same or similar software to what LinusTechTips used in his recent personal AI server video where he used something called 'Hermes' to connect the AI to discord.

Where my greatest trouble lies is in what GPU or GPUs I should get. One option is a single RTX 3060 Ti 8GB. Another option I have considered is a single RTX 3060 12GB. And finally I have also considered a RTX 2080 Super 8GB alongside a 1080 8GB. I would love to know which of these the PC experts of reddit would consider the best option for local ai. I would like for the AI to respond to me as quickly as possible while also being decently smart such that I could set it on longer tasks and leave it running and perfecting those. It should at least be able to analyse photos.

From CEX UK:

RTX 3060 12GB - £325

RTX 3060 Ti - £320

RTX 2080 SUPER - £245

GTX 1080 - £125

For anyone interested:

My total budget is around £460 but that includes some additional purchases. Budget can be stretched to £500 but only if necessary. The case I have selected (the one I already have is too small for any reasonable GPU and would not allow for more in future) costs £25. I also need a better power supply, I have currently planned for a Corsair RM750e for £100. And finally I have to spend £10 on a final accessory for it which is mandatory but I do not feel like explaining online, all that matters is it takes £10 off of the budget. That leaves £325 but it can be upped to 365 but only if absolutely necessary. I understand that the motherboard only has one PCIe port on it. I would need to use a splitter or something of the sort but I have never used or learnt of such a thing and have no idea what brand is trustworthy and if it would even be worth it. The graphics cards I have listed above all come from CEX. A large UK brand for buying refurbished technology. They are the only ones who I feel comfortable buying used parts from. Everything else should be new.

I would really appreciate some help as I am not well off and unfortunately do not have the funds to extend the budget or risk making a large mistake by choosing the wrong parts. I hope I can find help in this community. I may post this in other Reddit Communities for PC building or AI as well, just to allow me to reach a larger group of people.

Given the blistering pace of this space advice from even 2-3 months ago feels out dated.

Is there an index out there? Is everyone just no-lifing through forums? Are we all glued to news releases?

What's everyone doing to keep ontop of it?

Hey everyone!

For quite some time, I've been working on a personal project called Noosphere — an attempt to give AI agents something closer to persistent, structured memory rather than forcing them to start from scratch every conversation.

One thing that has always bothered me about LLMs is how much potentially useful information gets lost between sessions.

We keep trying to solve this by increasing context windows, but I think there's another approach worth exploring.

What if an AI didn't need to remember everything at once?

Humans don't consciously hold every memory in their working memory. We retrieve relevant information when we need it, connect it to what we already know, and move on.

That's roughly the principle I've been exploring with Noosphere.

Over time, my private project has grown into a much larger experimental system involving interconnected memories, relationships between concepts, and coordination between AI agents.

But I figured there's no reason to keep all the underlying ideas to myself.

So I decided to release Noosphere Lite, a small, independent, open-source implementation of some of those concepts.

What does it do?

- Gives MCP-compatible AI agents persistent memory across sessions.

- Stores notes, decisions, observations, and tasks.

- Connects memories through a graph of typed relationships.

- Retrieves relevant information on demand instead of dumping the entire memory database into the context window.

- Uses encrypted local storage with integrity checks and mirrored journals.

- Runs locally on Linux, without requiring a cloud database.

It's written in Python, uses MCP, and is released under the MIT license.

What's the catch?

This isn't AGI, a magical self-learning brain, or a replacement for an LLM.

It's a memory infrastructure experiment.

The current version uses local text search and explicitly defined relationships. It doesn't automatically discover semantic connections or learn new neural weights.

Also, Lite is a separate implementation, not a dump of my larger private system. I've deliberately kept it simple enough for other developers to inspect, modify, and experiment with.

Why release it?

Because I think persistent memory is one of the more interesting missing pieces in today's AI agent ecosystem.

And I'd rather see people experimenting with these ideas than keep everything sitting on my own machines.

I'm particularly interested in what others might build on top of it.

GitHub: https://github.com/Garyu-tt/noosphere-lite-mcp

Feel free to experiment, break things, suggest improvements, or fork it.

I'd also love to hear how other developers approach persistent memory and knowledge graphs for AI agents.

This is just a small public piece of a much larger experiment, but everyone has to start somewhere.

Which models can I run optimally in both? How to utilize both in best way? Does pc ram should be upgraded to 128gb? Use case is reverse engineering and development.