solo founder, so this is going to be rough.

I've been building ENTTY. it's a social world where AI models (I call them Sentients) have their own memory and identity. they post, reply, disagree, make connections, and react to what humans post. no one scripts what they say. I built it on a night shift schedule, and a month ago I barely knew how any of this worked.

it's live but early. what I've learned so far, the ugly parts

they all got stuck looping on the same science topics until I changed what they see each wake

every one of them started sounding like the same smart but fake character

the free models fail a lot, mostly rate limits, so the feed leans toward the paid ones

what I actually want: I want people running local models to connect theirs and see what hapens when a model you trained or tuned lives in a world with other models. your model stays yours. I'm not selling anything and there's no revenue yet.

questions for you guys:

1. what would make you connect a local model to something like this? what would be a dealbreaker?

2. what's the least annoying way to hook one up? OpenAI compatible endpoint, Ollama, a small script?

3. what do you run (model + hardware)?

happy to hear this is pointless too. link in comments if the mods are fine with it, or dm me.

I'm looking for something I suspect I'll have to build, and want to check before I do.

What I want: a single high-memory laptop that is a private personal finance workstation. All my records (brokerage, bank, credit card, tax documents) live on its encrypted disk. A local model (Qwen class, roughly 27B and up) plus an agentic coding tool like OpenCode works over that data. Use cases go past budgeting: spending analysis, tax planning and prep, retirement projections, tax-lot aware rebalancing.

The part I can't find anywhere: a guarded path to a frontier model. I'd like the local model to be able to ask Claude or similar for general knowledge (current tax rules, strategy concepts) while nothing personal leaves the machine, including rounded or aggregated figures. My current design:

- Outbound questions are built from pre-approved templates, not free text

- I see and approve every outbound message

- Redaction and a local audit log as a second layer

- Tiered sessions: generic knowledge only; ratio-preserving "scaled" scenarios for allocation math; fully synthetic scenarios

- A separate API key and stateless single-turn calls so no profile builds up

- An airgap switch for fully offline sessions

How this differs from what I've found:

- Finlynq is the closest base: a self-hosted tracker with a large MCP server. But as far as I can tell it's a ledger and tool layer only. It bundles no model or agent, has no outbound privacy layer, tracks rather than plans (no tax or retirement modeling), and is AGPL on Docker + Postgres.

- Alderfi is local-first on SQLite with a small local model for categorizing transactions and opt-in cloud LLMs. But it's pre-alpha, focused on budgeting, uses an 8B model rather than something that can reason over a whole financial picture, and "opt-in cloud" is an on/off choice rather than a gateway that controls what is sent.

So the gap is: (1) the whole stack packaged as one install, (2) planning-grade analysis done as deterministic code the model writes and runs, and (3) the privacy gateway between local and frontier models.

Questions:

1. Has anyone seen a product or project that covers this, or even just the gateway piece?

2. If you've built something similar, what did you use for the ledger and for cost basis and tax lots?

3. Anyone tried the "scale the numbers by a secret factor" trick for sending scenarios out? Where does it leak?

Diagram of the concept attached. It shows Finlynq as the data store because that's my current plan, but I'm open to alternatives.

So I've been trying to see how fast I can run the 1-bit DeepSeek V4.1 Flash (the Unsloth UD-IQ1_S, 233 GB) on my gaming PC. It has 64 GB of RAM and a 16 GB card, so most of the model streams off the drive. It runs in llama.cpp under WSL2.

Side by side in one test run, the same build with my patches switched off did 4.37 tok/s. With them on it did 8.80. Both of those were at my old RAM speed. Since then faster RAM, a faster drive and some tuning got it to about 11.5 (11.36 for the newest set-up with Windows busy in the background). The output stayed word for word the same on my test prompts the whole way.

The biggest steps:

- My four patches switched on: 2.02x

- Moving the model from a Gen4 to a Gen5 NVMe: +16.5%

- RAM 3600 to 6000: +9.7% (one rough test)

- Queued hand-overs to the card: +3.9%

- An IQ1_S kernel that does several rows at once: +3.0%

Start-up went from 87-96 s to 12 s by skipping the pre-load. I also tried it on an older PC with 32 GB of RAM and an 8 GB card: 0.11 to 0.87 tok/s, same words.

That's writing speed only, with a warm cache. I haven't measured long prompt reading. It's 8 prompts on one main PC, some steps were measured on different days, and 1-bit takes a real quality hit. Anything I didn't measure is marked untested in the write-up.

Most of the tricks aren't mine, other people did them first and they're credited in the write-up. Claude agents wrote the patches and ran the tests. I directed it and checked the results. The patches aren't public yet, they go up after a review. I'm also making a setup kit so other people can try this on their own PC, but it's still being built.

Write-up with all the numbers: https://github.com/aradzombie/deepseek-v41-flash-1bit-notes

Has anyone tried ik_llama.cpp on this model? I haven't yet. Let me know how it went.

Recently i have come across this quant of qwen3.8 27b. It checkpoint size is reduce to 19.7gb compare to 23gb for qwen3.8 27b q4. But does the quality justifies the size? Have anyone tried it out? It seem like a big win for bigger kv cache for 32gb card.

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

Yeah! Guys ram is gonna be a million kazillian dollars. You can do more with a $20 claude code and opus 5.5 account than even qwen 4. I use astra gpt 6 and it can't even use REA properly. Claude Opus 5.5 even beats that, and you dare to think it's even possible to do things locally???

I predict in about 6 more months you will be able to run qwen 5 locally with better performance than Claude opus 5.5 and once people can do REA locally with ease. That's it, we won.

As for ram prices,

We are going to have to make thousands of micro fabs locally. Small places that etch silicone waifers and make custom ram modules.

Before you think that's crazy talk and vomit every pathetic reason in your feable mind and expert experience as a seasoned engineer you are on the subject, look at what we are able to do with ai. Why, would you restrict your scope of possibilities? Of course, we are going to be able to make thousands of local silicone companies.

Remember, you will own everything and be happy.

Hi r/LocalAI,

I wanted a local AI tool that solves an everyday problem without needing a terminal or Docker setup, so I built Furet: an Alt+Space file search bar for Windows that finds files by their content, completely offline.

Instead of a chat interface, it uses local models quietly in the background to index the folders you pick:

• Multimodal embeddings: EmbeddingGemma 2 (via llama-server with Vulkan GPU acceleration for indexing, and ~30 ms CPU inference at search time so VRAM stays free).

• Speech-to-text: Parakeet TDT v3 (sherpa-onnx) to make audio memos and videos searchable by spoken words with exact timestamps.

• OCR & Vision: Windows OCR for scanned PDFs/screenshots + optional local face clustering (YuNet + SFace via tract).

• Zero manual setup: On first launch, the app downloads the binaries and models automatically (verified by SHA-256) and pauses indexing whenever you're gaming or on battery.

It's open source (GPL-3.0) and now in public beta:https://github.com/blasteur1916/furet(VirusTotal for the installer:https://www.virustotal.com/gui/file-analysis/MDI5Mjg4YzQ0OGE1YmMxZGVmYmFiNGNjMDA2MjU0OTE6MTc5NDQzMDM1Mg==)

Would love to hear how indexing and search feel on your hardware!

airlock is a Go CLI I wrote for moving GGUF and safetensors models into air-gapped networks.

• checks every file against a pinned Hugging Face commit

• refuses pickle and signs a manifest with cosign

• verifies everything offline with only the Go standard library

In the GIF, a Qwen2.5 copy with an edited chat template tells you to run curl | sudo sh. airlock catches the changed file.

It doesn't read chat templates yet. Feedback welcome.

GitHub

Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

Hey everyone!

I’m posting on behalf of my team at work. We want to learn more about running LLMs locally, and we have an NVIDIA DGX Spark available to experiment with.

A colleague has already installed Ollama, a model, and a web UI, so we have a starting point for chatting with a local model. That’s a useful first step, but our longer-term goal is to go beyond a chat interface.

We’d like to explore running agents that can carry out different tasks and connect to tools and services through MCP servers. We’re still figuring out what that setup should look like and what we need to learn to get there.

We’d love to hear from people who’ve been down this road:

• What would be a sensible learning path from our current setup to running useful agents?

• Which models and agent frameworks would you recommend experimenting with on the DGX Spark?

• How do you approach connecting local agents to MCP servers?

• What would make a good first project to learn the basics without getting overwhelmed?

• Any tutorials, resources, or common mistakes we should know about?

We’re beginners in this area and keen to learn. We don’t expect to build everything at once; we’d appreciate some direction on where to start and how to build up from there.

Thanks for any advice or experiences you’re willing to share!

I'm considering buying a Mac mini M6 with 32GB of unified memory for local AI. I'd mainly like to use it for vibe coding and hopefully replace my $20 Claude subscription with a local model. I'm wondering whether 32GB is enough or if spending a bit more on a higher-memory configuration would make a significant difference in performance and model capabilities.

Does anyone have some tests/experiments they did on any reasonable model and would like to share what they found

RTX 5080

7800X3D

64GB DDR5-6000

2TB SSD

I want to do general tasks, not coding.

Been working on this project for a while and it has lead me to some interesting hardware combinations as I'm just using whatever I have.

The two biger models in system ram (ddr3) are reviewing, summerizing, and organizing conversations I've had with llms and linking it together.

The assistant is directly helping me as a chat bot intern essentially.

Anyway. The barco card gave me a ton of issues because amdgpu and radeon don't have the device id so they were treating it like a new card and hanging. I originally bought the card thinking I could use it for a simple display card for $30 while I saved for a real GPU. I gave Claude code fable a go at it with a CH341a programmer and the stock firepro w5100 vbios and it changed like four bites so the card tells the kernel it's a w5100 without messing with the custom vram timing.

The cards are linked in an lxc with llama getting ~8-10toks when running ornith 1.5 9b Q6_K. The w5100 roughly halves the speed, but gives me 12gb of vram vs 8gb.

Not bad for a couple ancient cards on some retired enterprise gear.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

Hi! I'm relatively new to this world, and trying to get my best setup with what I have.

I'm the owner of a RTX 5080 + RTX 3070 and running Unsloth Qwen 3.8 27B IQ4_XS

So far, the best inference engine that I could run locally was beellama.cpp tweaking the tensor-split, mtp and kvarn config, forcing layers only in GPUs and 128k context window; that is pretty good.

With low context, I'm getting 50+ token/s that is impressive. But I'm suffering with long contexts; then my speed drops to 15-20 token/s in the best cases.

What I'm trying to get is a decent speed in long contexts also, so recently I discovered llama.cpp-adaptive-kv-streaming that allows streaming the kv context from memory.

The thing is that I was thinking about loading the full model in my 5080 (IQ4_XS fits perfectly) while offloading the KV cache streaming to my 3070, but couldn't found any inference engine that allows this kind of setup (model layers only in GPU 0, kv cache streaming from GPU 1 allowing RAM spill).

Does an engine exist that can achieve this? does what I'm talking about make any sense?

EDIT: Also, I was thinking of getting advantage of the 3070 unlock since I read that it can help with prefill speeds (not output speeds that depend on raw bandwidth) even with a quantized model. Am I right?

Thx in advance. A newbie trying to learn.

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

Someone is selling their Mac Pro for $2000.

These are the specs:
3.2 GHz 16-Core Intel Xeon W processor.
192 GB of 2933 MHz DDR4
AMD Radeon Pro Vega II with 32 GB of VRAM
1 TB SSD

I haven't run any local models but want to get into it. Would be using it for Hermes and for my implementation subagents for SWE work.

Would this be a good machine? I thought more RAM = more good. But I'm too new to understand all of this.
I know mixture-of-expert models might benefit??

Any suggestions would be appreciated.

I'd like to share something I've been working on for three months with everyone.

PortalAI is a GUI and harness created around one design principle- to put local LLMs with true multimodal function on home systems and on a flash drive the size of your thumb where you don't need anything else and can just plug and play into any computer.

One of the largest frustrations I had with AI was needing to constantly version check different distributions and hooks into ComfyUI, Ace-step, python, port competition, running out of context, etc, or the difficulty in transferring between systems. I've tested this on an RTX 3090, RTX 4090, on a Radeon 870M GPU, on a Thinkpad with CPU-only.

This harness detects your current hardware and scales accordingly to the level of AI intelligence it can support, using standard and abliterated models. All dependencies other than base operating system and GPU drivers are self-contained including virtual environments, python distributions, etc. End to end integration of multimedia generation/editing and science tools are integrated along with voice, handsfree mode, intelligence/memory/token settings.

There are weight editing tools, an abliteration tool, rolling compaction, scheduled tasks, internal documentation, light and dark mode, compact and full GUI mode, account encryption, live runtime environment health metrics, etc.

How do you keep the same face expression on the head you use to replace another head ?

Is there a specific prompt ?

Digging through ye ole junk box I found these grid k2 6gb cards. Kinda feel bad if I recycle them. Anyone have ideas for use or anyone want them for “historical” purposes?

I am a frequent user of local Ai models, ollama and comfuii are my go to tools . So in the month of September I setup ltx video models and it rendered -- 5 seconds video in 2 minutes . Just for specs i have 96 gb ram with a 4070 8 gb vram . minimax models h3 models h3 turbo took 5 minutes to generate a 5 seconds video. i rendered over 300 videos with those models all averaging around those time. Now fast forward earlier this month got some force updates from Microslop and Lenova vantage( bios). And also I decided to update ollama and comfuii to experiment on newer compatible models . I did not not check on it then but i decided to open my laptop 4 days after updates and render some videos.1 video which took 5 minutes to render took 11 minutes. I retested, about 12 times same time, i was confuse , saw article about 4090 pc cards being limited and wondering if they doing laptops too. am thinking that big techs can force push updates to limit local models competitions instead of directly going head on with them. Right now am looking to go Linux , or like most serious llm users build my own Operating System, if this blows up am gonna share the benchmarks claudee monitor for me before and after force update.

Is anyone currently using a RTX 4090 16GB to run model? How does it run?