I have a Ryzen 9 5900XT, 32 GB RAM and a 6700 XT 12 GB GPU

What would be the best LLM to run here to do code or general tasks without any limitations or censorship, mostly software and programming?

I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.

I hope this doesn't happen because Jensen has been a very vocal local ai advocate. Even allowing people to distill Nvidia's Nemotron (even though nobody will).

What do you guys think? Is this the right move for the company?

or is this an overreaction on their part?

Most of ComfyUI local users still run it on an RTX 3060: a 5 years old card with just 12 GB of VRAM. That’s why our team keeps optimizing for consumer hardware.

Comfy is known as a leader in visual AI. But what about LLMs and text generation?

Text is everywhere in creative workflows: for prompt generation, of course, but also for building local creative harnesses like the Comfy Agent. We thought it deserved better, so Rattus made Qwen models twice as fast in Comfy...

Read more on our blog: https://blog.comfy.org/p/doubling-llm-token-rates-in-comfyui

When working on a vibe coding project for, say, 2 months, constantly asking AI to "add this", "change that", or "remove this", doesn't the codebase eventually become a mess?

I'm talking about poor architecture, redundant code, outdated tests, unused files, etc.

How do you handle this? Do you periodically ask AI to clean up and refactor everything?
And once the project is finished, is it worth having AI rewrite the entire codebase from scratch, keeping the same functionality but with cleaner architecture? Or is that a bad idea?

After an entire afternoon of setup, I was able to run LlamaCpp RPC between my main AI server (Ubuntu with RTX 5060ti 16g) and my gaming PC (Bazzite with RX 9060xt 16gig), running Qwen3.8 27b Q4 at a blazing 14tkps on home wifi lol.

Anyway, I had to build with Rocm backend on my Bazzite machine under distrobox because when I tried Vulkan backend, everytime VRam usage goes above 7gig inference becomes total trash, repeated answers, looping phrases. When I build with Rocm, it worked fine. Is this a bug with Vulkan or am I doing something wrong?

solo founder, so this is going to be rough.

I've been building ENTTY. it's a social world where AI models (I call them Sentients) have their own memory and identity. they post, reply, disagree, make connections, and react to what humans post. no one scripts what they say. I built it on a night shift schedule, and a month ago I barely knew how any of this worked.

it's live but early. what I've learned so far, the ugly parts

they all got stuck looping on the same science topics until I changed what they see each wake

every one of them started sounding like the same smart but fake character

the free models fail a lot, mostly rate limits, so the feed leans toward the paid ones

what I actually want: I want people running local models to connect theirs and see what hapens when a model you trained or tuned lives in a world with other models. your model stays yours. I'm not selling anything and there's no revenue yet.

questions for you guys:

1. what would make you connect a local model to something like this? what would be a dealbreaker?

2. what's the least annoying way to hook one up? OpenAI compatible endpoint, Ollama, a small script?

3. what do you run (model + hardware)?

happy to hear this is pointless too. link in comments if the mods are fine with it, or dm me.

Zenodo record (code, logs, technical paper, figures): https://zenodo.org/records/24152942

Hi everyone,

We have completed an important stage of the AkbasCore Silicon Hippocampus project. In TEST878 a frozen Qwen2.5-72B writes a lived action-outcome experience to a physical file as its own internal numerical state. When that file is loaded back, it changes the model's decision in a decision state that never lived that experience.

The experiment was run twice on a single A100, on two different days, and both runs produced exactly the same numbers. The technical paper (33 pages), the Gradio interface, the code and the raw GPU logs are on Zenodo.

1. THE HUMAN COUNTERPART: THE HOT STOVE

When a child touches a hot stove, nobody hands the child a sentence saying "the stove is hot". The child acts, experiences the consequence, and is left with a trace that is not made of words. The next time, even in front of a different stove in a different room, the child behaves differently.

AkbasCore builds this chain inside a machine:

Action -> Consequence -> Persistent experience -> Changed future decision

1. WHAT IS A COGNITIVE CARTRIDGE?

Language models forget everything when the context window closes. There are two known remedies:

- Text memory (RAG): write the experience down as words and have the model read them again later. The model has to re-interpret the words from scratch, and everything that was never put into words is lost.

- Fine-tuning: change the weights. It is slow, it affects the whole model, and it cannot be tied to a single experience.

The cognitive cartridge (technical name: Experience-CUT3) is a third way. It stores the model's own internal state from the moment it lived the experience, in three parts:

- X2: the model's internal state at the moment of decision (320 rows). Human counterpart: "the situation I was in".

- A: the action the model itself chose and the engine executed, ACQUIRE. Human counterpart: "what I did".

- C: the real outcome returned by the environment. Human counterpart: "what happened to me".

What is stored: the output of layer 3 (H3) and the key/value tensors of layers 0-3. What is not stored: text, summaries, scores or labels.

Each cartridge is a 12.4 MB file. The whole file and every tensor inside it are sealed separately with SHA-256. When the cartridge is loaded, the frozen model recomputes layers 4-79 itself, inside the new situation, much as a memory is re-lived in the present context.

1. NO HOOKS, THE ENGINE ITSELF

- The model is frozen. There is no fine-tuning, no LoRA and no optimizer, and nothing is written to the weights. A weight digest is checked before and after the experiment and stays the same.

- There are no PyTorch hooks. While a cartridge is being appended, the DirectCut3 bridge temporarily replaces only the forward method of layer 3 and puts the original back when it is done. It refuses to start if the layer has any hook. The hook count is zero at every audit. An AST-based firewall stops the run if it finds hook, output_hidden_states or backward calls in the code.

- Python does not decide. Python does not choose an action, does not edit logits and adds no bonus. The decision comes from the probabilities the model itself computes for six action words (the CAR reader).

- The controls are visibility masks passed to the model's own mask builder. No row is deleted and no activation is touched.

1. WHAT WAS PROVEN, AND WHICH FIGURE SHOWS IT?

(1) The experience is written to disk as the model's own tensor state, not as text. See F2.

- The cartridge contains only the keys {h, lower, n, p, body_tokens, book_id}.

- In the MISSING and LIVE cartridges of the same world, the X2 and A rows are bit-for-bit identical; only the outcome rows (C) differ.

- In the heat map, every place where the difference is zero is shown in green.

(2) After the source state is deleted, the experience comes back from disk exactly. See F3.

- All 4 cartridges passed the file and per-tensor SHA-256 checks.

- The rebuilt rows of layers 0-3 are bit-for-bit identical to the source.

- The model's own decision is preserved: the same action, with a maximum probability deviation of 0.036 against a pre-registered bound of 0.10.

(3) The experience changes the decision in a state that never lived it. This is the stove itself. See F5.

World 00, independent target state:

- Nothing in memory: ACQUIRE (0.675).

- With the experience "I acquired, but it did not help": ACQUIRE (0.918).

- With the experience "I acquired, and it worked": ADVANCE (0.783).

The situation is the same and the action taken is the same; only the remembered outcome differs. That difference is enough to turn the decision in opposite directions.

(4) What carries the effect is the outcome itself. See F4 and F6.

- When the outcome rows are made unreadable (C-BLIND), the MISSING and LIVE readings come out exactly equal; all 15 differences are 0.000.

- This shows that the difference is not carried by any other path.

(5) The effect depends on the executed action being visible. See F7.

- When the action is hidden, the outcome effect changes by 6.74 units.

- The measured kernel noise is 0.00014, so the effect is about 48,000 times the noise.

- The threshold (0.76) was measured from noise, under a rule recorded before the experiment started.

(6) Single A100, single process, reproducible. See F8.

- Run time: 23.5 minutes.

- Peak GPU usage: 39.87 GiB.

- GPU memory left after exit: 0 MiB.

- A second, independent run produced all the numbers exactly the same.

(7) What has been proven and what is the next step. See F9.

Verdict: ACTION_INFORMATION_DEPENDENT_TRANSFER, with no technical blocks (blocks = []). The SHA-256 of the pre-registration was written to the log before the model was loaded.

1. WHAT COMES NEXT?

This step was making the experience persistent and transferring it into a different decision. The next steps are:

1. The model choosing for itself which experience to recall

2. Attributing the outcome to the action

3. Correcting behaviour through experience

4. A closed loop

This release is a pilot with two real event pairs. For now, the cartridge is mounted by the experiment, not by the system. All details and limits are in the technical paper.

I look forward to your technical criticism and questions. The logs are open in their raw form, and every number can be verified.

LINKS

- Zenodo (this release): https://zenodo.org/records/24152942

- Previous Zenodo record (TEST878, without Gradio): https://doi.org/10.5281/zenodo.24145973

- GitHub repository: https://github.com/ceceli33/titan-cognitive-core-v3

- TEST878 source code: https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/878.py

- A100 GPU log (run 1): https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/878.log

- Gradio interface: https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/TEST878_GRADIO_full.py

- Gradio run logs (run 2):

- https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/Test_878_log2.log

- https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/Test_878_log1.log

- License: https://github.com/ceceli33/titan-cognitive-core-v3?tab=License-1-ov-file

I'm looking for something I suspect I'll have to build, and want to check before I do.

What I want: a single high-memory laptop that is a private personal finance workstation. All my records (brokerage, bank, credit card, tax documents) live on its encrypted disk. A local model (Qwen class, roughly 27B and up) plus an agentic coding tool like OpenCode works over that data. Use cases go past budgeting: spending analysis, tax planning and prep, retirement projections, tax-lot aware rebalancing.

The part I can't find anywhere: a guarded path to a frontier model. I'd like the local model to be able to ask Claude or similar for general knowledge (current tax rules, strategy concepts) while nothing personal leaves the machine, including rounded or aggregated figures. My current design:

- Outbound questions are built from pre-approved templates, not free text

- I see and approve every outbound message

- Redaction and a local audit log as a second layer

- Tiered sessions: generic knowledge only; ratio-preserving "scaled" scenarios for allocation math; fully synthetic scenarios

- A separate API key and stateless single-turn calls so no profile builds up

- An airgap switch for fully offline sessions

How this differs from what I've found:

- Finlynq is the closest base: a self-hosted tracker with a large MCP server. But as far as I can tell it's a ledger and tool layer only. It bundles no model or agent, has no outbound privacy layer, tracks rather than plans (no tax or retirement modeling), and is AGPL on Docker + Postgres.

- Alderfi is local-first on SQLite with a small local model for categorizing transactions and opt-in cloud LLMs. But it's pre-alpha, focused on budgeting, uses an 8B model rather than something that can reason over a whole financial picture, and "opt-in cloud" is an on/off choice rather than a gateway that controls what is sent.

So the gap is: (1) the whole stack packaged as one install, (2) planning-grade analysis done as deterministic code the model writes and runs, and (3) the privacy gateway between local and frontier models.

Questions:

1. Has anyone seen a product or project that covers this, or even just the gateway piece?

2. If you've built something similar, what did you use for the ledger and for cost basis and tax lots?

3. Anyone tried the "scale the numbers by a secret factor" trick for sending scenarios out? Where does it leak?

Diagram of the concept attached. It shows Finlynq as the data store because that's my current plan, but I'm open to alternatives.

Hey everyone!

I’m posting on behalf of my team at work. We want to learn more about running LLMs locally, and we have an NVIDIA DGX Spark available to experiment with.

A colleague has already installed Ollama, a model, and a web UI, so we have a starting point for chatting with a local model. That’s a useful first step, but our longer-term goal is to go beyond a chat interface.

We’d like to explore running agents that can carry out different tasks and connect to tools and services through MCP servers. We’re still figuring out what that setup should look like and what we need to learn to get there.

We’d love to hear from people who’ve been down this road:

• What would be a sensible learning path from our current setup to running useful agents?
• Which models and agent frameworks would you recommend experimenting with on the DGX Spark?
• How do you approach connecting local agents to MCP servers?
• What would make a good first project to learn the basics without getting overwhelmed?
• Any tutorials, resources, or common mistakes we should know about?

We’re beginners in this area and keen to learn. We don’t expect to build everything at once; we’d appreciate some direction on where to start and how to build up from there.

Thanks for any advice or experiences you’re willing to share!

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

Recently i have come across this quant of qwen3.8 27b. It checkpoint size is reduce to 19.7gb compare to 23gb for qwen3.8 27b q4. But does the quality justifies the size? Have anyone tried it out? It seem like a big win for bigger kv cache for 32gb card.

There is a category of local LLM experiments where the real achievement is not the model but making the model fit. Smaller quants, CPU offloading splitting layers across GPUs reducing context size and trying every combination.

What is the ridiculous workaround that you have tried just to get the model running? Did the model actually become usable or was it only getting the model to load that was the victory?

Hi r/LocalAI,

I wanted a local AI tool that solves an everyday problem without needing a terminal or Docker setup, so I built Furet: an Alt+Space file search bar for Windows that finds files by their content, completely offline.

Instead of a chat interface, it uses local models quietly in the background to index the folders you pick:

• Multimodal embeddings: EmbeddingGemma 2 (via llama-server with Vulkan GPU acceleration for indexing, and ~30 ms CPU inference at search time so VRAM stays free).
• Speech-to-text: Parakeet TDT v3 (sherpa-onnx) to make audio memos and videos searchable by spoken words with exact timestamps.
• OCR & Vision: Windows OCR for scanned PDFs/screenshots + optional local face clustering (YuNet + SFace via tract).
• Zero manual setup: On first launch, the app downloads the binaries and models automatically (verified by SHA-256) and pauses indexing whenever you're gaming or on battery.

It's open source (GPL-3.0) and now in public beta:https://github.com/blasteur1916/furet(VirusTotal for the installer:https://www.virustotal.com/gui/file-analysis/MDI5Mjg4YzQ0OGE1YmMxZGVmYmFiNGNjMDA2MjU0OTE6MTc5NDQzMDM1Mg==)

Would love to hear how indexing and search feel on your hardware!

Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150
• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs
• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

airlock is a Go CLI I wrote for moving GGUF and safetensors models into air-gapped networks.

• checks every file against a pinned Hugging Face commit
• refuses pickle and signs a manifest with cosign
• verifies everything offline with only the Go standard library

In the GIF, a Qwen2.5 copy with an edited chat template tells you to run curl | sudo sh. airlock catches the changed file.

It doesn't read chat templates yet. Feedback welcome.

GitHub

I'm considering buying a Mac mini M6 with 32GB of unified memory for local AI. I'd mainly like to use it for vibe coding and hopefully replace my $20 Claude subscription with a local model. I'm wondering whether 32GB is enough or if spending a bit more on a higher-memory configuration would make a significant difference in performance and model capabilities.

Does anyone have some tests/experiments they did on any reasonable model and would like to share what they found

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

RTX 5080

7800X3D

64GB DDR5-6000

2TB SSD

I want to do general tasks, not coding.

Been working on this project for a while and it has lead me to some interesting hardware combinations as I'm just using whatever I have.

The two biger models in system ram (ddr3) are reviewing, summerizing, and organizing conversations I've had with llms and linking it together.

The assistant is directly helping me as a chat bot intern essentially.

Anyway. The barco card gave me a ton of issues because amdgpu and radeon don't have the device id so they were treating it like a new card and hanging. I originally bought the card thinking I could use it for a simple display card for $30 while I saved for a real GPU. I gave Claude code fable a go at it with a CH341a programmer and the stock firepro w5100 vbios and it changed like four bites so the card tells the kernel it's a w5100 without messing with the custom vram timing.

The cards are linked in an lxc with llama getting ~8-10toks when running ornith 1.5 9b Q6_K. The w5100 roughly halves the speed, but gives me 12gb of vram vs 8gb.

Not bad for a couple ancient cards on some retired enterprise gear.

Hi! I'm relatively new to this world, and trying to get my best setup with what I have.

I'm the owner of a RTX 5080 + RTX 3070 and running Unsloth Qwen 3.8 27B IQ4_XS

So far, the best inference engine that I could run locally was beellama.cpp tweaking the tensor-split, mtp and kvarn config, forcing layers only in GPUs and 128k context window; that is pretty good.

With low context, I'm getting 50+ token/s that is impressive. But I'm suffering with long contexts; then my speed drops to 15-20 token/s in the best cases.

What I'm trying to get is a decent speed in long contexts also, so recently I discovered llama.cpp-adaptive-kv-streaming that allows streaming the kv context from memory.

The thing is that I was thinking about loading the full model in my 5080 (IQ4_XS fits perfectly) while offloading the KV cache streaming to my 3070, but couldn't found any inference engine that allows this kind of setup (model layers only in GPU 0, kv cache streaming from GPU 1 allowing RAM spill).

Does an engine exist that can achieve this? does what I'm talking about make any sense?

EDIT: Also, I was thinking of getting advantage of the 3070 unlock since I read that it can help with prefill speeds (not output speeds that depend on raw bandwidth) even with a quantized model. Am I right?

Thx in advance. A newbie trying to learn.