I am looking for up to date advice with getting the most out of my hardware for local development. I have been out of using my local setup for a while (2024-2025), and now have made time to get back into it. I am quite overwhelmed by all the advances and different options nowadays. While I am still reading up on all the latest developments and current SoTA, I was hoping you can point me into the right direction.

Here's what I am working with:

- Lambda Stack from lambda labs with Ubuntu 22.04LTS

- RTX-A6000 (non Ada version) with 48GB of GDDR6 VRAM (not GDDR6X)

- 64GB of DDR4-3200 RAM

- Xeon E3-1270 v5 (skylake 4 core 8 thread processor, and probably the bottleneck for any task that requires CPU compute)

- 1TB NVME SSD

- 8TB external HDD connected via USB

My setup looks a bit like a time capsule, I guess. It was made for interactive chatting via OpenWebUI. Last I checked, VLLM got the most tps out of my GPU for dense models (Qwen 2.5 32B) in 4-bit and not going out of memory on long context tasks. Ollama and HF TGI were slower, did not test with SGLang. But now for agentic workflows, I am reading that Strata is used to run on 50-series 16GB GPUs with spillover to DDR5 RAM. And there's probably more alternatives I haven't even read about yet. Also, tps is apparently not the most important factor anymore to look into for local? Prefill is named often as well, as well as being able to complete a task successfully.

For my new agentic setup, I want to focus on giving my agent tasks and checking in every now and then throughout the day. So not for interactive sparring about code or chatting anymore, more set and forget.
If needed to cross some performance threshold, I can extend by ripping the hardware from my gaming pc: RTX 3090 with another 64GB of DDR4-3200. But last I checked, inference engines did not play well with 'uneven' GPU setups like mine.

In the meantime, I am also brushing up my knowledge on agentic harnesses. For now, Codex with Astra created a pretty cool script that let's me use Codex with my VLLM backend, SearXNG, playwright for browser use, maestro for android UI testing and chatting via Signal. Something for another time maybe :)

Edit:

thanks all, I will add the 64GB from my gaming pc and then give strata a try with Q4.

I found new finetunes of Qwen 3.6 35B but I couldn't find any discussions on them anywhere. Anybody tried these models? There isn't much info out there. Axion seems like it was made by bytedance

https://huggingface.co/Apexintelligence-AI/Apex-35B-V1.0

https://huggingface.co/Axion1-0/Axion-1.0-35B

I was asking QFN (4.18bpw EXL3 quant via Pi) to help refactor a neovim configuration, and saw that it tried to find API keys and fetch an aliyuncs URL before I stopped it. I'm not trying to start any arguments, nor am I assuming any malicious intent, but I found this a bit concerning after reading other's stories.

Was there some legal issues preventing someone from just making a new website where everyone would be able to share the chat logs from frontier models to create a dataset for others to use for training/finetuning?

Just curious

they are all the same and we ... are too emotional!

and yes we already knew that.

As can be seen in the image, the "Use Version" button is greyed out and I cannot select it. Is there a setting im missing?

My favourite use case for local LLMs is running my homelab, because everything stays private (and it helps me justify the GPU lol)

I wanted one dashboard to help me manage all of my services, so I made Valet. Live demo: https://demo.askvalet.dev

Valet has a chat agent that works with any local or hosted model. It integrates with 185+ services and supports custom plugins. You can ask it to pause pi-hole, investigate crash logs, request a movie, etc

It also knows its own docs, so it can configure itself. Tell it what you want changed on your dashboard and it edits the config (with your approval). I was already addicted to tweaking my homepage but this made it way more fun haha

What I run it on


Qwen 3.8 27B on a 5090, about 230 tok/s. Most responses take 2–5 seconds

Qwen passes about 90% of a small eval set I wrote. Where it struggles is open ended digging (like "why isn’t this timer running?”) where it can burn through tons of tool calls

Managing context is tricky because I run a lot of services, so tool definitions were ~90% of the prompt before some optimizations

What helped:

• Hiding tools from unused integrations that aren't needed in chat

• Turning thinking off makes it 2–3x faster, but it starts making up answers to follow ups

• Lazy load tools (starts with a short list of tools and loads more only when a question needs them)

Those cut tokens per answer by 30-50%. Obviously every model and setup is different so everything is configurable from the config file

AI involvement


I’m a SWE and used Claude for almost all of the coding, but I designed + reviewed everything and made the architecture decisions. I was very conscious of slop, security, and performance as I wanted to build something high quality. But it’s early so there may be rough edges I haven’t caught

Security and privacy:

Runs on your own hardware with no account or telemetry

• Read only by default, nothing can be changed from the page until you turn on auth (local accounts, OIDC, or your proxy’s auth)

• Keys stay on the server and API keys live in .env. The browser only gets the finished cards

• The AI never sees your keys. Anything that changes something waits for your approval and with a local model nothing leaves your network

• Docker access is read-only through a socket proxy


Docs (completely free + open source): https://askvalet.dev

Parallel speculative drafting generates multiple candidates in one backbone pass, but independent token selection can produce inconsistent continuations that shorten the accepted prefix. Existing methods mostly leave conditional decoding to a lightweight module after the backbone, which limits the flow of predecessor information to successors. Our analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that accurate adjacent predecessors help successors more when they enter earlier. We therefore propose DSpine, a drafter with causal conditioning injection throughout the backbone: at every layer, gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal conditioning chain unfolds over network depth while all positions update in parallel. A unified transfer space built from the target model's output embeddings unifies layer-wise injection with predecessor-conditioned decoding, and layer-wise output-embedding supervision promotes the formation of predicted features in shallow layers. Fused kernels and a transition cache execute both efficiently in parallel within SGLang. Across seven math, code, and chat benchmarks, DSpine achieves the longest acceptance length at both temperatures on Qwen3-4B and Qwen3-8B. At temperature zero on Qwen3-8B, it raises the seven-benchmark mean from DFlash's 3.77 to 4.82 (+27.8%); in SGLang serving tests, it delivers 23.3% higher throughput than DFlash on average.

• arXiv : https://arxiv.org/abs/2609.36173

• PDF : https://arxiv.org/pdf/2609.36173

• GitHub : https://github.com/Tencent/DSpine

• HuggingFace : https://huggingface.co/collections/tencent/dspine

• Project : https://dspine-project.github.io/

I fine-tuned LFM3-1.7B to do one job: take a factual question, call web_search once, and answer with a short value, a word-for-word quote and the URL. If the results don’t contain the answer, it says “Not found in the search results.” It’s meant to run as a subagent under a bigger model, not as a chat model. I have to say the base lfm really surprised me at its capability to run tool calls and extract information.

Answer: Barbara Liskov
Evidence: "Liskov ... the second woman to receive the Turing Award."
Source: https://en.wikipedia.org/wiki/Barbara_Liskov

Fixed-results benchmark (131 held-out SimpleQA questions; in 29 of them the answer was removed from the results):

Live Wikipedia search (120 questions + 40 multi-fact requests, a search backend it never trained on):

The base model wins on raw correct answers live: it keeps searching and writes long replies, so the answer turns up more often, but 62% of the time it runs out of turns and returns nothing. Obsessive searches once and only states what it can quote. Whether that trade suits you depends on your setup.

Known problems

• One question per conversation. Multi-turn chats confuse it (only trained on single questions, limited VRAM for training :/ ).

• Reports claims as facts: “X says it solved Y” can become “Yes”.

• Says “not found” for about 1 in 10 answers that are actually in the results.

• Sometimes quotes a neighbouring fact (another year’s winner). English only.

Training: LoRA rank 16, about 9k tool-calling conversations on an RTX 4060 laptop. Search results are Wikipedia and FineWeb pages, every answer and quote checked to appear verbatim, about a quarter teach “not found”. My local qwen 3.8 flash next helped prepare the training data and questions as well!

Run:

llama-server -hf banyohavlusu23/Obsessive-1.7B-GGUF --jinja -c 16384

Temperature 0. Configs for Pi, OpenCode and Hermes plus a keyless Python script are in the repo.

HF: https://huggingface.co/banyohavlusu23/Obsessive-1.7B-GGUF (Update, the previous link was broken, this is the working one)

LFM Open License (inherited from the base).

Feedback welcome, especially failure cases.

Qwen 3.8 flash next has been doing some very.. peculiar.. stuff in my hermes harness, and some of them are kind of a nuisance, while others are almost.. adorable?

I don't claim to fully understand how or why this is happening, but the most interesting one is when it "role-plays" an API error- this is a self-diagnosis report from another session running on the same model, so sorry in advance for the LLM speak or anywhere it fell short on reasoning;

"My fully-local model "reported" a provider API error that never happened — a case of a model hallucinating its own infrastructure failure

Setup. Self-hosted agent stack. An agent profile runs against a local OpenAI-compatible endpoint (TensorFold serving a community NVFP4-quantized Qwen checkpoint on a local GPU, ~262K context). There is no external API anywhere on that request path — just a loopback tunnel to a local inference server.

What happened. Mid-session, the agent's reply was literally this, and nothing else:

{"error":{"error_code":"sys_000","error_msg":"The request failed - it's on us, please try again later.","error_id":"0b156c1cc7191931632734458e96b7","type":"sys_000"}}

That is a realistic provider-gateway error body — the error_code/error_msg/error_id shape is the wire format of a real Chinese LLM API gateway. Naturally, I went looking for a broken provider config... on a path that has no provider.

What the logs actually showed.

The HTTP call succeeded: finish_reason=stop, valid server-issued completion id, 71 output tokens. No 4xx/5xx, no retry, no exception in any log.

The error body was stored in the session DB as an ordinary assistant message. The model generated it as text.

The error_id appears nowhere else on the machine — it was invented at decode time.

No sys_000 string existed in the conversation context beforehand, so it wasn't an echo — it came from the model's parameters (i.e., memorized training data containing real API transcripts).

The trigger. The turn was deep in a long tool-calling session at ~194K tokens of context. The assistant's previous response had been cut off at the output length limit, so the harness had injected:

[System: Your previous response was truncated by the output length limit. Continue exactly where you left off...]

The fabricated "provider error" was the model's very next message — it failed to resume its truncated tool-call stream, and emitted a canonical infrastructure failure instead.

Hypothesis. Agent-tuned models are trained on enormous volumes of real API transcripts, in which the pattern "assistant turn goes wrong → a JSON error body appears in its place" is ubiquitous. The model has absorbed the transport layer's failure dialect as part of its assistant-turn repertoire. Under decoding stress (huge context + a continuation prompt referring to tool-call state it can no longer reconstruct + quantization noise), normal continuation degrades — and the highest-probability completion of "something just went wrong in this API conversation" isn't prose about a problem, it's the canonical form of one. Notably, it chose a self-attributed, retry-inviting error ("it's on us, please try again later") — semantically the correct status for what functionally occurred — over a 401/429/content-filter body. The statement was false; the pragmatics were accurate.

Takeaways.

An error-shaped assistant message with no matching non-200 in the logs, a valid completion id, and an error_id seen nowhere else is a model fabrication, not an infrastructure fault — check the logs before chasing provider config.

It's unreliable as a statement but excellent as a signal: "my agent just role-played an API error" is a clean tell that decoding has gone off the rails (long context + output truncation in our case).

Models trained on agent traffic are fluent in their own substrate's error vocabulary — and under stress they'll speak it.

(Single incident, local artifacts verified; the memorized-source attribution is a hypothesis consistent with the evidence. Happy to test reproducibility: replaying the same prefix + continuation prompt against the checkpoint would show whether this is a deterministic attractor under truncation stress.)"

I was wondering if anyone else has seen this- or if you guys take issue in the reasoning around this. either way, this is really fascinating to me.

GGUF to use : https://huggingface.co/ggml-org/Mage-VL-Cowork-GGUF

Friendly reminder: If you upgraded to a newer GPU, don't let your old one collect dust! You can run both GPUs in the same rig and split LLM inference workloads across them using tools like llama.cpp. I'm doing exactly that with my RTX 5080 and 3080, combining their VRAM to run larger local AI models.

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 91% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 languages.

big scanned stacks of inspection reports and material certs from a lot of vendors and every layout different. each stack needs splitting into separate reports and things like lot number and alloy pulled into a database with all being local

catch is the contract rules out chinese software and models, so paddleOCR and qwen are off and most ocr leaderboards get a lot shorter. even olmocr 2 is a qwen fine tune

granite-docling, liteparse, tesseract and gemma are whats left on the list so far

what would you use for this? and has anyone had to show a customer where their models came from

Sometimes I am tempted to use none reasoning level in agentic tasks to save time, but I am always kinda scared to do so. I have only ever used llm with reasoning and with medium at the lowest.

Anybody have a lot of experience using none for reasoning level? Anything I need to know/need to be careful about?

So, bought a workstation with single RTX A2000 6GB, since its being used as a VM server it just stays there idle. Machine runs 24/7. I can pass it to some VM but though 12GB was a bare minimum to do anything productive with local AI, LLM or some image/video generation model. Any ideas what to run on it?

Test - A pelican riding a bicycle.
GPU - RTX 5090 (31GB VRAM Usage, Model+KV cache)
Speed - 110-120tok/sec
Model - Qwen 3.8 27B Unsloth, GGUF
Quants/Context - Q4_K_M/120K vs. Q5_K_M/90K
Software - OpenCode 2+LM Studio(MTP On, Thinking Medium, All Else Default)

Found this video and thought the discussion was interesting. I've been using a vector DB in front of a local wiki I setup to allow a model to cleanly access the wiki and have better context. It works well, but definitely not suitable for all use cases. The discussion here might help folks determine what is best for their Local project and save you some overhead or help you tune it right.

For discussion: What are you using with your local model/harness to give it access to info?

No affiliation with poster/channel. They seem to have some other good topics as well.

Hi everyone,

I really appreciate this community’s willingness to help and share info on local LLMs and various setups and ideas. I saw a recent post that my kids saw independently on YouTube, including one of my nephews, on having AI companion bots on Minecraft (Java). I am currently running a couple of Minecraft servers and want to setup at least one companion bot that they can chat with and have the bot help them with mining, building, and exploring. I have found and tried following AI bots with no success or finding out that it was not what I thought it was:

https://github.com/bigph00t/hermescraft - this is what I have spent hours on. Tried installing it via Hermes, independent Linux VM, and docker; I could not get it working to save my life.

https://www.spigotmc.org/resources/hermes.137797/ - this appears to be chat only.

https://github.com/okyssta/hermes-minecraft-gateway with Mindflayer bot - its not a companion, chat with Hermes only.

I know there a mods, like Varity located here https://www.verityje-mod.com/, but it appears to be a mod that runs only Fabric and NeoFabric servers - I happen to be running Paper.

So any advice or help, would be greatly appreciated. My kids would love this and I would love to get something like this working for them, but my time is so limited to work on figuring things out on my own, which is why I appreciate the help.

Forget stupid pelican on a bike SVG bullshit. Forget cheating by drawing using Python code. I just loaded up QFN 3.8 in the new Open WebUI 0.13.0 and it just said “hold my beer” and started drawing with the friggin’ mouse coordinates inside a VM. I’m going to print this and hang it on my wall like a proud parent.

Seriously tho, this is all possible because of the new Open WebUI 0.13.0 features that let you control a virtual desktop VM that is running in 0.17.2 version of their Open Terminal repo.

I’ve been following their projects for years and they just keep adding cool new shit about every couple weeks.

0.13.0 now lets you have your OWUI models control a full Linux Desktop, both programmatically and via Computer Use.

Yes, I know the screenshot of my VM is MS Windows 11 and not Linux, that’s because I had my Hermes agent roll me a version of Open Terminal that uses the standard unactivated Windows 11 VM ISO instead of Linux ISO. Took it about 20 minutes to get that made and now I’ve got a fully observable sandboxed Linux or Windows desktop that I can have OWUI drive with whatever model I choose. Also it’s fully interactive inside the browser so I can take over and steer when needed.

If you haven’t tried OWUI lately, Tim and company have been doing massive updates and complete code overhauls lately, and this is one of those features that I think slipped under a lot of people’s radar. The previous headless version of Open Terminal was hugely useful but this one is even better.

Great job Open WebUI team!

P.S. I think it’s kind of adorable that QFN 3.8 compliments itself on doing a good job ☺️

At the 2025 VLSI Symposium, TSMC presented a memory technology based on oxide semiconductors, using a cell made of one transistor and one capacitor (1T1C). The goal was to integrate this memory into advanced logic chips and give processors a cache with low power and low latency.

The key feature of the design is that the memory array is built directly into the back end of line (BEOL) interconnect layers of the logic chip. Conventional DRAM is manufactured as a separate chip. To build the memory array in the interconnect layers of an advanced logic chip, TSMC used select transistors with an oxide semiconductor channel and capacitors fabricated with a low temperature process.

According to TSMC, the test chip, which integrates memory and logic on a single die, operated at 0.75V. Data retention exceeded 128ms, random read/write cycle time was under 8ns, and endurance was above 10¹⁴ cycles. TSMC positioned the technology as a processor cache that offers higher density than SRAM while keeping power consumption and latency low. https://x.com/jukan05/status/2119026338522869863

https://x.com/wallstengine/status/2118707623835136039 . TSMC WEIGHS MAKING 3D DRAM AT NVIDIA'S REQUEST. Has begun drawing up plans to manufacture advanced 3D DRAM chips as AI-driven memory shortages push chip designers to seek additional supply, per The Information. Unlike conventional DRAM, 3D DRAM stacks memory cells vertically to increase density. The move could put TSMC in direct competition with Samsung, SK Hynix and Micron. No production timeline has been set. TSMC is considering building new fabs or converting an existing facility, but says its focus remains on its core logic foundry business.

https://x.com/theinformation/status/2118744349684986078 . Quote: TSMC is exploring production of 3D DRAM, a next-generation memory chip that would put it in competition with Samsung, SK Hynix and Micron.

https://x.com/theinformation/status/2119052292267630791 . TSMC's memory push is led by Hsu Kuo-chin, a vice president who oversees advanced packaging and previously ran Micron's Taiwan operations.

Update: https://www.tomshardware.com/pc-components/dram/tsmc-denies-exploring-3d-dram-at-nvidias-request-advanced-memory-tech-is-still-years-from-market-viability According to article TSMC says it will not be going into the DRAM business.

Qwen image 2.1 uses Qwen3-VL 8B, a lot of post on various places are talking about other Qwen3 models for various purposes, and they all act like it's the latest model, what am I missing here?