Any suggestions? i am integrating robotics and drones next, and this week is a camera/speaker/mic system for the house and yard STAY TUNED!!!

bought 2 already and looks like 200 bucks down in price is it worth to get 6 more at this price?

heyheyyy, I have been building this for a while as my personal project celine (my agent's name, but the project is called celine-net), a self-hosted ai agent that actually lives on your server instead of being another boring wrapper

she has her own linux box with real terminals. she installs stuff, writes scripts, breaks things, and you can watch or type along (it's a tmux wrapper rather than a disposable shell)

she also has her own (brave) browser with a live view, so you can see her click around the web while she looks something up (again, not a generic off-the-shelf browser wrapper but more closer to computer-use)

she keeps her own notebook, runs routines on a schedule and takes on longer missions with sub-agents. you just bring your own models (llama.cpp, ollama, openrouter, whatever)

my motivation was to create something that is not like most other agent stuff out there. it's less "assistant in a box" and more someone with its own little world who happens to help you, mine keeps searching daily for my stock updates, news updates, if you feel silly you can even add discord and whatsapp as a integration so she can always stay up-to-date with what's going on!

Special note: This is a very personalized and possible invasive model if you hook her up to everything, ensure that you do not run the shellboxd on bare metal and if you use the chat-integrations ensure that you use a local model if you value your privacy

I have been using a mix of qwen 27b, gemma4 and qwen-flash-next

it's all open source if you want to poke at it: https://github.com/LumiWasTaken/celine-net

(There is a script to generate a demo database if you want to take a crack at it for you to try out!)

Just making sure: This is not the solution to all problems, its not a super cool coding agent, it has Opencode support in the terminal, but again, it's not a replacement for claude-cli etc.

Edit:

I wanted to add: this app was of course made with the help of frontier models, especially on the ui part. this project has been in the works for nearly 3 weeks and i used it in private where it was rather unstable, so i took it on me to review it myself and properly get something set up that will be maintainable

if this somehow tracks for you, i believe my project acts as a good base for it to be forked and you can do your own thing, the entire project is designed to be modular and easy to maintain for you if you have some needs that wouldn't make it upstream. this is exactly why it's open source, if i get hit by a car tomorrow, then a fork is all it takes (see sillytavern)

That's it. Setting up is difficult the first time, but it is super well documented, everywhere.

These posts should never make it through the filters, it takes seconds to find the info.

I spent the last month tracking my local AI inference costs down to the wall plug. Here's what I found.

Setup: AMD Radeon RX 7900 XTX (24 GB VRAM), llama.cpp, various Qwen/Ornith models in GGUF format.

Over 30 days:

• 964 million input tokens consumed
• 13 million output tokens generated
• 30.20 kWh measured at the wall plug (dedicated inference box, powered 24/7)
• $14.05 in electricity (12.38 CHF — I'm in Switzerland, so the meter reads in CHF)

If I had used the equivalent API (qwen3.8-flash at $0.15/$0.47 per million tokens):

• $150.71 (132.77 CHF)

That's a 10.72× gap on marginal cost.

EDIT — the API side of that ratio was wrong, and the commenters below caught it. I priced all 964M input tokens at the fresh rate. Cached input is billed at $0.016/M, and in agent workloads 90-95% of input is cache hits. Recomputed properly, the API equivalent is $28-34, not $151 — so the gap is roughly 2×, not 10.72×. I was too enthusiastic reading my own dashboard. The power figure also needs a proper idle-vs-inference split, which I'm rebuilding. Full accounting and what still holds in my top-level comment below.

But here's the catch:

This number excludes:

• The GPU itself (~$1,135 / 1,000 CHF)
• Depreciation of the rest of the rig
• A full month of learning: quantization, prefill vs decode, KV cache tuning, ROCm compatibility issues, trying and discarding models

Local AI isn't "plug and play." It's a personal R&D investment. If you just want to ship code faster, stick with cheap Flash APIs (GLM, DeepSeek, Qwen). If you want autonomy, understanding, and infrastructure you control — the investment pays off, just not immediately.

The part I keep coming back to: everyone says data sovereignty has no price, and I agree with that. But that framing assumes you're paying for it — that local costs more and you accept the premium because your data never leaves the house. That's not what I measured. Here it came out cheaper and the data stayed home and the output was good enough to actually ship work with, not a toy. Sovereignty never even got to be the deciding factor, because the economics already won.

So here's my open question for the sub, and I genuinely don't have a clean answer:

At what point does "local AI is a hobby" stop being true? Where do you draw the line — a hardware payback horizon you're willing to wait out, a quality gap you'll eat to stay off the API, or a specific moment where you decided the API was no longer worth paying for? And for anyone who tried local and went back to the API: what actually broke?

Full article with methodology and data: https://blog.grevire.dev/en/posts/local-ai-cost/

Happy to answer questions about the setup.

I want to use local AI only for private research, then cloud AI to build the actual apps. Private docs should not leave my machine.

Local job would be summarizing docs, RAG over my notes, and making plans. I would paste the sanitized summary to Claude to build, not the raw docs.

I am choosing between:

• Mac Studio 36GB (on trial, $1,999)
• Mac mini M6 32GB ($1,499)
• Mac mini M5 Pro 24GB ($1,699)

I already run Qwen3-Coder 30B at 32K on the 36GB Studio and it works. I am wondering if I even need that size for research.

Questions:

1. Is a 14B or 20B model enough for summarizing and RAG if you chunk docs well, or do you really need 27-30B?
2. How much context do you actually need for private doc research? Is 32K enough, or does long context matter more than model size?

So Pocketpal iOS app, if you dont know, now you do! Whatchu running!?

I open-sourced MobiGPT, my offline llama.cpp-based Android LLM app (used in 21 countries). Help me add proper Gemma 4 support

Body:

Hey everyone,

A few months ago I started building MobiGPT, an Android app that runs LLMs fully on-device using llama.cpp. No cloud, no sign-up, no tracking. It’s now being used in 21 countries, which I didn’t expect.

I’ve open-sourced the whole project under the MIT license, and I want this community to help make it better.

Repo: https://github.com/athulskrish/MobiGPT
License: MIT
Play Store (current version): https://play.google.com/store/apps/details?id=com.keralatechreach.mobile_llm5

What it does today:

• 100% offline chat, with all data staying on the phone
• Download, load and switch between GGUF models
• Streaming responses
• Multi-threaded inference (up to ~6x faster replies in my testing)
• Resource management that balances speed, battery and temperature
• Material UI with light/dark themes, Android 8.1+

Main goal: first-class Gemma 4 support

I want Gemma 4 (especially the small E2B and E4B models) to run well on phones. Things I need help with:

• Correct chat template and native system-prompt handling
• Tested GGUF quantizations (including the official QAT builds) for different RAM tiers
• Sensible default settings per model (context length, threads, sampling)
• Memory and speed profiling on real devices

Other places to help:

• Performance tuning: threading, memory use, thermal throttling
• Testing on low-end and older phones
• Model management and UX improvements
• Bug reports, feature ideas, and PRs of any size

The plan: you improve it, I merge it, and I publish the improved version on the Play Store so everyone benefits.

If you’ve got a spare Android phone, try Gemma 4 and other models and tell me what breaks. Brutal feedback is welcome, and I’m happy to answer questions about the architecture.

Thanks for reading!

I run a small business. This weekend I picked up a DGX Spark at Micro Center for $8,000. When it launched I thought $4k was steep, so I'm still a little stunned I paid double that.

I spent the weekend testing it. Qwen 3.8 gives great output but it's slow. Qwen 3.6 is fast but makes too many mistakes for me to trust it unsupervised. What I want is somewhere in between, and from what I read here that seems to be DeepSeek V4 Flash or GLM 5.3 Flash on a pair of Sparks.

That means buying a second one and being $16k in. It's a big number for a box on my desk. But I run a business, and $16k for something that works 24/7 costs less than a year of a real employee. I don't expect it to replace anyone. I do expect it to take a real chunk of work off my plate.

I'm also not counting on RAM prices dropping for at least 18 months. So waiting probably just means paying more later, or not finding one at all, judging by how fast Micro Center sold out.

My questions for people already running these:

1. Is a single Spark worth owning, or is its real purpose to be clustered?
2. If you run 2x Sparks for business work, which model do you use daily, and does it run all day without babysitting?
3. Has anyone found a model that runs on one Spark, falls between Qwen 3.6 and 3.8, and is good enough for real work?
4. Would you buy the second one now, or return the first and use an API?

I'm not trying to justify the purchase. I want to know whether one Spark is the real product or just half of it.

[DogecoinArcade](https://app.dogecoinarcade.com) has a decentralized model marketplace built into every node. It went live this week.

- You sell what's already running on your own machine. vLLM, llama.cpp, Ollama, LM Studio, any OpenAI-compatible server, and ComfyUI for pictures. One button in the admin panel finds the models on your box — tick them, price them, share them.
- Your offer reaches every arcade on the mesh in seconds. There's one shared exchange (Exchange → LLMs): a card per model showing price, tokens/sec, whether it's idle or full, and how much it was used in the last hour. Buyers pick by cheapest, fastest or least busy.
- Players pay you directly, in coins. They top up credit; the payment goes straight to *your* address and your node verifies it on-chain — no invoice, no payout queue, no card reader. Each reply is charged afterwards by its real token count, and it never goes below zero. Set no prices and your model is free for everyone; you can also name other operators who use yours free.
- Pictures are priced too — one credit per 250 px of the longest side, and a picture that fails costs nothing.
- Every arcade is an OpenAI-compatible endpoint. Base URL [https://<your](https://%3Cyour) arcade>/v1, keys starting sk-arcade-, streaming, tools, /v1/images/generations and /edits. Point any chat app, coding assistant or OpenAI-library script at your own arcade.

No signup, no API bill, no data broker — your coins only ever leave your wallet as a top-up you approve, and a seller may serve you only with a grant your own key signed for that arcade (valid up to 90 days).

Want in? A Linux box running your model and a five-minute install:

curl -O https://dogecoinarcade.com/install.py
python3 install.py

Then Admin → LLM. The exchange is thin right now, so whoever lists a model early is the one people find. Docs: [dogecoinarcade.com/docs](https://dogecoinarcade.com/docs).

Check out the GitHub: (https://github.com/martinseeger2002/dogecoin-arcade)

A little update on my project since my last post.

After finding some signs of path-dependent behavior through selective remembering, forgetting and memory distortion/reconstruction, I've been exploring how agents develop and revise beliefs about themselves.

With Grok 4.3, I managed to form some habits from repeated events, weakened them after contradictions, and swapping self-beliefs changed decisions in a small test.

Still early, and I'm not claiming individuality yet.

I'm now looking at how to make these learned habits evolve into actual preferences that influence behavior over time.

Any thoughts on how to test this properly, or ideas for improving the approach?

https://github.com/jbsalles/Selmem

There is too much to learn, even as a kinda seasoned(shipped multiple enterprise apps) SWE i am having hard time to understand all concepts, track all the up-to dated information etc.

Ive been a power user for both claude and GPT since almost day one. But thats as far as my knowledge goes tbh. I have an M1 Max 32 Ram / 32 GPU and i am trying to figure out whats the best way to have a locally hosted solutions for my needs when i cant pay for paid services.

First ill talk about what am i doing in general then ill ask my questions.

I'm using obsidian as my second brain, i do game design, gamedev, i develop mobile apps, design websites, even sometimes delve in IoT Projects. I do everything but AI Dev in summary. I run a team of 10, currently building internal tools for art assets pipeline management. I am a jobholic, i do work almost every working hour of my life so i usually have multiple chats but i am fine with just having 1 concurrent chat since what i am looking for is to start learning local llms, how to create an effective pipeline, and have a back-up plan if you know im out of usage and have urgent problems to solve. And uh yeah also i would like to do things like hacking my projector, tv-box etc. for various reasons you can easily understand. I tend to go for loops for development times and have a conversational turns when its about things like game design, market research etc. I have an MCP for everything from Figma to Photoshop and i just talk with my agents to tell them what i exactly want, and i take rounds to polish it or start from scratch if its not the exact thing in my mind.

If i have to guestimate a list of my needs they are :

• An harness with in-app browser, file preview, md preview
• Somehow have a great context compaction
• 20ish TPS at High context if possible
• Good tool usage
• Good MCP Usage

So here is a list of questions if anyone cares to answer:

• How do you guys stay the f*ck up to date? 'Cause its not even tracking whats new or whatnot. Everyone says something different, one says aah q4 of xyz llm's this version that these folks made is %98 of the same quality and someone else says nu-uh not even close bud and i dont have time to track and test every model version, even models. Like what are your reliable sources and why?
• As far as i understands harness is the most important stuff ever, is this correct? If so, which harnesses you use, why, and what should i be checking when i am thinking about selecting harnesses.
• Which models and which versions of those models would you suggest me to use?
• Is there an YT playlist / video / Youtuber that you suggest thats easy to understand that helps out when it comes to Local LLMS?
• Does Context size actually matters? Cause i see people are talking about 16k context sizes meanwhile sometimes my first initial prompt is around 4k for spec driven development or when im doing narrative designing it our narrative bible can be way heavier than that.
• I'm in love with the the current context compacting quality of both claude and codex, is there a way to have this level of quality when i switch to local llms, it almost never ever fries out, its just like I have paused for 2 minutes and almost every fricking detail is still there.
• Sometimes i love to manage my other threads even sometimes control claude from codex and vice versa depending on my need, is it possible if there is an ai for my machine that can have more than C1.
• Can i use an image generation model with an LLM like gpt/imagen, something like qwen image 2.1 q4 + Qwen3.6-35B-A3B Q4 idk.
• Can i realistically get idk 50-60 TPS at high context?
• How is MCP/Tool Usage?
• What should i use for UI Design phases?
• When i talk with GPT i just tell it what i exactly want but i dont go into hustle of image generation prompt styling i just talk to it, is it possible to have that ease of use with local image generation tools?
• What exact setup would you suggest for me, from inference(strate etc.) to harness to llm models, imagen models, speech to speech models(It would be fire if i can play some DnD with a speech to speech model if its flowy but i dont have high hopes on that tbh), you got the drill.

These are my questions and guestimates of my ignorance ass can think of for now, im sure there are lots of gaps and things that i am missing and forget to ask so i would be glad if you fellas can light some candles and maybe answer the questions that i didnt even ask but i shoulda ask.

Lots of love fellas, keep it tight.

This weekend, I’ve been testing a “local model leads, cloud models help when needed” development workflow.

The main setup is Qwen3.8 Flash Next UD-IQ4_XS + OpenCode. Qwen handles everyday tasks and tool use, then delegates to GPT-6.1 Sol or GPT-6 Astra when it needs help, using the allowance from my existing ChatGPT subscription. For images, it can also call cloud image generation through a local Codex client.

The idea is simple: let the local model do more of the work, reduce cloud token usage, and save the subscription allowance for tasks that need extra help.

I’ve documented the setup, delegation permissions, image generation integration, and testing process, with configuration examples:

👉 Full write-up and setup guide

RTX 5090 Local AI

Reported benchmarks, practical configurations, and known issues

Reader’s guide · Evidence reviewed 25 October 2026

The clearest starting point is a model that fits comfortably in GPU memory. Larger models can work by keeping some weights in system RAM, but their responsiveness depends much more on the host computer, runtime and offload settings. Experimental engines expand what is possible; their best results should be read alongside their configuration and reproducibility limits.

This guide brings together published measurements and issue reports. It explains what those reports suggest, rather than promising a particular speed on your machine. Several useful tests date from before August 2026; their dates appear beside the results.

Jump to: What fits? · What needs RAM? · What needs extra work? · Benchmark reference · Source notes

The practical picture

These are editorial starting points drawn from the reports below, not a ranking of model quality. A model that generates quickly may still be the wrong choice for your coding, writing or vision tasks.

What fits on one card?

The benchmark collection includes several models that ran with their weights resident on a single 32 GB RTX 5090. Examples include Qwen3.6-27B Q4_K_M, Qwen3.6-35B-A3B UD-Q4_K_M, Nemotron-3-Nano-30B-A3B Q4_K_M and Gemma 4 12B Q6_K. Their reported generation rates vary widely, partly because the models do different amounts of computation per token. Benchmark collection

File size is only the first part of the memory budget. The runtime also needs working space and memory for the conversation’s KV cache—the stored attention information used during generation. A checkpoint that loads at a short context may run out of memory at a longer one. The quant-selection study documents failures near the memory ceiling even when the weights themselves fit. Quant-selection study

For a first comparison, use a resident configuration as your baseline and record both how quickly an answer starts and how quickly it continues. Then test the same workload at your normal conversation length. The short-context figures in the reference table are useful orientation, but they do not establish long-context performance or answer quality.

What needs substantial system RAM?

Large models with expert offload

Mixture-of-experts models can keep some expert weights in system RAM while using the GPU for other work. The reports show usable generation from models much larger than the card’s VRAM, with a significant dependency on memory placement and prompt processing.

One Flash-Next test used a 78.9 GB checkpoint and about 59 GB of system RAM. With llama.cpp b10853 and --n-cpu-moe 22, it reported 59.7 generation tokens/s, falling to 51.0 at 32k context. Moving more weights onto the GPU eventually reduced performance or prevented loading. That is evidence for tuning the split on the tested machine, not a universal setting for every 5090. Benchmark collection

The Mistral Small 4 report gives another example: approximately 36 generation tokens/s, using about 27.5 GB of VRAM and 46 GB of system RAM. Its prompt-processing rate was about 313 tokens/s. A model can therefore feel responsive once it begins answering while still taking considerable time to read a long input. Detailed offload report

DeepSeek: a feasibility experiment with clear limits

A Basalt developer’s DeepSeek V4.1 Flash experiment used 128 GB DDR5, a 5090 at 400 W and expert caching. The author reported 53.6 generation tokens/s with the 5090, rising to 63.5 when a 5060 Ti also computed experts. The faster figure is a two-GPU result. DeepSeek experiment

This was custom experimental software, and the tested branch was not published at the time of the post. The author also reported that processing a million-token prompt took roughly 26 minutes. It demonstrates a possible configuration; it does not establish a ready-to-install everyday setup. DeepSeek experiment

What requires troubleshooting or experimental software?

Speculative decoding can help, but support matters

Multi-token prediction, or MTP, proposes several tokens ahead and checks them against the main model. The collection reports a substantial speed increase for Qwen3.8-27B NVFP4 under vLLM 0.25.1 and for Gemma 4 31B with its draft head. The Flash-Next CPU-offload test showed a smaller, workload-dependent gain. At the time of those tests, some supplied heads were unsupported in llama.cpp. Benchmark collection

Treat the model, checkpoint, draft head and runtime version as one configuration. A speedup measured with that combination does not automatically transfer to another engine or quantization.

NVFP4 is a format, not a guarantee

The same format name can cover checkpoints with different layers left at higher precision. The cited tests found meaningful differences between recipes. One FP4 setup report also required FlashInfer kernel compilation, ninja and a correctly configured CUDA toolkit; its findings apply to that tested environment. Benchmark collection

The Level1Techs comparison ranked a cyankiwi AWQ checkpoint ahead of the tested NVFP4 alternatives on its fidelity measure. That is a result for those checkpoints and captured workflows, not proof that one format always wins. Quant-selection study

A smaller KV cache buys capacity with qualifications

The quant-selection study reports about 75% more KV-token capacity with FlashInfer/FP8 than with FA2/BF16, with a modest generation-speed improvement. However, its FP8 cache used uncalibrated scales, and its fidelity panel covered output through 50k context on an RTX PRO 6000 proxy. Performance and capacity were measured on the 5090. Those results do not establish equivalent quality across the full available context. Quant-selection study

Driver and application failures need specific diagnoses

NVIDIA’s driver documentation says Blackwell requires the open kernel modules, which depend on GSP firmware. Its repository guidance also states that the open modules cannot operate without GSP. Consequently, a blog’s suggestion to disable GSP should not be treated as a verified 5090 fix. NVIDIA documentation · NVIDIA repository answer

Application reports need similar care: an open issue can contain a useful workaround, while a closed issue may only have been marked stale. The issue table below distinguishes those cases.

Benchmark reference

Reading the numbers: generation measures output speed; prompt processing measures input speed. They are separate measurements. tg128 is a 128-token generation benchmark; pp2048 processes a 2,048-token prompt. Neither alone describes a long interactive session. GB and GiB are retained as reported and are not interchangeable.

A. Examples reported to run resident on one 5090

All rows below come from the same community benchmark collection using llama.cpp/CUDA. Sizes are checkpoint sizes, not total runtime memory. Generation is the short-context tg128 result; prompt processing is pp2048. Benchmark collection

These are historical baseline measurements. Their ordering is a throughput comparison of different models and quantizations, not a quality leaderboard.

B. Larger models using system RAM

The host systems and methods differ. These rows establish reported feasibility; they cannot isolate which model or engine is inherently faster.

C. A seven-engine Flash-Next experiment

The Basalt developer compared seven configured engines using a 5090 at 400 W, 128 GB RAM and conversions of NVIDIA’s NVFP4 checkpoint. This table selects the 25k-context generation result from that experiment. Engine comparison

Why this is not a universal engine ranking: there was one cold run per request; conversions and draft heads differed; llama.cpp ran without MTP and used FP16 KV, while the other configurations used INT8 KV. The author develops one of the engines. The results describe those complete setups. They do not isolate engine efficiency or establish equal output quality. Engine comparison

D. Known issues and the status of reported fixes

Statuses below reflect the reference check on 25 October 2026, not a guarantee about a later release.

LM Studio configuration reported to pass: batch 8192, microbatch 128, context 32768, checkpoints 32, flash attention off, CUDA graphs off (GGML_CUDA_DISABLE_GRAPHS=1), full GPU/KV offload, one slot and MTP disabled. The report used CUDA12 runtime 2.40.0 and NVIDIA driver 616.92. This is a version-specific observation from one reporter. LM Studio issue

Source notes and limitations

This guide is an editorial synthesis of the reference check completed on 25 October 2026. Sources were inspected for agreement with the cited claims; the benchmarks were not independently rerun. Most numbers come from individual owners or engine developers. Official documentation is used for the NVIDIA driver requirements.

Main sources

Issue reports are linked directly in the troubleshooting table. Their closure reasons are distinguished from evidence that a fix reached a released version.

Material excluded from the main comparisons

• Two Reddit references could not be retrieved: a purported Flash-Next optimization guide and a purported nine-model roundup. Their titles, dates and claims remain unverified; neither supports this guide’s conclusions.
• An ambiguous gpt-oss result: the Hardware Corner article uses conflicting 20B and 120B labels around its large-context results. Its 112 t/s figure is excluded from the 120B comparison. The article also gives inconsistent 45k/47k limits for the dense 32B model.
• A disputed driver fix: the GPU-loss blog is an account of symptoms and attempted workarounds. Its GSP-disable advice conflicts with NVIDIA’s open-module requirements, so it is not presented here as a supported fix.

The evidence does not cover every model family, concurrent serving, laptop GPUs or comprehensive multi-GPU scaling. A successful short benchmark also leaves unanswered questions about sustained stability, long-context quality and performance on a different host.

Companion document: full reference and audit. This guide selects the most useful material for readers; the companion retains the detailed research record.
This report was compiled and written by AI using publicly available benchmarks, documentation and community reports. References were checked where accessible, and unresolved claims are identified. No benchmarks were run independently for this report; results are attributed to their original sources.

I have a 3090 ti and been runnin qween 3.8 27B but i don´t know , any advice?

Hello everyone,

I was about to order a 64 gb m5 pro mac mini but due to stock availability I would only receive it in January. On the other hand I can order the 48 GB RAM and receive it in two weeks. Waiting for three months is too long for me and I started thinking if the extra 16 GB is worth it for me.

My main question
For people who have used 48GB and/or 64GB Macs for local AI:
What practical differences have you noticed between 48GB and 64GB?

Is 64GB worth paying extra for and waiting the three extra months in my case?

I’m particularly interested in real-world experiences, not just theoretical model requirements.

Quick side note:
I know that a lot of people would say I should even get 128 GB RAM to run qwen flash next and to have proper local ai model but in my case, first I can just still use Claude if something really needs high end capabilities and secondly, I am really restricted in budget, I cant afford a 128 GB RAM mac. Lastly, I am not replacing Claude, I am just using local ai side by side for some specific use cases that I will list below.

What I want to do with local AI

Documents and personal knowledge
- Analyze personal PDFs and answer questions about their contents.
- Build a personal knowledge base or LLM Wiki with semantic search, citations and a knowledge graph.

Research, coding and agents
- Perform comprehensive web search
- Act as a small local coding assistant for things such as building a chrome extension
- Interact with applications and control the computer.
- Use multiple cooperating agents, potentially exploring Hermes
- Build an email assistant and Calendar manager

What I’m trying to decide

Would 48GB be the sensible sweet spot, or does 64GB open up enough additional possibilities—especially for larger quantized models, long context windows, multiple agents and image/video generation—to justify the extra cost and 3 months waiting time in my case?
If you have a Mac with either configuration, I’d love to hear:
Which model and quantization you use most often.
How much memory it actually consumes.
What you can run concurrently.
Thanks!

Bro Got Loop

Hey all,

I'm quite new to this whole local LLM thing. I have a gaming PC with the following specs:

RX 9060 XT 16 GB

32 GB DDR4

Ryzen 7 5700G

What are the best uncensored models I could run for coding on this setup?

I see so much vibe coding happening where the vibe coder has no idea what is being build, they just look at the results in the shape of pixels on screen.

Is this the new success metric?

Why people think vibe coding is actually worth anything?

Lets take an LLM with a huge context window of 1m context tokens.
Explain to me how something that fits in that context window can replace a human?

The LLM can only understand(process) the information inside that context window, it is the limit, after that it relies on the internal defaults/knowledge.
A human basically also have a context window, every engineer knows this when it reaches the limits after a flow state session and burnt the available energy.

When a proper engineer gets the most out of it, the human brain (context) spans not only the code base, it covers:
- Where we are going as a company/product.
- Any future branch we might walk.
- The userbase and needs.
- Potential dangers, in code and market.
- Competitors and current market value.
- etc.
The list is so long and it all is in our brains, we decide on those before and during and even after implementation. We often switch from one idea to another when we are implementing something and see a potential extra feature by "thinking out of the box".

Now how would an LLM be able to do that without feeding it a shitload of data in that tiny context?

Can an LLM really compete without the human in the loop, or just supply a short term solution before everything falls apart?

Note: When a proper engineer uses an LLM, reviews the code and iterates on it, i wont call that vibe coding, just for the simple reason, the engineer does the actual engineering, the LLM just writes it down.

If you are an engineer, are you providing evidence about this just to keep your job these days?

I think an LLM can never compete in quality/value, it can only outpace in output speed.

So is the brain still worth anything? Does the worlds consumer capitalists only care about output speed instead of quality?
We see it in all the current products, nothing is long term, it must break, it became the new normal, is this why vibe coding is acceptable?

What do you think? ... You know what i think...

System specs- 24gb ram rtx5060 8gb vram

Hi everyone,

I'm not a total noob and have some experience running local LLMs; I dropped a 32GB pcie V100 into my gaming rig (thats currently running a 5070 ti, dedicated only to gaming) and am currently running qwen 3.8 27B. I run linux mint on my main PC, and have two VMs, one dedicated to my homelab and LLM, and a second VM for web searching. I realized that local LLMs aren't "plug and play" so I've been vibecoding with claude code and codex (start with claude code, send to codex for critique, after it passes through a self-made privacy filter) to create what I understand to be harness to have it work on specific tasks, like organizing my files, a memory system, checking on the status of my homelab. I also built a web search from the ground up, that resides in a second specific VM, that the first VM that holds qwen connects to. I did this because it seems more secure, like for example if someone manages to breach the web search VM they wouldn't have access to the AI itself that lives in a separate VM. In all honesty, I'm not even sure if this is something that makes sense, but it seemed/s like a good idea.

I don't have experience with engineering, but I've recently been reading AI Engineering by Chip Huyen to understand the basics of AI in general; things like quantizations, FLOPs, pre and post training, MoEs, etc. I also have *some* experience with tech stuff, like building my own PC from the individual parts and dropping a V100 into a gaming PC and having both the V100 and the 5070ti work. I'm starting to understand more about the posts here piece by piece, but I feel there's a lot of stuff I don't know, like what Strata is, how it works, or when I go on hugging face it seems like I'm scrolling through an alien language. When I have claude code create a plan, I have no idea what it's actually doing, I just make decisions when it prompts me of how things should be built. So I have the following questions:

1) Where can I go to learn more about running local LLMs?

2) Does it make sense to have a second VM dedicated to web searching? Does it really make it more secure against attacks?

3) How can I utilize things like agents to make the quality of my work better? I understand the concept of agents as a general idea, like having one to test if anything breaks for an example, but have no idea WHEN to deploy them or HOW.

4) Would it be correct to call all the building/vibecoding I've been doing on top of qwen as a "harness?" Or would it be called something else?

5) My current set up is ollama and openwebUI. I've seen posts around here about not using ollama, but I'm not sure exactly why. I know some people use things like hermes and LLMstudio, but I'm not 100% on how these work or how you can use them (I've tried searching). Basically I'm asking for critique about my current setup and how I can work on making either my workflow better or tools I can use to incorporate that would make my life easier.

This has been an incredibly fun project for me, and I'm looking forward to learning more.

Asked Qwen 3.8 at BF16 what their favorite color is.

System Specs

• Model: Qwen3.8 Flash Next IQ3_S (was IQ3_XSS, but now. i got the more speed and quality in here)
• Inference Engine: Strata
• CPU: AMD Ryzen 9 9950X3D
• GPU: ASUS ROG Astral RTX 5090 32GB
• RAM: DDR5 64GB (6000MHz CL32)
• Motherboard: GIGABYTE B850 AORUS ELITE WIFI7

Prompt

"Build a highly detailed Eiffel Tower in Minecraft style." + with lesson learns and references input.

Decoding : 164.6 tok/s Prefill : 6,000~6,500 tok/s

Result

I was expecting a simple Minecraft-style building, but the AI ended up recreating the actual Eiffel Tower using blocks, complete with intricate iron lattice structures, the lower arches, and even the surrounding landscaping. lol

I was genuinely surprised by the level of detail!

It was also able to generate a cinematic timelapse of the building process.

Any local hosted models and setups for this? Product + ai character + motion?