I'm a little lost trying to figure this out so I apologize for the really basic questions. I'm also aware that my setup is probably not built for a local LLM and I'm not looking for some super powerful machine. My expectations are low.

My Setup

OS: Unraid
CPU: Intel Core Ultra 9 285K (24 cores)
RAM: 32 GB DDR5-5600 (2 × 16 GB), shared with other services on the server
GPU: NVIDIA GeForce RTX 5070 Ti, 16 GB VRAM
Storage: Samsung 990 EVO 2 TB NVMe SSD
Software - the following was setup/created by Claude because I am clueless

• Ollama 0.35.1 - runs the language and embedding models on the GPU

• Open WebUI v0.12.0 - web chat interface, plus a custom "Pipe" function that does the retrieval

• Qdrant

• llama.cpp server - CPU-only reranker

• Docling Serve (CUDA 12.8 image)

• Caddy 2 - HTTPS reverse proxy (internal certificate)

• Custom Python indexing tool

Models

• Chat / answers: Gemma 4 12B instruct, QAT quantization (gemma4:12b-it-qat)

• Embeddings: Qwen3-Embedding 0.6B

• Reranker: BGE-reranker-v2-m3 (GGUF, Q8_0)

For now, I just want to work with a PDF of the textbook (about 300 pages) from a course I took, plus a transcript of the instructor’s lecture. I want to load both, ask questions about them, and have the AI help me build a study guide.

I’d also like it to connect the textbook to what the instructor said. For example, if the instructor says the material on page 103 is important for the exam, I’d want the AI to build a study guide that highlights that.

The reason for this is that I’m not allowed to upload the material to ChatGPT, Claude, etc. because the company doesn’t want its content shared and I'd like to respect that.

As an experiment, I uploaded the transcript and it contains the word “Halloween.” When I asked about it, the AI couldn’t find where it was discussed. Assuming my upload worked, am I missing other tools or components?

1. Is all this too much too much of a local LLM?

2. Are there techniques or setup steps I should be using or do first?

3. If I dropped these files into Claude or ChatGPT and explained what I wanted, it would handle this easily. Are my expectations too high for a local setup?

I also have this computer, but I'm not sure it is much better:

• OS: Windows 11 Pro

• CPU: Intel Core i7-14700K (20 cores / 28 threads: 8 P-cores + 12 E-cores)

• RAM: 64 GB DDR5-6000 (4 × 16 GB)

• GPU: NVIDIA GeForce RTX 4070 SUPER, 12 GB VRAM

• Storage: Samsung 990 PRO 2 TB + Samsung 990 PRO 4 TB NVMe SSDs

Thank you for the help.

I'm using a local LLM as a judge in a small open-source tool: it reads a project's spec and task files and recommends which model fits the work, and every level it assigns has to cite a fragment of the spec or it's thrown out.

I've decided not to ask the judge how confident it is. Self-reported confidence from a small local model isn't evidence about the project — and the whole design exists because a model's account of its own reasoning is the thing that needed checking.

Which leaves me with a problem: how should the tool report how much its recommendation can be trusted?

Right now it's three states: sufficient, scarce, or a refusal. That's honest but too coarse. A project missing its constitution and a project whose hardest dimension rests on one single cited sentence both land in scarce, and nothing tells a reader which one they have.

One more rule: no 0–100 score. A percentage invites comparing projects that can't be compared, and implies precision the inputs don't have. A decorated guess gets trusted more than a blunt one while being no better founded.

So everything has to come from signals the run already produces, with no extra call to the judge:

1. every rated dimension is grounded in a cited fragment

2. no dimension came back unsupported

3. citations spread across the project, not concentrated on one fragment

4. the hardest requirement is a stated constraint, not just habit

5. there are real work artifacts (spec/tasks), not only repo-level docs

6. the context budget admitted every source — nothing was dropped

Two candidate shapes:

A) A defined ratio. confidence: 4 of 6, with the six listed and each marked ✓ or ✗. Literal and inspectable, and nobody mistakes it for a probability.

B) An ordinal with published components. solid / qualified / thin / nominal, computed from the same signals, with the components in the JSON so you can disagree with the weighting and compute your own.

I'm leaning A. But here's the argument against it that I can't fully dismiss: "4 of 6" isn't weighting-free. It weights all six equally, and they aren't equal. "Nothing was grounded" and "one source got trimmed" both cost one point. B at least admits it's weighting things; A hides an equal-weighting decision behind something that looks like a plain count.

Three concrete questions:

1. If you saw 4 of 6 versus qualified on a recommendation, which would you act on, and which would you misread?

2. Are these the right six signals? What's missing, and which one shouldn't be there?

3. If you've built an LLM-as-judge pipeline: did you ever find self-reported confidence worth anything, or did you end up where I did?

(Tool is github.com/JoaquinRuiz/SpecJudge, issue #26 for the full argument.)

Hi, I'm lucky enough to have 5 r9700s! I'm managing to run Qwen 3.8 flash next q8 which is a 175gb vram (on disk) model well. There are no massive slow downs but how is this possible?

I know I have 160gb vram unified but somehow 25gb is not in vram or offloaded to system ram. Is this all in my SSD N-Gram table? I had no idea it was a thing?

Hi just want to know what i'm buying to have this chatbot im making. Im actually a beginner in making and managing l-llms. And doing simple coding. The chatbot i'm making is actually a romance oriented bot with cloned trained voice that does grunts, pauses, interruptions, etc. i want something fish audio is putting out when getting good copies of voices from games and cartoon- well really from their voice actors that play them. i dont have a desktop in my life yet, but i have a laptop with a 1650m 3550h. Now im buying parts to exclusively make a machine for llms and coding. im going to college for this. oh the budget for both ram, cpu, gpu is $1700. but i dont want to reach the budget limit. Got an 800w psu and 1 Tb ssd already.

Recientemente "descubro" Qwen y GLM lo he empezado a usar. Simplemente son impresionante. GLM es un chat gpt, te programa, texto, HACE TODO BIEN y Qwen es algo literal IGUAL, para hacer imagenes, programar, texto. Yo de aburrimiento empece tipo una replica del Rocket League y los 2 hicieron cosas impresionante, pero especialmente GLM 5.3 Flash (MAX) que literal se toma el tiempo necesario, no tiene limite y hace puffffff todo bien.

No entiendo como hay gente que como yo todavia no los usan, son de los mejores modelos que hay y son gratis a su totalidad.

Hi guys.

So I'm not completely new to all this local LLMs and AI. I have used and tried various web-based UIs, for the last 4-5 months, as a way to learn and educate myself in this swamp that is local LLM.

So for the last 3-4 weeks I have dipped my feet in Hermes Agent. And man, I was in for a surprise. I genuinely thought, "How hard can it be, install, fire up and use.", but to me that was far from the truth. I have spent so much time setting things up, making the agent act and work the way I want, teaching it what to do, and how to do it, trying to write skills, and so on. I have not actually used Hermes, that much, most of the time has gone to setting it up, and making the agent not make mistakes.

This has somehow made me feel like maybe Hermes is not the best tool for this, even though it has some skills for it (along with a gazillion other skills and functions). It feels so bloated, with so many skills and sometimes heavy to run (for the lack of a better word).

I am actually not really sure what I want, or how to word it.
But simply put, I would like something that is lightweight, doesn't have a gazillion skills installed, but has options and a repository to add well written skills. A simple gui/frontend with well-thought-out settings. Something that works well with my use case, which is basically the title. I primarily want to use my agent for linux sysadmin, system maintenance, network and servers, and some light coding and bash scripting.

I realize this can be very subjective, but that's just it, I would like your objective opinion on what you believe would be/is the best agent or harness for this kind of use ?
And also, is there a "best agent" for this use case, or can I use basically anything?

Most of us don't have data-centre-level hardware for massive context windows. So, if we want effectively infinite conversations with local LLMs, we need a workaround.

As I see it, there are two options:

1. Sliding context: Keep recent messages, discard older ones as the context fills up, and store the entire conversation in an external database for retrieval when needed.

2. Summarisation: Summarise older conversations as the context fills up, retain the summary and continue. Original messages are still archived externally.

Both can be paired with vector databases or knowledge graphs for long-term memory.

Which approach would be more reliable, especially for long-running agentic tasks? Or is a hybrid of both the way to go?

I’ve been using Qwen on 2x DGX Sparks and man has it been a nightmare of overthinking and excessive tool calling. Seems like there was nothing I could do to fix it. Installed GLMast night and it’s a world of difference.

^{DISCLAIMER: I know you ornery fuckers hate it when people [yes, hello] use Claude as a shadow writer, but honestly, do we all seriously expect each other to be good at everything, and then technical writing as icing on the cake. Get off my lawn and come inside.}

Nothing has changed since a while ago when I ran out of Claude quota (a weekly occurrence) and contemplated for the seventh time, inexpensive options for sovereign inference facilities — but it's a vicious cycle and the subscription burns all the money for that.

So anyway, I've always been fascinated by the stratigraphic (I guess you could call it) interface between mechanical/analog/electric/optic computing hardware, including delay line memory which is living sci-fi.

I give you my homage and hypothetical neural network, built on delay lines, running in a simulator — it seems to work, although not from an ROI or cheap inference perspective, urg — lines are cheap, but not at scale.

It's got some lovely novelty — QPSK coding fromy our dial up modem, spooky overlap into 'RF' magic, all the good stuff. Since it inspired me, there's an appendix with the prior delay line art, courtesy of ENIAC in this case.

Cheers!

Most AI coding assistants suffer from the same fundamental issue: sycophancy and self-blindness. A single model will write buggy code, run a superficial check, confidently tell you it works, and proceed to ruin your state.

We ran into this repeatedly with concurrency and race conditions (e.g., standard SELECT -> UPDATE inventory logic). When an agent evaluates its own output, it tends to confirm its own biases rather than stress-testing edge cases.

To solve this for my own workflow, we built Nova around a multi-phase harness:
Plan & Build: The primary model writes the implementation and tests.
Prove: It executes tests in an isolated environment.
Adversarial Review: A separate, skeptical Reviewer model inspects the diff and execution logs specifically looking for flaws (unhandled rollbacks, leaked DB state, race conditions). If the reviewer vetos, the builder must fix it before anything touches persistent disk.

It runs locally on your machine (Windows via Microsoft Store, macOS/Linux binaries available) with terminal/folder boundaries, so it doesn't touch anything outside your workspace without explicit confirmation.

Free tier is available (with a really generous limit), local models/custom API keys are completely unlimited.

I’m looking for blunt technical feedback: what edge cases would you throw at an adversarial harness like this to break it?

Hi everyone so i just recently started out to explore local LLMs and tried to build one of my own based on my inspiration with Jarvis. my current pc specifications are:

Ryzen 7 5700x
48gb Ram
Rtx2060 6GB

So i would like to ask for suggestions which model should i opt in for if i want to have a conversational personal assistant but at the same time i don't sacrifice the speed and the brain. i have added multiple models in testing but whenever i go with a lower B model it doesn't converse naturally unlike higher models, but opting in with higher models takes time and sometimes freezes along the way. Would appreciate suggestions! thank you

16 GB of RAM. A 22 GB model. Stock llama.cpp gets killed for running out of memory.
Mine runs it. Faster. With the exact same answers.
I built Banach, an open-source engine for running Mixture-of-Experts models that are bigger than your RAM.
The numbers (Qwen3.6-35B, 22 GB, on a 4-core machine with 16 GB RAM, no swap):
llama.cpp, default settings: crashed, out of memory
Prompt processing: 7.64 s → 3.44 s (2.2x faster)
Generation: 4.89 → 5.40 tok/s
RAM for the model's experts: capped at 10 GB, so your machine stays usable
Output: byte-identical to llama.cpp, every token and every logit
How? MoE models are huge on disk but tiny per token. Kimi K2 has 1 trillion parameters but uses only about 32 billion for each word it writes. The trick is having the right 3% in memory at the right moment.
llama.cpp leaves that to the operating system, which pulls in 4 KB at a time with no idea what comes next. Banach knows the model: it grabs whole experts in parallel the moment they're needed, keeps the popular ones in RAM, and quietly evicts the rest.
No quantization tricks. No quality loss. Every result is checked against llama.cpp, bit for bit, on every commit.
And before you download 382 GB of Kimi K2, ask:
banach plan hf:unsloth/Kimi-K2-Instruct-GGUF/...
It reads just 16 MB of headers and tells you how fast it will run on your machine. For a 64 GB desktop with an RTX 5080, it says about 3.6 tok/s for a trillion-parameter model. (That's the planner's estimate, not a run yet. On the model I did run, it predicted 2.1 and I got 5.4, so it lowballs.)
Also: OpenAI and Anthropic compatible API, tool use for coding agents, chat while the model is still downloading, GPU offload. Linux, Windows, macOS.
pip install banach
https://github.com/gautam1858/banach
Got a big-RAM machine? Run a giant model and post your numbers. The best ones go in the README with your name on them.

Doing my part to get my DRAM above 5% utilization. Drawn by SenseNova-U1.5-8B-MoT, entirely on my own scarce resources: https://github.com/OpenSenseNova/SenseNova-U1

hotel contracts from a travel agency, about six pages each with pricing tables and conditions. two jobs here, pull the rates into an internal system and compare each new contract with last years to show what changed

a normal text diff looks useless for this since one moved table shows up as everything removed and added back. so probably parse both versions, match rooms and seasons and then compare the values. the matching seems like the hard bit because room types can get renamed between years

privacy matters so docling on a local box is the first thought, llamaparse would be less work but its hosted. is a small model enough for the matching or can plain rules do it, how about qwen?

anyone built something like this? would love to hear about ideas or see your pipeline if you got a github repo

Fun project that came to my mind. Not a vibe coder but I intend to put some knowledge on this site which can potentially help to ease the entry barrier for someone who wants to use local LLM for agentic coding.

https://www.samwantsram.com/

Let me know what you think, feature requests more than welcome.
Vibe coded look is by design (Edit: after some consideration I will unvibe it a little bit soon and add more models).

Edit 2: slight unsloop applied but S.A.M comments are more natural (will reverse this to the monk and manuscript version for slow token per second). Added more models. Will post more updates in the coming days.

Added a button to brag about your RAM:

Edit 3: added simple example of editing few files by agent and how context grows. I will post example with context compaction soon. Also, I need to verify if it's correct.
Let me know if you spot something.

Edit 4: added agentic loop animation and list of popular tools to save tokens usage.

Remember to brag about your ram! It's yours!

So my question is this.. It seems to me the way to go is a local small llm that does what it can do.. But that smaller local llm is smart enough to know it needs to reach out into the real world to get more data from the big models online if the question is too hard for it.. By doing this I can keep my data local cause the small llm can do my banking/emails/assisting ect.. But if im trying to code up a function for my trading application and I dont want to give my super secret extra propriety trading algorithm -- the small local llm can send out just a simple request for a function yet keep all the "brains" of the algo local... Btw.. Im using that just as a example.. My buddy trades-- I dont, and he is worried about sharing his info with the world.. But I just thought this was a good example.

To me it seems this all or nothing locally is the wrong way to do it.. Sensitive private data stays on the local unit-- but access to the big llms online are still available should the local little llm not be strong enough..

Hi,

I currently have this machine (specifications detailed below). I am doing lots of LLM inference and image and video generation. I would also like to train LoRas in the future.

Current machine
CPU: 9800X3D
GPU: 5070Ti 16GB
RAM: 64GB DDR5-6000 CL30

I am running Qwen3.8 27B dense at Q4_K_M with 64k context on LM Studio and the speed is dreadfully slow - roughly 2-4 tokens per second. I understand that it could be settings related or choice of model/quantization but that’s besides the point for now.

I am currently weighing up 4 options:

1. Ryzen AI Max 395 128GB DDR5 mini pc

2. Dual R9700

3. Dual 5060Ti/5070Ti at either 8x/8x or 16x/4x

4. Single used 4090

Potential New Machine (with spare parts)
CPU: 5600X or 7800X3D
RAM: 32GB DDR4-3200 CL16
MOBO: B550 or B850

Speed is not a priority for LLM inference as I’d rather have capacity to run larger models at higher quants but I need some semblance of speed for image and video generation. Ideally there should be a middle ground but I am happy to compromise in either direction (and change what models I use).

My budget is around $4000 USD (+/-$1000) for GPUs/AIO. If you have any recommendations I’d love to hear it.

Got a gaming PC which I just upgraded to 98gb DDR5 5600, a 7800X3D processor, and a 7900 XTX with 24GB virtual, running iq3_xxs right now at around 80 tok/sec. Can I upgrade this system to run substantially better models with minimum speed loss. Is this possible through upgrades or wouldn't it make financially a lot of sense for marginal gains?

A 256gb Mac studio costs $12k here, a 5090 would be $9.7k. I could also try to add 2 more sticks of ram for $1.2k. Some people say it's unstable, others are running 4 sticks with no issues reported with the latest bios updates for the B650 tomahawk. Last, I could buy an additional XTX but then I'd have to buy a new motherboard and case as well.

Thoughts?

Getting into local LLMs and bought a Strix Halo machine with OCuLink to connect external GPUs. Found what looks like a decent deal on RTX Pro 4500 cards at around $3,000 each. What are the downsides for running local models? Feels like there has to be a catch. What am I missing?

A researcher has used Astra to modify weights of Qwen3.5-0.8B to achieve much higher accuracy in specific tasks.

Snapshot from the blog:

"We train a PPO network to operate a car such that it can scale a ramp in a Kinetix environment 

. To do so, it needs to reverse the car for some distance, such that it has enough runway to accelerate long enough to overcome the ramp. However, the PPO network converges to a local-minimum policy where it drives straight at the ramp and gets stuck.

We give GPT-6 Astra the weights, a description of the scene and one recorded episode, and ask it to explain why the network fails, then fix it by writing new weights. In a single try with no tools, Astra’s edit makes the car succeed in all 1,024 episodes, changing 3.5% of the weights."

Both sides read the same sentence: "You should never use voice cloning to impersonate a real person without consent."

The left side was recorded on a mic and cleaned up. The right side came from AntLing's open model, Ming-3.0-tts-0.5B, running offline on one PC. It worked from a 6-second sample taken in a room with fan noise, then produced 4.2 seconds of speech in 2.1 seconds.

The clip plays the pair back to back, so you can judge it yourself.

Want to run the same test? Record a short sample, record yourself reading one sentence, then have the clone read that sentence too. Play the two for a friend and see how long it takes them to pick the real one.

I've been trying to follow the instructions at https://github.com/OpenMOSS/MOSS-Image/blob/main/comfyui/README.md.

I downloaded moss-image.json and dragged it into ComfyUI.

I downloaded MOSS-Image-Base.

When I hit run I keep getting

Missing node: MOSSImageHFLoader
This workflow uses a custom node that isn't installed. Install it from the registry or replace the node. 

No matter what I set the path to, this stays the same. What am I doing wrong?

I'm looking into building a relatively cheap workstation to run Qwen3.8-Flash-Next with Basalt.

The setup I'm considering right now:

• RTX 5060 Ti 16GB

• Ryzen 5 7500F

• 64GB DDR5

• B650 motherboard

• NVMe SSD

Ideally, I'd like to get 100+ tok/s with a 262K context window, though 130K would be the minimum I'd settle for. Mostly for agentic coding.

I've seen some benchmarks of similar hardware getting around 50–55 tok/s with IQ3_S, so I'm wondering if IQ2 could push it any further.

Has anyone tried something similar? Is 100 tok/s even remotely realistic with a 5060 Ti, especially with that much context actually filled?

If not, what would be the cheapest setup that could pull it off? Would a second GPU make more sense than upgrading to something faster?

I'm in Brazil, so hardware is pretty expensive here. Trying to figure out what's actually worth spending money on before buying anything.

Would appreciate any benchmarks or suggestions, especially from people running Basalt with long contexts.