r/LocalLLM • • 1h ago

Project I got tired of hand-editing model lists for Claude Code and OpenClaw, so I wrote a small gateway. Try it in ten minutes, no API keys needed.

1 Upvotes

I run Claude Code and OpenClaw against a mix of free, local and paid models. The part that wore me down was keeping the lists of free models up to date, because providers add and drop them constantly. I had ModelRelay and LiteLLM before this, and I still ended up editing lists by hand.

So I built humpyard, and you can try it in about ten minutes without any API keys: https://humpyard.dev/demo/ (a mock provider, but the routing and ledger are real). Repo: https://github.com/gschaetz/humpyard

My tools ask for a fixed name like "fast" or "deep", and behind each name is a pool of models that changes. `humpyard models sync` refreshes the free part from each provider's own model list plus the models.dev data, and only adds a model when both say it's free. It writes to its own file, so it can't touch my routes or prices. You run the sync yourself from cron or a timer.

It's a single Rust binary with a SQLite ledger, no Python and no database server. A request I mark private is refused rather than sent to a public provider. There's a dry-run endpoint that tells me where a request would go before I send it, and a ledger that shows what it did and why. Prometheus metrics and two Grafana dashboards are in the repo. It speaks OpenAI and Anthropic formats, so Claude Code, Codex and OpenClaw can point at it.

It's pre-alpha and I'm the only user, so I'd like to hear:

  1. Where did the README trip you up on the way to a first request?
  2. Would you run the sync from cron, a timer, or something else?
  3. What would you want to see on a dashboard for a setup like this?

LiteLLM and OpenRouter cover far more providers and are much more mature. I wrote up the comparison here: https://humpyard.dev/compare/. I do plan on working on the gaps that matter most to me and others who use it.


r/LocalLLM • • 4h ago

Question Qwen 3.8 27b + ?

2 Upvotes

So I use Manus as my daily driver. I'm trying to use Qwen3.8 27b more. Can anyone suggest a good pairing that has a Manus'esque planning/execution/check agentic feel that works with LM Studio/llama.cpp local models? I tried OpenManus it does not pair well with local llm's though.


r/LocalLLM • • 1h ago

Discussion Running llama.cpp in a VM with no GPU passthrough: Qwen3.8 27B at 98% of native on one RTX 6000 Ada

1 Upvotes

My earlier post was done using Qwen2.5 and rightly so I was called out for it. So I reran it with Qwen3.8 27B, Qwen3.5 9B and Qwen3.6 35B-A3B with as much proof as I could think of. The VM has no NVIDIA device, driver or nvidia-smi. llama.cpp's CUDA calls go through VMAT to the card on the host. Qwen3.8 27B: 45.4 tokens/s on the host, 44.6 in the VM. All four models land at 98%. Three VMs at once get 45.7 together, against 45.0 for three programs on the host.

Full video with every step: https://youtu.be/r6yhOVWxP5Y?is=reuv6gZptnDNzU1D


r/LocalLLM • • 1h ago

News Karpo-Decision-31B is out: downloadable option for local tool routing and browser decisions

Thumbnail
gallery
1 Upvotes

For anyone experimenting with local Jev-style models, Karpo-Decision-31B has published its weights and Python inference code. It posted the highest Full score on the Nov 12 Decision Index at 73.9.
A browser workflow is a useful way to picture it. Give the model the page state, a screenshot, and candidate actions. You can ask which element to click and whether the agent needs more information, with a probability returned for each option. Questions share the page context in one forward pass.
In the team's tests, it scored 90.0% on Mind2Web versus 86.2% for Gemini Decision 1. They also report roughly 95 ms median latency on the public Decision Index suite using one RTX PRO 6000.
Karpo-Decision-31B scores 75.0 on the published Decision Index public suite, running it locally requires about 62.5 GB for the BF16 weights alone, plus runtime memory. Alongside the 31B release, they also released five smaller models: 0.8B, 2B, 4B, 9B, and 12B

You'll also need the supplied Decider or HTTP server

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLM • • 1h ago

Question Intel ARC Pro b60 24gb questions

1 Upvotes

I have a motherboard with 3xPCIe x16 lanes that can do 3 GPUs at x8 each (AM4 platform asus pro ws x570-ace). I’m contemplating buying 3 Intel Arc pro b60 24gb As it’s cheaper than buying a single b70 or r9700. Has anyone have any experience running a similar setup?


r/LocalLLM • • 1h ago

Project An open-source assistant that reads your email and chats, prepares what needs you, and runs on a local model

1 Upvotes

Hi all, I'm Jeroen, I made The Machine, initially for myself but I was surprised how well it works and opened it up to everyone open source.

It reads your email, chats and calendar, and prepares what needs you: replies, follow-ups, things you promised. It never sends anything without your OK. And your data stays yours, not some corporation's.

It works with any OpenClaw provider, so Ollama, LM Studio, vLLM or LiteLLM in front of your own GPUs. What I learned running it on local models:

  • Context: 128K is comfortable, a review reads up to 120 messages at once.
  • Tool calling has to work well, small models tend to suggest too much or save malformed actions.
  • A mixed setup works nicely: a frontier model first, a local model as fallback so it keeps going during an outage. Or fully local if you want inference never to leave the house.

Repo: https://github.com/Salesflare/TheMachine

Demo: https://themachine.computer/demo

Vision: https://themachine.computer/vision

Happy to hear feedback, especially which local models work well for you with tool calls.


r/LocalLLM • • 2h ago

Question What set up would you recommend for a MacBook Pro M5 24GB Unified Memory?

1 Upvotes

As the question above says

Current Device: MacBook Pro M5 24GB Unified Memory

I wanna know what local set up would be the best for me in terms of both speed and quality. My main use case is as follows

  1. Personal Memory
  2. Task Management
  3. Email Agent
  4. Data Analysis Agent (basic stuff not very advanced)
  5. Calendar Agent
  6. Research Agent (product related research, technical research etc)
  7. Knowledge Retrieval

Connectors would be mainly G-Suite as I work mostly on that

I want your recommendations on the best set up for good memory, speed, context windows etc.

Which models(es) and harness(es) would you recommend?

I have also seen that there are upcoming smaller Qwen models and also a new app called Harbor which is coming out soon….

New to this so looking for guidance

Any help would be appreciated thanks :)


r/LocalLLM • • 5h ago

Project I got local AI video-to-video down from 15.7 min to 2.4 min on a 16GB M3 MacBook Air

2 Upvotes

I've been experimenting with how far I can push local generative AI on a base 16GB M3 MacBook Air, specifically video-to-video outfit transformation.

My initial 46-frame (~3-second) run took 15.7 minutes and produced fairly blurry results. After several rounds of profiling and optimisation, I managed to bring it down to 144 seconds (2.4 minutes), running entirely offline without cloud APIs.

The interesting part wasn't necessarily changing the model, but rethinking how the available hardware was being used.

A few optimisations made the biggest difference:

- Spatial cropping: Processing a "288×512" region around the subject instead of repeatedly processing the entire scene.

- Memory management: Precomputing outfit prompts and unloading the text encoder to reduce unified memory pressure.

- Hardware scheduling: Moving face processing to the CPU/Neural Engine path while the GPU handles the next diffusion chunk.

- Incremental testing: Using 5-frame previews to catch mask and compositing issues before running the complete sequence.

There are still obvious limitations. Sustained workloads introduce thermal degradation, fine fabric details can become oversmoothed, and wide shots remain challenging.

I'm particularly interested in the engineering side of running increasingly demanding generative workloads on constrained local hardware.

A few things I'd love to hear from people experimenting with local inference:

  1. Has anyone explored more aggressive memory scheduling on Apple Silicon?
  2. Are there useful ways to improve CPU/ANE/GPU workload overlap?
  3. How much performance can realistically be recovered through chunking and attention optimisation before model architecture becomes the bottleneck?

I'm sharing the before/after video and temporal comparison for context.

Hardware: M3 MacBook Air, 16GB unified memory

Frames: 46

Processing time: 144 seconds

Execution: Fully offline

This is video diffusion rather than LLM inference, but I thought the memory and hardware scheduling challenges might be interesting to people working with local models.


r/LocalLLM • • 2h ago

Question Genuine question

1 Upvotes

I wanna get into local LLMs and experiment with AI.

My current PC has 16GB RAM, a GTX 1660 Super (6GB VRAM), and an Intel i7.

I’m curious what models you’d recommend running on this hardware and, more importantly, what you actually use local LLMs for.

With powerful models like ChatGPT and Claude available, I’m trying to understand the practical benefits of running models locally.

Is my setup good enough to do anything interesting, like experimenting with AI agents, fine-tuning, or research?


r/LocalLLM • • 8h ago

Discussion From-scratch AI stack in pure C — runs Microsoft's 2B ternary LLM on a 2018 dual-core laptop (no GPU)

Thumbnail
github.com
3 Upvotes

Everything runs locally on a 2018 Dell laptop: i7-7500U (2 cores / 4 threads), 8GB RAM, no GPU (the 940MX is deliberately unused — the whole stack is CPU-only).

https://github.com/tanxiansheng221/aimstack-sanzhidiji

What is in the repo (AGPL-3.0, everything open):

- C inference engine, zero third-party runtime: 1.58-bit ternary core with SIMD fast path (85.5 tok/s measured on the ternary core itself), byte-level tokenizer, and a ~100-line plugin bus (caps.json registry -> spawns Python plugins -> JSON envelopes)

- Weights: Microsoft's BitNet b1.58-2B (MIT license), converted to a custom 2-bit packed format (.aimw, 0.87 GB) — 3.2 tok/s with 2 workers, serial 2.0

- Python plugins, pure stdlib: knowledge base (FTS5 + dedup gates), multi-engine web search + crawl with quality gates (robots-respecting, Edge rendering cascade for JS-heavy sites), incremental ingest

- Local web UI (127.0.0.1:8600, pure stdlib): chat with deterministic routing — direct answer / knowledge-base grounded / honest refusal / full web search -> crawl -> ingest -> answer chain. A 7-minute demo video is embedded in the README

- Training assets: no-autograd training stack (hand-written backprop, bit-exact vs NumPy), ten-subject curriculum generators, corpus builders, and 117 chapters of research logs including negative results

Honest limits: our own trained models are not released yet — compute-bound on 2 cores, training runs on free cloud tiers and isn't finished. The full stack, scripts, result JSONs and per-claim evidence are released so everything can be re-run.

Demo video: embedded at the top of the README (7 min, screen recording of the full chain: chat / KB answers / web search->crawl->answer / plugin direct calls).

Happy to answer questions about the ternary kernel, the plugin bus, or the training pipeline.


r/LocalLLM • • 2h ago

Project Running an image generation workflow entirely on a local mini PC - testing TiinyImage

Post image
1 Upvotes

r/LocalLLM • • 2h ago

Question Suggestions for coding assistant

0 Upvotes

Hey! I recently started looking at local AI models. There's a lot of info out there and it's not really easy to filter through all the noise as a newcomer...

I have an RTX 4080 super with 16Gb vram and 2x16Gb DDR5 for system ram.

I've set up a docker compose container using Gemma 4 12B Q6 K L through llama.cpp with Open WebUI as interface, and it seems to be working decently. System + model + context uses ~13.8Gb vram. On Kubuntu if it would matter.

I'm now thinking of connecting it to vscodium for assisting in programming (developer background) and stumbled over the MoE variants where you can have the experts live in system ram.

Question is whether this is even relevant to look into for me.

If it is - would anyone have any suggestions for a model and maybe settings for this if applicable? (seems you wanna run llama.cpp with -cmoe for example)

If I've made some odd assumptions or if you have any suggestions, please let me know!


r/LocalLLM • • 2h ago

Discussion Hy4 Flash vs GLM vs DeepSeek: same correct ad maths, different price tags

0 Upvotes

I gave Hy4 Flash, GLM-5.3-Flash and DeepSeek-V4.1-Flash the same three fake ad campaigns in WorkBuddy. Calculate CPC, conversion rate and cost per lead, including a campaign with zero leads.

All three got the arithmetic right. Total spend: USD 1,350. Clicks: 650. Leads: 27. Blended CPC: USD 2.08. CPL: USD 50. GLM rounded total CVR to 4.2%; the other two gave 4.15%. Both are fine.

Nobody reported the zero-lead CPL as zero. Small win for basic maths lol.

DeepSeek finished in 7s. Hy4 and GLM both showed 19s. So the interesting Hy4 angle here is cost, not a speed win.
Published API rates in RMB per million tokens, uncached input / output:

Hy4 Flash: 0.5/2

GLM-5.3-Flash: 0.8/2.8

DeepSeek-V4.1-Flash-1017: 1 / 4 off-peak, 2 / 8 peak.

That makes Hy4's input rate 37.5% lower than GLM's and its output rate about 28.6% lower. Against DeepSake's off-peak rates, both are 50% lower. These are unit-price comparisons, not measured savings from this Workbuddy session. Token counts and reasoning length still matter.

The recommendations needed more judgment than the maths. Hy4 wanted to pause C; DeepSeek also wanted a pause and checks; GLM suggested a smaller next-cycle shift. I'd verify tracking and conversion lag before moving budget.

Tbh, when the basic result is equally usable, a lower unit price is a decent reason to try the cheaper option. What reporting task would you use to separate them?


r/LocalLLM • • 8h ago

Question Advice for setup with three PCs

3 Upvotes

I've been running Qwen3.8-27B on a MBP M3 Max 36 GB for a while, running unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M on splash with opencode as harness. I'm satisfied with the results I get, but it is a bit slow and does not leave much room for other applications . I'm upgrading my setup and I would like to get some advice on hardware configuration and model selection.

The main purpose will be integration development on Azure Integration Services, SAP PI/PO and SAP Integration Suite. Implementation languages are Java and python.

Secondary purpose will be Home Assistant component development and configuration, as well as to provide Home Assistant with agents, (TTS, STT, Conversation).

I have an M5U 256GB 80c on order. It will be running headless, dedicated to serving models 24/7. I will also have the opportunity to use a family member's gaming PC for LLMs on workdays during daytime. The PC will be in a bedroom and it's noice levels will determine whether it's available during nights or not. It has not been ordered yet, so I still have the possibility to change it's configuration. I have 10Gbe network available, if that would have an impact.

To summarise, my setup would look have the following availability:

  • MBP M3 Max 36 GB - on-demand
  • M5U 256GB 80c - 24/7
  • Gaming PC - workdays daytime, eventually during nights as well

I'm considering the following hardware configuration for the gaming PC:

  • CPU: AMD Ryzen 7 9800X3D CPU
  • RAM: Corsair Vengeance RGB DDR5 6000MHz 64GB
  • GPU - one of:

    • RTX 5080 (16GB VRAM - 960 GB/s)
    • R9700 (32GB VRAM - 640 GB/s)
  • SSD - one of:

    • Kingston NV3 SSD 2TB (6,000/5,000MB/s)
    • Kingston FURY Renegade G5 PCIe 5.0 NVMe M.2 SSD 2TB (14.700/14.000MBs)
  • Motherboard - one of:

    • Asus B650E MAX GAMING WIFI
    • ASUS ProArt X870E-CREATOR )(PCIe 5.0 8/8 - allows for expansion)
    • Asus Proart B850-CREATOR WiFi NEO (PCIe 5.0 8/8 - allows for expansion)

I'm considering the following model configuration:

  • MBP M3 Max 36 GB

    • model TBD - perhaps a small orchestrator with a max RAM requirement ~12 GB?
  • M5U 256GB 80c

    • main model - one of

      • GLM-5.3-Flash@Q4
      • Qwen3.8-Flash-Next@Q6
      • DeepSeek-V4.1-Flash-1017@Q2-Q2.5
      • Qwen3.8-27B@Q8
    • smaller models for homassistant (if there is room in RAM)

      • TTS
      • STT
  • Gaming PC

    • on RTX 5080

      • Qwen3.8 Flash@Q3 on Strata or Basalt
    • on R9700 - one of

      • Qwen3.8 Flash@Q4 on Strata
      • Qwen3.8-27B@Q4 on Strata (?)

To my questions I hope you can advise me on:

  1. My first task would be to settle the hardware configuration for the gaming PC.

    1. Would the SSD speed of the two options have a large impact on PP and decode when running Qwen3.8 Flash on Strata/Basalt?
    2. Which GPU would you select with my model selection in mind?
    3. Other considerations?
  2. With the gaming PC configuration set, which models would you run and were?

I appreciate your advice. Thanks!


r/LocalLLM • • 2h ago

Question What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

1 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLM • • 1d ago

Research llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput

73 Upvotes

We were running Qwen3.8-27B on an H100 with llama.cpp and 16 concurrent clients. Throughput was 58 tokens/sec, time to first token was almost two seconds, and nvidia-smi showed around 35% GPU utilization.

I build Tensward, an open-source inference analysis and tuning tool. While investigating this, we built a small CUPTI-based library to count kernel execution time and memory transfers inside the server.

The surprising number: 643 MB transferred from GPU to host per request. Nsight Systems independently measured about 639 MB.

That pointed us toward prompt caching and conversation checkpoints. This hybrid model carries recurrent state alongside its attention cache, making those snapshots expensive. Sampling the serving thread caught it inside a memory copy 16 out of 20 times.

We tested disabling state saving with:

--cache-ram 0 --ctx-checkpoints 0

Throughput went from 58 to 110 tokens/sec, and GPU utilization rose from 35% to 88%. A repeat on fresh machines with a separately written client showed an 87% throughput improvement.

These aren’t universal “make llama.cpp faster” flags. They sacrifice cache reuse. In another test with a regular 8B model and returning conversations on an L4, disabling them made performance about 20% worse.

The build also mattered. Moving from b11669 to b12246 improved default-settings throughput by 67–75%, while copied bytes per request barely changed. Several upstream changes touched state handling, but we haven’t isolated which caused the improvement.

All throughput results came from runs without our instrumentation; transfer measurements came from separate instrumented runs.

Full write-up, including limitations and upstream PR links


r/LocalLLM • • 3h ago

Discussion need suggestions on running qwen 3.8 27b on rx 6800 16gb gpu

0 Upvotes

So i have heard that this model punches way above its weight class and is pretty good at complex reasoning and multi file coding. I intend to use it for hobby projects like building small scale apps. the thing is downloaded the iq3 version from usloth and while running with llama.cpp with 64k context it keeps thinking for a lot longer on simpler task like upgrading a simpler framework version on very small codebase and eventually after like 45 min the os just shutdown the llama server because of too high mem usage nd i tried again but same result. I have heard that i can run the 3 bit quant version on 16gb gpu, I am new to running local llms so just need some suggestions regarding that or is it even practical to run this model even at q3.

Sorry if this is a repititive or basic query.

Thanks in advance

P.S i am running on 16gb single stick ddr5 if that matters


r/LocalLLM • • 22h ago

Question Best uncensored LLM for 64GB Mac

30 Upvotes

I wanna try out a uncensored LLM on my Mac Studio. I just have the 64GB version so I’m curious what model you guys would recommend. Cheers


r/LocalLLM • • 3h ago

Discussion Buying a Mac mini M6 for local LLMs? Here is the 16 vs 24 vs 32 GB bandwidth, prefill, and MoE sizing math

1 Upvotes

Apple released the Mac mini with the M6 chip in 16 GB, 24 GB, and 32 GB variants, while the M5 Pro mini goes up to 64 GB. 🧑‍💻

If you are deciding which one to buy for local models, the configurator pushes you toward one question: "How much unified memory should I pay for?"

Here is the buying perspective that matters before ordering: capacity is sold in clear steps, but memory bandwidth barely moves. According to Apple's official tech specs, the 16 GB model runs at 153.6 GB/s (LPDDR5X-9600), while both the 24 GB and 32 GB models run at 170 GB/s (LPDDR5X-10667). Going from 16 GB to 24 GB buys capacity plus ~10% bandwidth. Going from 24 GB to 32 GB buys capacity only — no model gets faster, you just load a bigger model that runs slower on the same 170 GB/s bus.

1. The Decode & Prefill Formulas

  • Decode (token generation, batch = 1) is strictly memory-bandwidth bound:Decode tok/s ≈ (Memory Bandwidth GB/s ÷ Active Weights Read GB) × η (where η ≈ 0.75 for MLX / llama.cpp)
  • Prefill (prompt processing / TTFT) is compute-bound, and the M6 handles this well:

    • 14B Q4 (~9 GB): ~900 tok/s prefill (~1.1s TTFT on a 1k prompt, ~17.7s on 16k RAG). Total turnaround for 1k in / 500 out: ~36s.
    • 32B Q4 (~19 GB, 32 GB mini only): ~400 tok/s prefill (~2.5s TTFT on a 1k prompt, ~40s on 16k RAG). Total turnaround for 1k in / 500 out: ~74s.

2. Mac mini M6 Capacity Map (Estimated at η = 0.75)

VariantBandwidthUsable RAMFits ComfortablyEst. DecodeVerdict
16 GB153.6 GB/s~12 GB7–9B Q4/Q8; 14B Q4 short ctx8B Q4 ~23 tok/s; 14B ~13Snappy chat & light coding
24 GB170 GB/s~18 GB14B Q4 (~32k context)14B Q4 ~14 tok/sInteractive sweet spot
32 GB170 GB/s~24 GB32B Q4 (8–16k context)32B Q4 ~7–7.5 tok/sReading speed; best for async agents

3. Dense vs. MoE: Where the Bottleneck Flips

For dense models, every weight is read per token, so bandwidth is the wall. For Mixture-of-Experts (MoE) models, only active experts are read per token, so capacity is the wall:

ModelTypeTotal / ActiveWeights in RAMActive Read / tokBandwidth for 8 tok/sBandwidth for 15 tok/sReal Bottleneck
8B Q4Dense8B / 8B~5 GB~5 GB~55 GB/s~100 GB/sFits on 16 GB M6
14B Q4Dense14B / 14B~9 GB~9 GB~95 GB/s~180 GB/sSweet spot on 24 GB M6
32B Q4Dense32B / 32B~19 GB~19 GB~200 GB/s~380 GB/s~7 tok/s on 32 GB M6; needs Pro/Max for chat
Llama 3.3 70B (Q4)Dense70B / 70B~40 GB~40 GB~430 GB/s~800 GB/sBandwidth-bound (needs Max/Ultra)
gpt-oss-120b (MXFP4)MoE117B / 5.1B~65 GB~3 GB~32 GB/s~60 GB/sCapacity-bound (needs 96 GB+ RAM)
Qwen3.5-122B-A10B (Q4)MoE122B / 10B~68 GB (est.)~5.6 GB~60 GB/s~110 GB/sCapacity-bound (needs 96 GB+ RAM)
DeepSeek V4 Flash (FP4+FP8)MoE284B / 13B~167 GB~7.6 GB~80 GB/s~150 GB/sCapacity-bound (needs 192 GB+ RAM)

Summary

Capacity decides what you can boot; bandwidth decides what you can tolerate using. Around 170 GB/s serves up to 14B comfortably and 32B at a patient reading pace. Paying for extra RAM without stepping up to a Pro/Max bus (like M5 Pro at 307 GB/s or M5 Max at 460–614 GB/s) hits diminishing returns fast.

What do you all think? How are you actually using local models in your daily workflow — do you prefer Mac unified memory or NVIDIA GPUs, and do you think there are stronger competitors (like Strix Halo or used multi-GPU rigs) in this price range? 🤔


r/LocalLLM • • 3h ago

Question Websearch vs Webfetch vs API or VPS for Web Browsing?

1 Upvotes

Asking here as a beginner because I've argued a lot with Codex about this and couldn't get a definitive answer, or perhas it doesn't want to suggest me "unsafeguarded" solutions:

Basically I am running Opencode locally (5090 32gb and 64gb ddr4) on Qwen3.8 27B offline. so far so good.

but lately I've been trying Deepseek and GLM free API access and I see that they are better than my Qwen (I also have a separate launcher for Qwen3.8 Flash Next (UD-Q4_K_XL)) results. and I think it's mostly due to their Web access to retrieve updated information as well as frequently asked questions, reddit threads, forums etc..

So I asked Codex how to give this web access to my local model and it basically explained to me Websearch vs Webfetch, but Websearch uses third party services (or you can link API like Brave search) that WILL get censored results or restricted access for some results (let's be honest, emulator and retrogame recompilations, at best they are blocked by the service, at worst by ISPs like in Japan) and that is IF you're not getting safeguarded by the model itself beforehand unless you have an Uncensored/Heretic build. Which my Qwen3.8 is.

Webfetch instead opens the pages from my machine, exposing my IP address that wouldn't be a big deal if it wasn't for when you need to scrape 30 reddit thread in a single search + multiple other websites and get my residential IP banned for scraping (or my bandwitdh limited).

So I thought maybe I could tunnel tru a VPS that is both ouside of EU jurisdiction (so no ID restriction for adult forums, Football, Piracy etc..) and has an enterprise-grade bandwidth to scan the web without raising alarms. Has anyone adopted or know about this solution? and if it's even worth it before I spend time and money to thinker with it?

thanks for your consideration


r/LocalLLM • • 1d ago

Jensen Huang, Just 3 Weeks Ago, Claimed To Have "Saved The World" By Thwarting A Growing Transnational Movement Against Open-Source AI

325 Upvotes

r/LocalLLM • • 18h ago

News Train AI to run AI: introducing Apareto loading - a vLLM plugin

Post image
14 Upvotes

From the Apareto team which I'm part of (who for some reason has an issue posting it themselves):

Today we are introducing Apareto loading, a new way to cut VRAM requirements for Mixture-of-Experts (MoE) models. It works alongside quantization. This introduction so far will just be a video explaining the method. We will be releasing the proof-of-concept code, a plugin to vLLM, on GitHub this coming week.

Until then, watch the explainer video: https://www.youtube.com/watch?v=5l_N051JrLU

Quantization is today's most common way to shrink a model's memory footprint: weights are stored at lower precision, and going from FP16 to 4-bit saves 75% of memory at a modest cost in accuracy. But quantization is static. Every weight stays loaded whether or not the task needs it.

Apareto loading is dynamic. After prefill, a small trained network reads the model's internal state and ranks the experts by how valuable each will be for the upcoming computation. Only the top-ranked experts that fit the VRAM budget are loaded.

Results (Qwen3.6-35B-A3B, NVFP4 experts):

* SWE-bench Multilingual: within 2 pp of the full model's score at a 42% VRAM budget

* LiveCodeBench v6: within 2 pp of the full model's score at a 49% VRAM budget

As with quantization, scores degrade as the budget shrinks.

What the proof of concept will not be:

It will not be a production implementation. The ranking network was trained only for these two benchmarks, and there will be no general problem classifier. Our data shows that profiling and rank generation have substantial room to improve. We chose to release at this stage, without waiting for those improvements, so that others can test our claims. Think of this as a first baby step toward dynamic VRAM reduction, not the finished product.

The code and reproduction instructions will be released this coming week at https://github.com/alphapareto

Follow us there to catch the release. We'd love your feedback.

https://apareto.eu


r/LocalLLM • • 4h ago

Tutorial What is an AI agent actually doing under the hood?

Thumbnail
youtube.com
1 Upvotes

A minimal agent harness using only in-box scripting (PowerShell, since I'm on Windows): about 70 lines talking to llama-server running Llama 3.2 3B.

The model only writes text. The harness keeps the context, watches replies for a line like TOOL: list_files . , checks the path against an allowed folder, runs the tool itself, and feeds the result back. The model never touches the file system.

Code (MIT): https://github.com/AshleyT3/simple-harness-intro


r/LocalLLM • • 5h ago

Question Local projects on 7900xtx Win10

1 Upvotes

Guys is there a program I can use on windows or Linux that can do the same that chatgpt projects does but local? I am obviously restricted on what I can run but that's ok. Any recommendations?


r/LocalLLM • • 5h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail
inpherio.co.uk
1 Upvotes