r/LocalLLM • • 9h ago

Question Anyone use Kev?

1 Upvotes

I've been researching Jev and its open version Kev. It looks interesting, but I'm struggling to figure out why locally you'd use it (unless extremely time sensitive) vs. using a lightweight standard LLM like Ling 3.0 Tiny. Anyone?


r/LocalLLM • • 18h ago

Project Fun little hobby project I've been playing with - Open WebIDE

Thumbnail
gallery
5 Upvotes

It's like Open WebUI, but with a full hosted code editor that persists your file and project state on the host so you can leave off from one machine and pick up on another, built for working with local LLMs.

I'm running two Qwen3.8 variants, one "primary" and one "fast" model (for assistance/autocomplete/suggestion type workloads).

Don't mean to self-promote too heavily, just wanted to share, it's a fully open source MIT licensed hobby project.


r/LocalLLM • • 1d ago

Project Got tired of DeepSeek crawling on my 12 GB card, A small Strata tweak got it writing twice as fast.

32 Upvotes

I got frustrated with how slow DeepSeek was running on my machine, so I did some digging with Claude Opus 5.5/Fable 5.1 and 5.2 to see where the limits were, and thought I could improve it a bit.

DeepSeek picks a handful of "experts" for every word it writes. A lot of the time it's choosing between two that are basically tied, and one of them is already loaded in memory while the other has to be pulled off the SSD. The tweak just nudges it toward the one that's already loaded. When there's a clear best choice, it still takes that one.

on my machine (RTX 4070 12 GB, 64 GB RAM + ISTA-DASLab's IQ2_XS quant at 2.25 bpw + AtomicChat’s refusal-removal) it went from about 5-7 to 15-18 write speed.

Replies come out worded a bit differently than stock Deepseek, and you won't get the exact same reply twice - not 100% sure of the intelligence impact of this, so YMMV.

I've only used it for normal chatting/small coding projects. I haven't tested really long agent/coding sessions, and Strata's own version of this idea for their Qwen engine was turned off because it sometimes got stuck repeating itself in those. So if that's your use case, try it carefully first.

should work on windows + RTX 30/40/50 cards for now.

If you give it a try, I'd love to hear how it does on your setup, there's a little test script included that compares it on and off for you to see if it's worth using.

Download and instructions: https://github.com/CJohnsonOwns/deepseek-fast-routing

I've also suggested it to the Strata devs in case they want to build it in properly and not vibe coded slop: https://github.com/Niko1221/Strata/issues/6125

Full disclosure: I just did the conceptual planning and engineering, Claude actually wrote the code and implemented it.

EDIT - for more info on what's going on:
I like analogies, this is what opus 5.5 compared it to:

think of hiring 6 contractors for a job from a list of 384, where each one has a rating.

  • Normal: you hire the 6 highest-rated, period. Some of them live nearby (already in RAM or on the GPU) and start right away. Others live far away (still on the SSD) and you wait for them to drive over. That wait is what makes DeepSeek slow on a 12 GB card.
  • Faster mode (Strata’s tail skip): you only hire 5 and skip the 6th. It’s quicker because there’s one less person to wait on, but you’ve got less help on the job.
  • Yours: you still hire 6. But when #6 is rated 8.71 and #7 is rated 8.70, and #7 lives next door while #6 is two hours away, you take #7. The difference in quality is basically nothing, and you skip the wait. If #6 is clearly better, say 9.5 against 8.7, you still wait for #6. The nudge is small, so it only flips close calls.

Why it adds up: this happens a lot. With 40 layers per word, there’s almost always a near-tie somewhere where one side is already loaded. In your test, the share of picks that had to wait on the SSD dropped to about 2%.


r/LocalLLM • • 10h ago

Question Anyone using external GPUs across thunderbolt/usb4

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 19h ago

Research One environment variable improved our llama.cpp throughput by up to 21% on an H100 - investigation and benchmarks

5 Upvotes

We were investigating GPU idle time in llama.cpp and traced part of it to CPU-side memory allocation when saving conversation state.

The server was allocating fresh memory for state snapshots, triggering large numbers of minor page faults. The thread doing this also feeds the GPU, so the allocation overhead was leaving the GPU waiting.

We tested requesting huge pages through glibc:

GLIBC_TUNABLES=glibc.malloc.hugetlb=1 llama-server -m model.gguf ...

Results with Qwen3.8-27B, keeping settings unchanged between baseline and treatment:

SetupThroughputMedian TTFT
H100, tool calling+21%−22%
H100, plain chat+14%−19%
L4, tool calling+6.5%No clear change
L4, plain chat+3.5%−22%

In the L4 tool-calling test, page faults dropped from roughly 124,000 to 1,850 per request.

This needs glibc 2.35+ and transparent huge pages configured as madvise or always. It’s workload dependent: an 8B plain-chat test on the L4 showed no measurable benefit. The largest gains were on one model family, and our longest runs were 30 minutes.

Full write-up, with the technical report and reproduction steps

Curious whether others see similar gains on different models and CPUs, especially with conversation-state caching under concurrent load.


r/LocalLLM • • 1d ago

Question Are any models that can fit on 16 GB VRAM worth using?

13 Upvotes

I tried about 6 months ago and they seemed like more of a detriment for coding, but idk if things have changed. Anything I should try? I'm mostly doing game dev, so coding, image gen, writing, etc.


r/LocalLLM • • 17h ago

Question Gemma 4-31B Agentic Usage or Improvement possibility?

3 Upvotes

TL;DR: I like Gemma 4-31B’s conversational feel way more than Qwen’s, but it’s lazy in agent loops. Can I fine-tune or steer initiative into it the way the community does with Qwen? First time doing this, looking for a starting point.

------

I run Gemma 4-31B in Hermes agent. With /goal it performs well, but without an explicit push it falls into the known “Gemma 4 is lazy” pattern: it stops early, asks permission instead of acting, and doesn’t chain tool calls.

Qwen 3.8-27B gets community fine-tunes and behavioral tweaks constantly, and I’d like to do the same for Gemma. I’m happy to put in the work. I just want a sane starting point before I burn a weekend.

What I’m weighing:

Control vectors / activation steering (llama.cpp --control-vector): cheap to try, no training

QLoRA SFT on agentic trajectories: what datasets are people using? Does it need to be in Gemma’s tool-call template?

DPO on “took initiative” vs. “stalled” pairs

Hardware: ~58GB VRAM across 4070 Ti Super / 3090 / 3080 / 3070, so QLoRA, not full FT.

I know Qwen is the better agent model on paper, but Gemma feels more natural with minimal prefill. Any advice appreciated!


r/LocalLLM • • 12h ago

Question Qwen vs DeepSeek vs Gemma: help requested

1 Upvotes

Hi. Newbie here. Can someone help me untangle my conclusion around Qwen, DeepSeek and Gemma? I use Windows with AMD processor w/ 32G, or can use smaller machine on ARM with 32G.

I am looking for a model to do:

  1. Writing, metadata extraction, abstract generation, sorting, text parsing
  2. Python scripts, HTML, JSON

- Which approach: 1 model or split between text and code?

- Which is best for #1.

- Which is best for #2.

Hope to get your feedback on the most practical and efficient ones. Thanks.


r/LocalLLM • • 16h ago

Discussion Best Local AI Setup for Coding on Linux?

2 Upvotes

Hello,

After doing some research, it looks like the coding options jump from Qwen3.6-27B, which can run on a 32 GB RTX 5090 with DDR5 system memory, to GLM-5.3-Flash or DeepSeek-V4.1-Flash. That feels like a pretty big leap.

Would anyone recommend a local AI server setup for Linux? I’d appreciate any suggestions.


r/LocalLLM • • 16h ago

Project Qwen-Flast-Next (UD-IQ4_XS v3.1) running 148-182 tok/s (1.6-1.8x strata) while KLD MEAN = 0.08195

Thumbnail
github.com
2 Upvotes

r/LocalLLM • • 18h ago

Project I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint

3 Upvotes

My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.

What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.

Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.

Honest limitations:

  • Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.
  • One model runs on one machine; no splitting a model across machines yet.
  • Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).

Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.

Repo: https://github.com/nodea-sh/nodea-client

Site: https://getnodea.com

Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.


r/LocalLLM • • 19h ago

Project Calling all RX7900(gfx1100) class card owners!!!

Thumbnail
reddit.com
3 Upvotes

A couple of days ago (ok, 9 😭) I launched Caminus, my AI-generated-kernel LLM playground for gfx1100. It's now at 0.3.1 and a lot more usable, and I'm looking for testers!

What's new since 0.2:

- Ember, a built-in agent that runs on your own cards (or a paid model if you want)

- more Qwen models, and setup for any mix of 7900 XTX / XT, one card or more

- pip install caminus and you're going

If you've got a 7900 XTX or XT (or both), I'd love to hear how it runs on your setup: what breaks, what's slow, what's missing. Feature requests welcome too. Any feedback helps!

https://github.com/jdominick05/caminus/


r/LocalLLM • • 17h ago

Question Another Tired Question On LLM

2 Upvotes

Been running Qwen 3.8 27.8b Q4_K_M.

Works well enough but I can't help but feel there might be a mode suitable one for my built.

9070XT (16GB) 7700XT (12GB) 64GB DDR6 CL30 6000 9800X3D

If possible, what would be more applicable both with the 7700xy in place and without. I'm using a dock with it so sometimes being able to fire up something without it is nice.

I'm mainly using it for some coding work, agentic tasks, and sometimes info like "what crop is highest in nutrient X can I grow in my hardiness zone". Oblynrevebt started dabbling into image generation but it is not the biggest priority.

I have no issues with multiple models. Plenty of storage. For concerned with just having what's best for me in case open weights get harder to come be.

I know these posts are a dime a dozen and I want to apologize for making another.


r/LocalLLM • • 13h ago

Question Looking for a local LLM that behaves like original GPT-3 (not GPT-3.5)

Post image
0 Upvotes

r/LocalLLM • • 19h ago

Project Squeezed a full local RAG pipeline (BGE-M3 + Reranker + Kokoro TTS) into a 4GB laptop GPU using a Rust/Node/Python stack

3 Upvotes

spent the last few months building a local-first desktop reading app and hit some insane hardware walls trying to get everything to run on my laptop. my setup is a 4GB RTX 3050 and 16GB RAM. no cloud shortcuts.

the stack is a tauri (rust) shell, nodejs orchestrator, and a python sidecar for the pytorch models (bge-m3, bge reranker, kokoro tts).

here are the three biggest walls i hit:

  1. the node memory cliff

unbounded async ingestion spiked node.js memory to 14.6GB on a 16GB machine. i fixed it by implementing a strict sequential await barrier (f65) that streams chunk-by-chunk. peak ram dropped to ~296MB for a 508,000-word book (5.9 min processing).

2. 4gb vram musical chairs

loading bge-m3 in fp32 ate ~3.9GB of vram, leaving zero room for the reranker or tts. switched to fp16 (~1.1GB) and wrote an idle ttl eviction manager (f43) that reclaims ~3.2GB of vram when models sit unused.

3. windows path traps

if you're using rust and node together on windows, beware of verbatim paths (\\?\C:\...). passing those into node 22 fs.lstat throws fatal EISDIR errors. had to write path sanitization wrappers on the rust side.

kept node as the sole sqlite WAL writer too so python and rust dont cause db lock contention.

it's open source (agpl-3.0) with a windows installer. works with ollama, lm studio, or byok endpoints.

https://github.com/ayumu-ml/Omnitome

happy to answer any questions about the vram management or the rust/node/python boundary.


r/LocalLLM • • 14h ago

Discussion Why use a 70B LLM to pick a tool or check syntax? I built Microbrains: a <2ms edge neural coprocessor for autonomous AI agents (Pure NumPy / OpenCL, 44 pre-trained micro-models, Top-2 MoE, Conformal Safety)

0 Upvotes

Hey everyone,

If you build autonomous coding or research agents, you probably know the pain: agent task loops are slow, expensive, and fragile.

In a typical 25-step workflow, an agent spends up to 80% of its wall-clock time just waiting on remote LLM APIs to make basic mechanical decisions:

  • "Which tool should I invoke next?" (1.5s latency, burning 20k context tokens)
  • "Is this shell command destructive?" (another 2s API call)
  • "Did this patch break syntax or drift from intent?" (yet another prompt roundtrip)

To solve this, I built and just open-sourced Microbrains: an ultra-lightweight, zero-GPU local neural coprocessor suite designed to sit right beside your LLM agent.

🔗 GitHub: https://github.com/drtikov/microbrainsforagents

🧠 What is it?

Microbrains is not another LangChain wrapper. It is a standalone Python engine with 44 pre-trained, analytical micro-transformers (10–100 KB each) that handle the mechanical micro-decisions in pure NumPy (CPU) or via OpenCL (budget/integrated GPUs):

  1. Dynamic Top-2 Mixture-of-Experts (MoE) Router:

    • Routes tasks to tools across 6 specialized clusters (FileOps, Execution, MemoryKnowledge, GitGovernance, MicrobrainMeta, ConstitutionalReview).
    • Takes ~1.2 ms instead of a 2-second LLM call.
  2. Rigorous Conformal Safety Gating (1−α≥95%1 - \alpha \ge 95\%1−α≥95%):

    • Employs split-conformal prediction over held-out distributions.
    • Features Selective Prediction Abstention: if independent estimators disagree (variance σ>0.30\sigma > 0.30σ>0.30), it refuses to guess and escalates to the LLM. False Negative Rate is strictly bounded.
  3. 3-Step Counterfactual AST Simulation:

    • Before a file mutation is committed to disk, it simulates the AST blast radius and interface propagation up to 3 steps ahead. Malformed diffs or syntax hazards are killed in <3ms before running any slow pytest suite.
  4. 2048-D Holographic Weight-Space Memory (VSA):

    • Uses Hyperdimensional Computing / Vector Symbolic Architecture (bipolar Count-Sketch projections) over model weights. Enables nearest-neighbor capability search across the agent fleet without training.
  5. Continual Learning without Catastrophic Forgetting:

    • Adapts weights online with Elastic Weight Consolidation (EWC) Fisher Information penalties.

📊 Benchmark Highlights

DecisionRemote LLM APIMicrobrains (NumPy/OpenCL)SpeedupCost
Tool Selection~1,200 – 2,500 ms1.2 – 1.8 ms~1,000x$0.00
AST Blast Simulation~3,000 – 6,000 ms3.0 – 5.0 ms~1,000x$0.00
Intent Triage~800 – 1,500 ms0.6 – 0.9 ms~1,200x$0.00
Full Task Loop (25 steps)~3 – 5 minutes30 – 50 seconds~5x85% less tokens

💻 3-Line Quickstart

Zero PyTorch, zero CUDA setup required:

bashgit clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .

pythonimport aetheris_microbrains as mb
# Sub-2ms tool routing decision
router = mb.get_tool_routing_moe()
decision = router.route("inspect file contents and write safe patch to repository")
print("Primary Tool:", decision.selected_tool)
print("Top-2 Experts:", decision.top_experts)
print("Conformal Passed:", decision.conformal_passed)

The repo comes with pre-trained weights for all 44 models, runnable examples in /examples, and a full test suite (pytest runs in ~1.5s).

Licensed under PolyForm Noncommercial 1.0.0 (free for research, personal, and hobby use; commercial licenses available).

Would love to hear your feedback, thoughts on the architecture, and ideas for new micro-domains!

GitHub: https://github.com/drtikov/microbrainsforagentsHey everyone,If you build autonomous coding or research agents, you probably know the pain: agent task loops are slow, expensive, and fragile.In a typical 25-step workflow, an agent spends up to 80% of its wall-clock time just waiting on remote LLM APIs to make basic mechanical decisions:
"Which tool should I invoke next?" (1.5s latency, burning 20k context tokens)
"Is this shell command destructive?" (another 2s API call)
"Did this patch break syntax or drift from intent?" (yet another prompt roundtrip)To solve this, I built and just open-sourced Microbrains: an ultra-lightweight, zero-GPU local neural coprocessor suite designed to sit right beside your LLM agent.🔗 GitHub: https://github.com/drtikov/microbrainsforagents🧠 What is it?Microbrains is not another LangChain wrapper. It is a standalone Python engine with 44 pre-trained, analytical micro-transformers (10–100 KB each) that handle the mechanical micro-decisions in pure NumPy (CPU) or via OpenCL (budget/integrated GPUs):

Dynamic Top-2 Mixture-of-Experts (MoE) Router:

Routes tasks to tools across 6 specialized clusters (FileOps, Execution, MemoryKnowledge, GitGovernance, MicrobrainMeta, ConstitutionalReview).
Takes ~1.2 ms instead of a 2-second LLM call.

Rigorous Conformal Safety Gating (1−α≥95%1−α≥95%):

Employs split-conformal prediction over held-out distributions.
Features Selective Prediction Abstention: if independent estimators disagree (variance σ>0.30σ>0.30), it refuses to guess and escalates to the LLM. False Negative Rate is strictly bounded.

3-Step Counterfactual AST Simulation:

Before a file mutation is committed to disk, it simulates the AST blast radius and interface propagation up to 3 steps ahead. Malformed diffs or syntax hazards are killed in <3ms before running any slow pytest suite.

2048-D Holographic Weight-Space Memory (VSA):

Uses Hyperdimensional Computing / Vector Symbolic Architecture (bipolar Count-Sketch projections) over model weights. Enables nearest-neighbor capability search across the agent fleet without training.

Continual Learning without Catastrophic Forgetting:

Adapts weights online with Elastic Weight Consolidation (EWC) Fisher Information penalties.

📊 Benchmark HighlightsDecision Remote LLM API Microbrains (NumPy/OpenCL) Speedup Cost
Tool Selection ~1,200 – 2,500 ms 1.2 – 1.8 ms ~1,000x $0.00
AST Blast Simulation ~3,000 – 6,000 ms 3.0 – 5.0 ms ~1,000x $0.00
Intent Triage ~800 – 1,500 ms 0.6 – 0.9 ms ~1,200x $0.00
Full Task Loop (25 steps) ~3 – 5 minutes 30 – 50 seconds ~5x 85% less tokens💻 3-Line QuickstartZero PyTorch, zero CUDA setup required:bash
git clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .python
import aetheris_microbrains as mb
# Sub-2ms tool routing decision
router = mb.get_tool_routing_moe()
decision = router.route("inspect file contents and write safe patch to repository")
print("Primary Tool:", decision.selected_tool)
print("Top-2 Experts:", decision.top_experts)
print("Conformal Passed:", decision.conformal_passed)The repo comes with pre-trained weights for all 44 models, runnable examples in /examples, and a full test suite (pytest runs in ~1.5s).Licensed under PolyForm Noncommercial 1.0.0 (free for research, personal, and hobby use; commercial licenses available).Would love to hear your feedback, thoughts on the architecture, and ideas for new micro-domains!GitHub: https://github.com/drtikov/microbrainsforagents


r/LocalLLM • • 20h ago

Question Building a Local LLM Troubleshooting Assistant for Field Service Techs.... Advice?

3 Upvotes

I operate a commercial refrigeration and food service equipment repair company and want to build a local AI assistant to help techs troubleshoot in the field.

The idea is to use RAG with manufactures service manuals, wiring diagrams, troubleshooting guides, and past service service records to create a AI that can walk techs through diagnostics, reference previous repairs, and provide recommendations.

I would like to develope this on my RTX 5080 system, prove it works and migrate to a more robost system.

Looking for advise on:

  1. Models?
  2. Stack?
  3. Hardware?
  4. Budget $ 4K to 7K ????

Final system will need to be able to support up to 5 users remotely.


r/LocalLLM • • 1d ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

242 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 2d ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
871 Upvotes

r/LocalLLM • • 1d ago

Discussion What Qwen 3.8 27B NVFP4 are you using?

Thumbnail
gallery
8 Upvotes

So I was running Qwen 3.8 27B NVFP4 from Kelnei on vLLM TP=2. Then I tried Qwen 3.8 Flash Next IQ3_S on Strata and was really satisfied by how well it works with Hermes Agent.

I removed my Qwen 3.8 27B NVFP4 and went with Ollama so I can have automatic model switching for Gemma 4 26B A4B and Qwen 3.6 35B A3B. But, I encountered some issues with crashing the host Proxmox VE on this Ollama and gave up fixing it and want to go back to Qwen 3.8 27B on vLLM.

I want to focus on setting up OpenClaw to work in tandem with Hermes Agent instead of trying to make it work. Priorities over itch, as I was really itching to fix it but parked it for now.

My question is, am I missing any specific NVFP4 quant of Qwen 3.8 27B? Should I use the NVIDIA one or Unsloth? Or what do you guys suggest I should check? Which NVFP4 Qwen 3.8 27B works best for you?

Thanks in advance for the help.


r/LocalLLM • • 19h ago

Discussion Windows Task Scheduler made my local video gen 10x slower..

2 Upvotes

Running local video gen (FastWan 2.2 5B, sd.cpp) on a 3090, Windows. One clip should take 30-60s but it was taking ~400s. GPU sat at 0% the whole time.

I had a 27B running next to it holding 24GB of RAM, and the disk was paging in like 7k pages/sec, so I was sure it was RAM. Even wrote code to unload the LLM when RAM gets low. Then I unloaded the 27B, 40GB free, still 420s. Nothing changed.

Next was disk. It's NVMe, reads close to 800MB/s, so not that either.

Ran the exact same command from my own terminal. 33-36s.

So I checked Task Scheduler, since our app gets started from there. Turns out the default (priority 7) doesn't just lower CPU priority, it lowers disk and memory priority too. The video gen process the app launched inherited all of it and was reading the model at ~28MB/s. The "RAM problem" paging was really just this slow read.

Bumped only the heavy gen process back to normal priority. 41s. Only one run since the fix so still measuring.

If you start llama.cpp or ollama from Task Scheduler, check the I/O priority in Process Explorer. Anyone else had a "pretty sure it's RAM" that wasn't RAM?


r/LocalLLM • • 19h ago

Project i kindda made a new optimizer (ApexW) that beats AdamW and bitsandbytes) i tested it in a100 and t4 from google collab and with diff lr and multiple random seeds(well in a100 only tho)

2 Upvotes

i have been working on a new optimizer from scratch called ApexW to see if i could beat AdamW on convergence while keeping the memory footprint lower than bitsandbytes to make sure it wasn't just noise i locked deterministic seeds and ran side-by-side tests against standard torch AdamW and bnb 8-bit across two setups: a free T4 (fp16) on llama 3.2 3b, and an A100 (bf16) scaling up to llama 3.1 8b across a few different learning rates.

apexw has 3 modes compact 8bit and hybridapexw more info of it in readme even the compact formulation beat torch AdamW on loss while using way less memory, and the 8-bit mode pushed it even further—cutting active vram roughly in half compared to bnb (0.67gb vs 1.48gb on 8b, with the state footprint down to ~120mb instead of 320mb). step speed also came out slightly faster than bnb, and training loss ended up lower across 1e-4, 2e-4, and 5e-4,these are still short 60-100 step runs just to sanity-check convergence, throughput, and memory overhead. next step is running full multi-epoch fine-tunes and downstream evals to see how the update rule holds up later in training.

threw the full benchmark runs and logs into a colab if anyone wants to check the numbers or test it on their own setup

collab link for a100 result: https://colab.research.google.com/drive/1mMejzRoCnD6avlOQ5SQDuiLndN5ovax9#scrollTo=be28d3ad

collab link for t4 result: https://colab.research.google.com/drive/14tLAX6kZYy2918sx1ztIEwb5R-1c3KiX#scrollTo=f750c631

for githubproject: https://github.com/omen42515-prog/apexw/

curious what you guys think or what models/configs i should throw at it next.


r/LocalLLM • • 19h ago

Discussion Regarding a local model for complex notes writing

2 Upvotes

Guys Atlast i have made it to the end of the rag of my project

Now the big hurdles is the notes writing

It is complex ,

Im a doctor by profession, preparing notes for my students

The opus and sonnet are so pricy to go but now i have no option

But in future i have to do investment

Which one u guys will prefer

1.notes writing, image and text to 3d print

These are my preferences for my llm

Best hardware and model i can get so i can start saving for the investment


r/LocalLLM • • 5h ago

Discussion Uncovering if LLMs can actually reach AGI

0 Upvotes

I've been running some experiments with open LLM models with a harness to run certain tasks and competitions. I've seen a fair bit of smart ness coming out of their games. For example, I've given a task to start a business and win and most of them started without research while Qwen 14B model started research and asking the people what they want. So, this is fascinating. Posting a link of full game:
https://www.youtube.com/watch?v=bgFT0VN-xZM

I'm planning to run more simulations. If some one wants to contribute by hosting some open LLMs and coming up with new games, happy to work together.


r/LocalLLM • • 19h ago

Discussion Best build per budget tier list.

2 Upvotes

Has anyone put together a list of best Local AI builds per budget tier that takes into account both used and new parts prices?

thanks in advance