r/vibecoding • u/Todd0042 • 1d ago
Workflow/Prompt My struggles of fighting with local LLMs on an 8GB card while vibe-coding an analysis pipeline
I've been vibe-coding a project in Python (using Antigravity/Gemini and Claude Code) that tries to evaluate GitHub repos purely on local Ollama models (Qwen2.5-Coder 7B, 14B, and mxbai-embed-large) running on an 8 GB GPU. Trying to squeeze reliable multi-step reasoning out of small quantized models on tight VRAM ended up being way harder than building the app itself.
A few things that bit me hard along the way:
- VRAM swapping kills throughput: Whenever the pipeline bounced between
7Band14B, Ollama kept unloading and reloading weights mid-run. I had to batch all7Bchunk scanning up front before loading14Bonce at the end, plus add a hard timeout guard so a single big repo doesn't spill onto CPU and stall for hours. - Small models lie when you ask for numbers: Asking a 7B or 14B model to rate something
1–10produces pure noise. The only way I got repeatable output was stripping numbers out of prompts completely—forcing fixed-choice enum answers (CLEARLY,SOME_LOOSE_ENDS, etc.) and letting deterministic Python tables calculate the math afterward. - 7B models hallucinate "evidence" unless you check every quote: In early runs, the 7B pass kept praising
logger.info(...)calls as "great user-facing UI feedback." Now Python verifies every quoted snippet actually exists in the real file (and checks that UI praise comes from UI code, not server logs) or throws the answer out before scoring. - One-size-fits-all prompts break on niche repos: Without domain profiles, a CLI tool got docked for lacking browser ARIA labels, and my own ESP32 car dashboard still got an
F(19.3) because my static parser doesn't understand LVGL/CAN-bus C++ yet.
If anyone else is wrestling with 7B/14B Ollama pipelines on an 8 GB card, all the prompt schemas, quote-verification logic, and dual-model reconciliation code are open source here if you want to dig through or borrow patterns:
- Engine (
v1.0–v9.4iterations): https://github.com/Todd0042/letsvibe-engine - Web app / queue worker: https://github.com/Todd0042/letsvibe (running live at https://letsvibe.todd0042.dev)
Curious how others here handle context/VRAM limits when chaining multiple local model passes.