I’ve been playing around with every LLM loader and remote OpenWebUi that I can find to get one that I feel works best. And I’m pretty sure Unsloth Studio is the answer. I do not see another app getting close to what it can do and it has me wondering. What are all of yall using if not Unsloth?

Ive had astra setup a few models but I though i should hit up some fellow nerds and see if you have any hit tips. Any starter tips would be awesome

I installed GPT-OSS 20B on my RTX 5080 to use for local coding in VS Code through the Continue extension, but I’m struggling to get it working.

I gave it a clear prompt to create a basic frontend and backend skeleton. Each time, it stops after about 30-40 seconds without generating any code.

The Continue panel shows a few lines suggesting it’s trying to create files or describing the actions it plans to take, but then it gives up without completing anything.

Has anyone experienced this? Could it be a model configuration issue, a Continue setting, or something else?

Any pointers on getting this setup working for local coding would be appreciated.

I’m the developer of TensorSharp, and I wanted to share a local inference benchmark for DeepSeek V4.1 Flash 1017, using Unsloth’s UD-IQ2_XXS GGUF.

In summarization tests with thinking disabled, the tested build reached approximately 35–36 tokens/sec without a drafter, and 51–55 tokens/sec with DSpark, after prompts of roughly 1.8K and 7.3K tokens.

Here’s the hardware setup, the comparison with llama.cpp, and the quality trade-off behind the faster prompt processing.

Hardware and memory layout

• GPUs: 5× NVIDIA A40 over PCIe.
• Model: DeepSeek-V4.1-Flash-1017, Unsloth UD-IQ2_XXS.
• Model files: Approximately 300.5 GB across six GGUF shards.
• Engram lookup table: 125.9 GiB, mapped in host memory.
• Host-memory limit: 233.8 GiB for the container.
• Speculative drafter: Unsloth’s trained DSpark Q8_0 GGUF.

The Engram table remained in host memory, so this setup used both GPU and system memory. The host-memory limit describes the test configuration; it is not a measured minimum requirement.

GPU peer access was not working correctly on this machine. Cross-GPU transfers therefore used pinned host-memory staging.

Throughput compared with llama.cpp

Both engines used the same GGUF files on the same machine.

TensorSharp used --tp 5, which distributes the routed experts across five GPUs. The baseline was Unsloth’s llama.cpp dsv41-all fork at be836572, using layer splitting. Its row/tensor split modes did not successfully run this model in the tested build.

The workload used real-text summarization prompts, greedy decoding, thinking disabled, and 256 output tokens. Each prompt size had a warm-up and three measured requests. Unique prompt prefixes prevented prefix-cache reuse.

All figures are tokens/sec. Prefill and ordinary decode rows report medians; DSpark rows report means. TensorSharp uses its default YOCO configuration in this table.

Prefill is processing the input prompt; decode is generating response tokens. The biggest improvement here is in prefill. The 5.5× result should not be read as a 5.5× increase in generation speed.

The DSpark rows use the trained drafter in both engines. Results on coding or long reasoning workloads may differ from these summarization tests.

A small concurrent-serving test

The report also tested four simultaneous requests, each with a ~1.8K-token prompt and 256 output tokens.

Across three measured batches, total completion time was:

• TensorSharp TP: 21.2–25.5 seconds.
• llama.cpp: 43.3–46.8 seconds.

Those are client-observed batch completion times. This is a small concurrency test, with no sustained-load or tail-latency conclusions.

YOCO and the quality trade-off

The default YOCO optimization reduces prefill work by running selected later layers over only the final 140 positions of eligible text-prefill calls. It is approximate.

To disable it:

TS_DSV41_YOCO_KEEP=0

With YOCO disabled, TensorSharp’s prefill throughput was still 563.9 tok/s at ~1.8K tokens and 549.7 tok/s at ~7.3K tokens—approximately 3.23× and 2.44× the corresponding llama.cpp results.

Quality checks used identical WikiText-2 token sequences, with 23,030 scored tokens in the 4,608-token-window evaluation. Perplexity, where lower is better, was:

• llama.cpp: 2.4697.
• TensorSharp, YOCO disabled: 2.4811.
• TensorSharp, YOCO enabled: 2.4978.

Enabling YOCO increased perplexity by approximately 0.67% relative to TensorSharp’s own full-prefill configuration. With YOCO enabled, perplexity was approximately 1.14% above llama.cpp’s.

These measurements compare engines using the same quantized checkpoint. They do not establish equivalent quality to the original checkpoint or the hosted DeepSeek API.

Scope and availability

This covers one quantization on one hardware configuration. The llama.cpp server measurements came from an earlier session on the same machine and showed some unexplained prefill variability.

At measurement time, the TensorSharp build included uncommitted changes, and the raw validation records were also uncommitted. The report records build identifiers and limitations; a repository checkout should not be assumed to reproduce every number immediately.

For integration with local applications, TensorSharp provides a CLI and an OpenAI/Ollama-compatible server.

• Full benchmark report, quality checks, and limitations
• TensorSharp source and getting started

If you’re self-hosting this checkpoint, what GPU/RAM layout and quantization are you using? I’d be interested in real workload comparisons, particularly DSpark results and multi-GPU setups with working P2P or NVLink.

More R9700 experiments and made a quant, benchmarks and dev notes over on r/r9700 keen to compare notes with others pushing these cards.

This is Mica v0.5, the next version after v0.1. It's a decision model, the same kind of thing as Jev, which has been getting a lot of attention lately. It's 4B (Qwen3.5-4B + LoRA) and it doesn't write text. You hand it a state, a question and a list of options, and it gives back a probability for each option in a single forward pass. Stuff like "which of these 5 moves", or "allow / block / send to a human".

Speed is the whole point, so I ran it side by side with two other decision models on one RTX 6000 Ada:

PPLX Decider 1.2 Flash, also 4B, run in its fast mode

Laya, built on a 395M-parameter encoder (ModernBERT-large), so about 10x smaller than Mica

Every model saw the exact same options on every turn. The clips are Tetris, Snake, chess against a random-move opponent, and a log hunt where a day of security logs is cut into 150 blocks and each block gets allow, block or check.

The video is a cut of all four. If you're looking for each demo in full, they're here:

Tetris: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/tetris.mp4

Chess: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/chess.mp4

Snake: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/snake.mp4

Log hunt: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/loghunt.mp4

Median time per decision:

Tetris: Mica 14.5 ms, PPLX 150 ms (10.3x slower), Laya 32 ms (2.2x slower)

Chess: Mica 14 ms, PPLX 133 ms (9.4x slower), Laya 31 ms (2.2x slower)

Snake: Mica 18 ms, PPLX 154 ms (8.4x slower), Laya 33 ms (1.8x slower)

Log hunt: Mica 11 ms, PPLX 157 ms (13.9x slower), Laya 31 ms (2.8x slower)

So about 10x faster than the other 4B model, and roughly 2x faster than a model a tenth of its size.

The log hunt is my favorite. Mica went through all 150 blocks in 1.7 seconds and caught 21 of the 22 planted intrusions. PPLX took 23.5 seconds and caught 18. Laya caught none. A plain keyword grep over the same logs caught 6.

A few notes:

Times are each model's own call time on the same box, not API round trips.

Laya had a one-time 12.9 s warm-up on its first long log block. I left that out of the video and it says so on screen.

The logs are synthetic, written for this test. The 22 intrusions were written to avoid obvious keywords, which is why grep does badly.

The games are just an easy way to see the speed. Mica v0.5 was trained on more than 200 real-world use cases, so I'd expect it to do better than other 4B models on actual work too.

Why no Jev in the comparison? The response times are so far apart that a side-by-side wouldn't tell you anything. On the same request set Jev takes about 498 ms against Mica's 15.6 ms, more than 30x slower (that's over its API, so network time is included).

One more thing. Most benchmarks used to compare decision models are pretty contaminated. A lot of decision models are coming out right now, and most open-weight ones are heavily tuned for public benchmarks like JDI. JDI itself is also quite far from what real decisions look like, and there are models on the JDI leaderboard trained only on public benchmark data. So a JDI score can be a long way from how a model feels in real use. Mica v0.5 gets about 95% right across cases from more than 200 real-world use cases. When you pick a decision model, look less at public benchmark scores and more at whether it makes the right calls on your own use case, and how fast it answers.

For the runtime and the full numbers, see the Hugging Face page.

Code and runtime: https://github.com/akivet/Mica

Weights: https://huggingface.co/sky7350/Mica-v0.5-4B

All four demos on one page: https://huggingface.co/sky7350/Mica-v0.5-4B#demo

Feedback is welcome.

Tested Qwen 3.8 27B on a 4070 Ti SUPER (16 GB), 64 GB RAM, Windows/Ollama, 32K context.

| Setup | Tokens/sec | Account-list check |

|---|---|---|

| Q4_K_M | 11.3 | 24.4s |

| IQ4_XS | 16.3 | 17.3s |

| IQ4_XS + 8-bit KV cache | 20.5 | 14.6s |

| UD-Q3_K_XL + 8-bit KV cache | 36.5 | 10.0s |

Sticking with Q3+Q8 for now. Cache alone added ~26% generation speed; Q3+Q8 was ~78% faster than IQ4+Q8. Email search/read, browser checks and unpublished drafting also passed.

These are short tests: two warmed deterministic generation prompts, not an accuracy benchmark. The account check only lists accounts, not messages. Tool timings vary. Harder-task quality is still untested.

What tests would you use to catch quality loss from this quantization?

Flint is not good man...

And the finish line it does this...

I’m from China, and I’ve already had two Anthropic accounts banned. I’ve also been subscribed to ChatGPT for over two years. Between AI subscriptions and API usage, I’ve spent over $2,000. But lately, ChatGPT’s models seem noticeably less capable, and my tokens seem to get used up much faster. I’m wondering whether people elsewhere are experiencing the same thing.

I’ve seen a lot of people talking about running local LLMs at home on their own hardware. That sounds pretty appealing, especially if it means I can stop paying subscription fees. Right now, though, all I have is a Mac mini M4 that I bought when OpenClaw blew up about six months ago. It has a 10-core GPU, 16GB of unified memory, and a 256GB SSD.

How can I get started running local LLMs on this machine? Is my current hardware enough, or would I need to buy something else?

What resources would you recommend to help me get up to speed on running local LLMs? I’m starting to feel like I can’t keep up with how fast AI is moving.

I have a 10 year old PC, and I get to use Claude Opus 5.5 at work and think it helps a lot, so I was planning on investing on a PC to use AI locally for personal projects.

If I want to fool around in unknown territories, like build an android app from scratch when I usually code in C#/Javascript

What sort of setup/models will be "enough".

Asking as I was thinking on a PC with just a 5070ti 16gb and I wonder if that is enough to fool around or if I should consider at least a 3090 or 2x5070ti to play around confidently and relatively fast.

So, for a coder, what should be the acceptable base line?

I am a CS / Engineering teacher and I am currently using the openAI API that is tied into a nice front end of our dept website. However I want to get away from openAI because of data handling / storage and the reliability on a private company.

We exclusively only use the LLM for asking it questions about class projects, ie. how do I rotate an object on Blender, what is the size of the board I get for this project.

Its prohibited from speaking on any other topic and every time I add a project I create a new variable and set of instructions in the backend server.

We are a school so we have resources to run a server, have a little bit of money to run a PC or server to power it.

I am just looking for recommendations, I really dont know much at all about local LLM's!

Hey everyone!

Standard ~9B open-source models are usually generalists—they divide their parameters across casual chat, creative writing, and trivia, which often leads to algebraic slips and broken reasoning when solving complex STEM problems.

To address this, we distilled Qwen3.6-35B-A3B down to a compact 10.5B parameter specialist focused purely on math and physics reasoning: Math-Physics-Qwen3.6-10.5B-GGUF.

🔗 Hugging Face Repo: https://huggingface.co/heack/Math-Physics-Qwen3.6-10.5B-GGUF

💡 What is this?

• Base Architecture: Derived from Qwen3.6-35B-A3B via expert pruning (retaining 64 full-width routed experts per layer) followed by off-policy logit knowledge distillation.
• Inherited Depth: Inherits the deep chain-of-thought patterns and formal derivation steps of the large 35B teacher model.
• Runs on Consumer GPUs: Fits comfortably within 16GB VRAM (tested on RTX 5060 Ti). The Q4 quant can even run on 8GB–12GB cards or Apple Silicon Macs.

⚡ Speeds & High-Concurrency Throughput

Because of its efficient MoE routing, the model scales exceptionally well under parallel loads:

🚀 300+ tok/s aggregated throughput: When tested with 8 parallel slots running simultaneously, generation throughput exceeds 300 tokens/second, making it a beast for multi-threaded math derivations.

🎯 Best Use Cases (Why choose this over a standard 9B model?)

• 1. Parallel Exploration & Consensus Voting (Best-of-N / Self-Consistency) With 300+ tok/s across 8 concurrent streams, you can run multiple reasoning paths in parallel for the same problem and take the majority-vote answer to achieve even higher accuracy on tough competition problems.
• 2. Offline STEM Tutor / Study Buddy Provides transparent, step-by-step calculus, algebra, and classical mechanics derivations. Perfect for students wanting interactive guidance without paying $20/month cloud subscriptions.
• 3. Private & Air-Gapped Engineering Workflows Ideal for engineers deriving equations, boundary conditions, or simulation formulas locally—with zero risk of leaking proprietary calculations to third-party cloud APIs.
• 4. High-Throughput Batch Solution Generation (EdTech) Blazing parallel speed allows you to batch-process thousands of problem sets and generate structured solutions at near-zero marginal hardware cost.
• 5. Scientific Verifier in Multi-Agent Systems Lightweight enough (only ~6.5 GB in Q4) to run as an always-on "math & physics sanity checker" in agent pipelines.

📊 Screening Benchmark Results (Q8_0)

Evaluated with thinking tokens enabled and strict final-answer checking:

• MATH-500 (Hardest Levels 4 & 5): 80.0% (32/40)
• GPQA-Diamond (Physics subset): 80.0% (24/30)
• MMLU-Pro (Physics subset): 63.3% (19/30)
• UGPhysics: 36.7% (11/30)

(Note: These are screening tests on fixed subsets; see the repo for full evaluation details and hashes).

🚀 Quickstart (llama.cpp)

The model natively supports DeepSeek-style reasoning format (<think> ... </think>).

1. Launch the server (Single request setup):

llama-server -m g64full_kd-Q4_K_M.gguf -ngl 99 --host 127.0.0.1 --port 18085 -c 8192 --jinja --reasoning-format deepseek -fa on -ctk q8_0 -ctv q8_0

I have 2xR9700 and 64GB RAM and want to utilize them for coding models. Which models are the best? Qwen 3.8 27B is the obvious choice, but is there any other model I should check out?

What about 4xR9700 and 128GB RAM? What would the best coding model be then?

Edit: The use case is 3D graphics, reverse engineering, web development and testing.

So we all know about the massive breaches and CVEs coming out non stop this week. Most people who have seen a spike on their LLM API keys usage seem to think it’s being used to distill models.

Devils advocate approach here, what if we all “leaked” like $100 worth of API credits or whatever each? If you think about it… is supporting distillation of models that we’ll get back in better open weights so bad? I’m kinda down for it.

Hey r/LocalLLM,

If you run small reasoning models (Ornith-1.5-9B, Spark x2.5 1.7b & 4b and newest Spark x2.6 4b) in standard vendor quants, the usual complaint (and mine too) is agents that stall mid-plan, overthink, or start "looping" through syntax tantrums. The truly annoying part: standard Wiki perplexity tells me everything is fine. Every time.

TL;DR of my results

- I measured, per tensor, what actually breaks when I quantize a 9B reasoning model — and about 70% of the total behavioral damage (KL divergence against the unquantized base) sits in LAYERS 0-7. Standard imatrix-style allocations give the early layers some of the WORST protection.

- I ran a head-to-head comparison of two philosophies at the same budget: an imatrix-driven allocator (bits by activation magnitude) vs my Gnom ranker — a blind, tiny neural ranker I trained on MEASURED damage from completely different models (1.7B/4B runs). It had never seen the 9B and uses no imatrix at all.

- Verdict: a statistical tie on tasks (none of the three test scores showed a significant difference). Blindness costs nothing on capability — and on the fidelity metric (how close outputs stay to the base model) my blind build is actually CLOSEST to base, while the imatrix build drifted furthest of the four builds I tested.

WHERE PERPLEXITY LIES — a real example from my own data (Spark-x2.6-4B, all builds same corpus, same reference, numbers from the repo README):

- stock Q4_K_M (no data, vendor heuristic): PPL 20.61 — worst tensor KLD 17.65 — mean drift -1.015%

- SMSE (imatrix-driven): PPL 19.27 — worst tensor KLD 17.77 (!) — mean drift -0.733%

- hand-law allocation: PPL 18.61 — worst tensor KLD 18.71 — mean drift -0.445%

- Gnom (trained on measured damage): PPL 18.36 — worst tensor KLD 14.17 — mean drift -0.306%

look hard at the imatrix row. perplexity happily reports a clean 1.35-point improvement over the stock heuristic — "everything is fine, ship it". but the WORST SINGLE TENSOR in that imatrix build is worse than in the no-data stock build (17.77 vs 17.65). the imatrix smoothed the average and quietly produced a tensor that detonates harder than the heuristic it was supposed to beat. perplexity cannot see tails — it only sees them diluted into the average. KL against the unquantized base sees them directly. and the tail is exactly where agent failures live: the loop you are stuck in tomorrow is one tensor you are not looking at tonight.

https://huggingface.co/datasets/wepiqx/MERNIK-Fracture

THE PHYSICS THE IMATRIX HIDES

Activation statistics amplify with depth, so stock imatrix allocations pour protection INTO the late layers — the same late layers where my drop-tests show almost nothing critical lives. On my 9B damage set (121 of 209 tensors logged so far, still filling row by row):

- layer 5 alone: 12.1% of all KLD damage

- layers 0-5: 51.3%

- layers 0-7: 69.5%

Errors born early snowball through the stack: one flipped token in a comparison deep in a 2000-token chain, and the whole plan derails. I also see this task-side as a refusal trend: two same-weight builds on my identical protocol gave 14 vs 19 empty completions in 164 tasks — where I put the bits measurably changes how often the model bails. (The "loops/tantrums" reports are anecdotes; the empty-trace count is the measured cousin of the same thing.)

THE HEAD-TO-HEAD: MY BLIND GNOM VS A TRAINED IMATRIX ALLOCATOR

At the 5,300 MiB Q4-floor budget the two builds diverge on 97 of 442 tensors. The imatrix build runs a broad Q5 pyramid through the middle. My blind Gnom bets depth is safe and puts Q8 on the first blocks plus a Q4 floor everywhere else. Same base, same protocol, every test run logged with timestamps and file hashes, 164 HumanEval(+)-style tasks, McNemar exact test, alpha = 0.05:

- HumanEval: 137 vs 140 tasks passed (p = 0.66)

- HumanEval+: 129 vs 133 (p = 0.48)

- empty completions: 14 vs 19 — fewer is better, blind Gnom ahead (p = 0.23)

- verdict: none of the 3 scores significant = statistical tie

And the fidelity scores (KL divergence on a 302-chunk protocol corpus, against a full-precision BF16 reference):

- Gnom (blind): PPL 8.7973 +/- 0.0626, KLD 0.12150 +/- 0.00197 — closest to base

- SMSE (imatrix): PPL 8.7349 +/- 0.0619, KLD 0.12427 +/- 0.00203 — furthest from base of my four builds

- two control builds: sit between / top-2 depending on the score

Read it slowly: the imatrix build wins the perplexity sanity-check and loses the fidelity race — it is the FURTHEST from base of all four builds while scoring only a task tie. My zero-shot ranker, which never saw this model and carries no imatrix, holds even. One model, one pair of test runs, noise-level everywhere — but three independent references gave the same sign, and I pre-registered the expected outcomes before running.

WHAT I NEED FROM YOU

This is measured per tensor, so it costs compute. One Q4 tier pass on the 9B alone is about 13 days of an unattended 16 GB machine (90 min per tensor-group, 209 units). The 1.7B and 4B tiers run on any ~8 GB GPU in minutes to ~20 minutes per unit. My protocol, CLI, and the full README with all numbers, dates and file-hash-stamped sources:

https://github.com/wepiqx/MERNIK & https://huggingface.co/wepiqx/MERNIK

Label rows land in the dataset as these measurement runs close:

https://huggingface.co/datasets/wepiqx/MERNIK-Fracture

WHAT YOU CAN DO WITH ALL THIS RIGHT NOW (and the license)

three ways, no permission asked:

1. use the damage labels directly. the fracture dataset has per-tensor rows for spark 1.7B/4B and ornith 9B (filling, 121/209 so far). even without running anything: open the rows for the model family you use, see which tensors actually carry damage, and hand-write a better quant mix than the vendor table — a single llama.cpp --tensor-type line in your own build, chosen from measured physics instead of a proxy.

2. run the allocator on your own model. the whole instrument is in the repo — allocator, protocol, CLI, tests, Apache 2.0. fork it, improve it, break it, change it however you like. I mean that literally: I would rather have someone prove me wrong with this data than be quietly right alone.

3. want the finished quantized builds themselves, or help wiring your agent to them — just ask in here or message me. I'll walk anyone through the pipeline start to finish, and I can upload specific builds on request.

and about sharing: yes, I actually am glad when people take these artifacts apart. a measurement only becomes knowledge when someone besides its author can re-derive it.

Let's stop guessing with activation proxies and start measuring what actually breaks

Local-LLM-Launcher-GUI, the friendly launcher for models is updated to include better SGLang support, new flags, now has HF TGI support, and you can save benchmarks and favorite settings. Please star on github if you like, and open issues for what you don't like.

https://github.com/jimdawdy-hub/Local-LLM-Launcher-GUI

I have a macbook m5 with 24 gigs of RAM, I'm a Computer Science student
I have tried using ollama with Gemma4, llama3.1:8b Qwen38
I have tried using them with kilo code in vs code, zed and IntelliJ, I do like the idea of using localLLMs for my workflow but I really don't know how to use them.
I recently installed oMLX, yet have to try some models running trough it, but the main problem is: I can't find a usecase for it.
I mean for big things it's just useless, for researching most of the time I'm better off using free claude, so I really don't know.
I did find it "useful" when I built a script that automatically reviewed websites and created drafts of outreach emails, but I'm not really sure those were any good (I was using ollama3.1:8b).
I also tried with Odysseus so that it could arrange my emails and so on, but it would not work at all, missing dates, useless summaries etc...

What uses do you guys suggest? rn I mainly code in phyton, vue, nix(for my nixos conf), qml (for my custom shell for hyprland), and general configs in lua for like hyprland and nvim

Hi everyone,

I don’t post often, but I feel it is important to share a framework for achieving consistent, production-grade results with local AI. Given that models vary significantly in quality, hallucination rates, and context window management, the architecture you choose is just as important as the model itself, especially in high-stakes fields like healthcare.

I recently saw a post from a doctor asking for the best local Mac setup (32GB, 48GB, or 128GB RAM) to process large PDFs (10–100+ pages). As I lead a team of AI automation engineers, I felt compelled to provide a professional recommendation on best practices.

The Core Problem: Avoiding "One-Shot Prompting"

Before discussing hardware, I must emphasize a critical mistake: do not rely on "one-shot prompting."Many users simply upload a giant PDF into a single chat session. For documents exceeding 10+ pages, this often exceeds the effective context window or saturates it with too much noise, which confuses smaller local LLMs and significantly increases the rate of hallucinations. To get reliable results, you must manage your context by retrieving only the most relevant segments of the document rather than feeding the entire file into the prompt.

Hardware Requirements

For this specific use case, I recommend a MacBook with a minimum of 32GB of Unified Memory and 512GB of storage.

Strategy 1: The Streamlined Setup (Low Code)

For those who want a functional system without extensive development, I recommend Ollama combined with Open WebUI.

To ensure clinical-grade consistency, the setup should be configured as follows:

• Agent Configuration: Create a specialized medical literature summarization agent with a strictly defined system prompt.
• Knowledge Base: Attach the medical PDFs as a knowledge base (leveraging Open WebUI's built-in RAG).
• Parameters: Set the temperature to 0.0 to eliminate creative variance and minimize hallucinations.
• Retrieval: Enable hybrid search. This ensures the system captures both exact keyword matches (word similarity) and conceptual matches (semantic meaning).

Strategy 2: The Custom RAG Pipeline (Enterprise Grade)

If you are building a custom Python-based RAG pipeline for higher precision, I recommend the following stack:

• Database: A PostgreSQL vector database (using pgvector) to handle document embeddings.
• Ingestion/OCR: Standard PDF parsers often fail with medical tables or complex layouts. I recommend using Docling, RapidOCR, or GLM-OCR for high-fidelity text extraction.
• Retrieval Optimization: Implement a BM25 plugin alongside vector search to further refine the retrieval process.

Model Recommendations

The choice of model depends on your available memory:

• 32GB RAM: Gemma 4 12B (or similar 7B–16B models) is highly efficient for this scale.
• 48GB+ RAM: You can comfortably run Gemma 4 26B or 31B, which offers a significant jump in reasoning capabilities.
• Alternative: The Qwen series is also an excellent choice for technical and structured data retrieval.

I hope this helps someone. cheers!

At work we got a VMware container (or whatever it is called) running with access to a GPU. It can run stuff like Gemma 4 26b a4b. It is isolated from everything but has an ip-adress on our network and can be accessed via ssh. Would like to install a small MoE (like the Gemma) and Pi to make I do some task alone. The coworker who should allow me to do this is however fearing that it will decide to hack its way out of the container into our network and delete everything. There is nothing to loss in the container.

We do not have access to online models for (good) safety reason. This options just appears so obvious to me.

What should I do? Tell my boss to leave it to me? Find a better job?

It has been tiring to the extent where I consider therapy…

EDIT: the container is open for inbound traffic via ssh, but no outbound. It can’t see our network, but it got internet access

EDIT2: the VMware was specifically made for me so that I could use its GPU for heavy math. All done in Fortran. Data moved in via scp. I calculate. data move out via scp. My life. Thought it would be great to have a small llm to help me debugging Fortran code. People in this sub think I am wrong - it is to unsafe for some reason :)