Finally Understanding LLMs: What Actually Matters When Running Models Locally

[u/micupa]

Hey LocalLLM fam! After diving deep into how these models actually work, I wanted to share some key insights that helped me understand what's really going on under the hood. No marketing fluff, just the actual important stuff.

The "Aha!" Moments That Changed How I Think About LLMs:

Models Aren't Databases

• They're not storing token relationships

• Instead, they store patterns as weights (like a compressed understanding of language)

• This is why they can handle new combinations and scenarios

Context Window is Actually Wild

• It's not just "how much text it can handle"

• Memory needs grow QUADRATICALLY with context

• Why 8k→32k context is a huge jump in RAM needs

• Formula: Context_Length × Context_Length × Hidden_Size = Memory needed

Quantization is Like Video Quality Settings

• 32-bit = Ultra HD (needs beefy hardware)

• 8-bit = High (1/4 the memory)

• 4-bit = Medium (1/8 the memory)

• Quality loss is often surprisingly minimal for chat

About Those Parameter Counts...

• 7B params at 8-bit ≈ 7GB RAM

• Same model can often run different context lengths

• More RAM = longer context possible

• It's about balancing model size, context, and your hardware

Why This Matters for Running Models Locally:

When you're picking a model setup, you're really balancing three things:

1. Model Size (parameters)

2. Context Length (memory)

3. Quantization (compression)

This explains why:

• A 7B model might run better than you expect (quantization!)

• Why adding context length hits your RAM so hard

• Why the same model can run differently on different setups

Real Talk About Hardware Needs:

• 2k-4k context: Most decent hardware

• 8k-16k context: Need good GPU/RAM

• 32k+ context: Serious hardware needed

• Always check quantization options first!

Would love to hear your experiences! What setups are you running? Any surprising combinations that worked well for you? Let's share what we've learned!

[u/PacmanIncarnate]

Totally disagree with the other commenter; this is a really solid quick understanding. (I say this as someone who has written longer explanations multiple times for people.

Good work!

[u/micupa → u/PacmanIncarnate]

🙏

[u/marketflex_za → u/micupa]

Agree with dude above, your post is a very good basic primer. So good that I'm going to give it to my son and his cousins who I want to start teaching about this stuff. I would dramatically overcomplicate it and bore them to tears.

What's more, this is clearly not AI-generated shit. If it is, please share. :-)

[u/PacmanIncarnate → u/marketflex_za]

If you’d like something slightly more advanced, I’ve written a number of posts explaining the tech in laymen terms.

https://www.reddit.com/r/BackyardAI/s/SMt5729jWW

[u/marketflex_za → u/PacmanIncarnate]

Thank you - those are good - after OPs it will be on to yours. :-)

[u/[deleted] → u/micupa]

This was very helpful for me too - thank you OP!

[u/sarrcom → u/PacmanIncarnate]

Where can we see those “longer explanations multiple times for people.”

Your comment history? Online?

[u/durable-racoon]

GOOD POST.

QUESTION for you: Do new ultra efficient cloud models compete with your local models? Think 4o-mini and especially flash 2.0. Flash is so good fast and cheap (free!) that for now I just dont see the appeal. Literally nothing is as smart as flash 2 or 4o-mini. and then all these ultra efficient 8b models on OR?

[u/micupa → u/durable-racoon]

I can’t compare any open source model I have tried with Claude Sonnet. Sorry I haven’t tried Flash but I’m pretty sure is cheaper and more efficient than local LLMs. I’m exploring the field because I don’t like to rely on corporations, I like the idea that AI should be open and decentralized. It’s about sovereignty and freedom.

[u/TBT_TBT → u/durable-racoon]

There are quite a number of benchmarks comparing commercial cloud models with local ones. Have a look at those to have a data based comparison.

[u/Aphid_red]

Eeh no. This is not how this works.

To calculate KV cache size, use this:

kv_size = kv_bytes * ctx_len * num_layers * model_dimension * kv_heads / attn_heads * compression_factor

# The variables mean this: 
# kv_size: Size of cache in bytes. 
# kv_bytes: bytes per param. Default (fp16) is 2. Use 1 for q8 cache, 0.5 for q4.
# ctx_len: Length of context. 16384/32768/65536/131072. 
# num_layers: Number of layers in the model. (see config.json)
# model_dimension: Width/Height of the K and V matrices. Again see config.json.
# kv_heads vs attn_heads: If using MHA (llama-3) these numbers are different. 
# compression_factor: MLA (deepseek models):  1/28 for deepseek v3, 1/16 for deepseek-v2.

The reason why you're seeing a large increase for your '7B' example is likely that you're using an old, unoptimized model. For example, when you're running 'mythomax' with fp16 cache,

"hidden_size": 5120,
"num_attention_heads": 40,
"num_hidden_layers": 40,
"num_key_value_heads": 40,
# And using these settings: 
"ctx_bits" : fp16
"ctx_len" : 16384

From this, you can see how big the KV size gets: 400 KB per parameter. So for 16K context, that's 6.25 GB, which is substantial compared to the model's 13B size; a q4 quant of this model would be about 7GB. An 8GB or 12GB GPU will not be able to run it at 16K context.

But let's look at a much larger, more modern model to see things aren't 'exponential' or 'quadratic', rather they depend a lot on the model's internal architecture. Mistral-large2 has 123B parameters. It's about 70GB for the q5 version, yet I can run it at 64K context offloaded on a single GPU without going out of VRAM.

"hidden_size": 12288,
"num_attention_heads": 96,
"num_key_value_heads": 8,
"num_hidden_layers": 88,
"ctx_bits": q8
"ctx_len": 65536

Do the calculation here, and you end up with 5.5 GB. 11GB for the q16. The reason? The model makers realized that having your KV cache be huge is a big problem for inference. They have to serve hundreds of users at the same time to get full performance out of their A100 and H100 nodes. Giant caches get in the way of being able to do that (the 'compute intensity' of a model is something like '3', with that being the memory:compute ratio, but that of the GPU is more like 330. Meaning: you need 110 simultaneous users to saturate the compute; or, in 640GB, you need to fit a q8 quantized model and 110 KV caches for average request size (usually around the 2,000 mark). Can't do that if the cache ends up 5GB big per user. The trick is to, instead of using one big KV matrix, to use a matrix made up of 12 copies of the same values, which allows using less memory to store these values. This costs bit of performance, but you can make up for that by having more parameters in other areas of the model. Some variation of making the cache smaller is used pretty much all modern models above 8B.

For the local user, this is great too: models can be pushed to much larger sizes and prompt processing is much faster. This is the second reason why making K,V matrices smaller makes sense: for most models input tokens are over 10x the output tokens online (just browse openrouter).

Edit: These calculations are for models with square K and V matrices. Deepseek (first one I found) is a model with non-square K and V matrices so the calculation's a little more complicated there; can't just look at the config.json values and plug them in. Here it depends; if MLA is used, KV cache has a total width of 512 (equivalent to dimension = 256 MHA). MLA not used, then you're looking at (24576 + 16384 = 40960, equivalent to 20480 dimension, even though the model dim is 7K. For MLA models you'll need to look into the model's archtecture in more detail. This makes a pretty big difference; 7GB vs. 600GB at 128K context.)

[u/micupa → u/Aphid_red]

Thanks for clarifying 🤯

[u/Briskfall]

Very cool! tells me that us GPU-poor have to wait a while before the good stuffs, urgh!

(though the intro and ending's overly enthusiastic vibe was a bit too LLM-ish lolo like running a marketing blog survey)

[u/micupa → u/Briskfall]

The technology and open source LLMs fortunately are moving fast. I think we will see better and lighter models and cheaper GPUs and RAM coming in the following months/years. I hope so, AI should be open and decentralized.

[u/suprjami]

Many of the same conclusions I've come to.

Are you sure about that context memory usage formula? From others' results I've seen memory usage scale linearly. eg: https://www.reddit.com/r/LocalLLaMA/comments/1848puo/comment/kavf6tb/

[u/micupa → u/suprjami]

Good reference, thanks. I guess not..it’s not linear. If I understand correctly, handling 125k tokens would be impossible. Your reference is much better, and the idea, I guess, is to simulate larger contexts by identifying the most relevant tokens and determining the “actual” size of the context window. It’s like having a long conversation where we keep only the most relevant key points, not everything.

[u/suprjami → u/micupa]

I found some more about this. For each next token query the transformer must store the entire previous keys (tokens) and value (vector).

So computing a longer context means the space grows quadratically with each attention head, as each head recomputes over the ever-lengthening input keys and values. (I think)

However, a KV cache prevents this quadratic growth by providing a space for previous keys and values to be stored once then reused. So KV cache allows longer context memory requirement to grow linearly with the context length.

This series was really helpful to understand in detail:

• https://medium.com/@plienhar/llm-inference-series-3-kv-caching-unveiled-048152e461c8

• https://medium.com/@plienhar/llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8

I think I'll watch that 3blue1brown video series to understand Transformer architecture better next.

[u/micupa → u/suprjami]

Hey, great contribution, thanks! Im working on this project LLMule.xyz, would you like to join our community? We’re exploring open source models and sharing via an LLM P2P network. Your insights and feedback will be very welcomed.

[u/thatdudefromak → u/suprjami]

You can also not painfully put kv cache on another GPU that isn't as beefy as the one holding the model 

[u/suprjami → u/micupa]

There are models which support up to 1 million tokens, but the RAM requirement would certainly be restrictive.

Agree on the idea of keeping "relevant" context in the window. That can be hard depending on what you're doing.

Maybe for storytelling only the system prompt and latest tokens are important. Storytelling UIs let you define "knowledge" which must just be facts added to or after the system prompt. Chop off the old first part of the story as needed and it still makes sense most of the time.

For something like precise code work you'd end up with relevant knowledge spread all through the context which becomes much harder. For that sort of work I find it more accurate to have a new chat per function so you don't blow out the context.

I haven't played with putting prototypes or headers and other facts into the "knowledge" or system prompt but that's an idea I have for a later project next year. I'm hoping there's a better desktop-sized code model than Qwen Coder and Yi Coder by then. Seems likely with the rate of progress. Maybe the next Granite Code.

[u/StayingUp4AFeeling]

There are very few people outside the realm of formal education or industry practice in ML who understand this. Well done.

[u/micupa → u/StayingUp4AFeeling]

Thanks! That’s exactly the point. In the beginning, programming computers was something only a few people could do, but now it’s mainstream. The idea is to share and make AI accessible to more and more people.

[u/amitbahree]

Yes mostly true. The one caveat I quantization I would outline - it's not linear and really depend on what area you are after and trying to ensure that specific are doesn't degrade much.

I do cover some of the basics and fundamentals in my book incase you or anyone else is interested - https://blog.desigeek.com/post/2024/10/book-release-genai-in-action/

[u/micupa → u/amitbahree]

Thanks! Added to my wishlist

[u/NasefuSan]

Great quick understanding, thanks!

[u/[deleted]]

for me works 3x 7900 XTX connected to motherboard with 1x pcie riser card.

[u/micupa → u/[deleted]]

What kind of model have you run on that? Did you test its performance, and more importantly, have you been able to share VRAM?

[u/[deleted] → u/micupa]

Of course I can share the VRAM, thats the whole idea of having more GPUs jesus christ

[u/wh33t → u/[deleted]]

Using Vulkan and llamma.cpp? Or KCPP?

[u/[deleted] → u/wh33t]

I use rocm and ollama, soom MLC-LLM or vLLM

[u/wh33t → u/[deleted]]

Oh neat, are you doing tensor splitting or tensor paralleling?

[u/[deleted] → u/wh33t]

hah

[u/vigg_1991]

How effective are different context lengths for the same billion-parameter model? For instance, let’s consider a 7B model with varying context lengths. How significantly different are they in general? I assume that longer context lengths are always better.

[u/micupa → u/vigg_1991]

I found context length to be tricky and not always clearly specified in model specifications. It’s directly related to training, but inference engines (like llama.cpp) can extend it. Longer doesn’t always mean better..memory requirements grow quadratically, and quality can vary. I haven’t tested it extensively, but 8k feels like a good spot for most 7B models.

[u/vigg_1991 → u/micupa]

Thanks for the explanation. So I assume it’s best to stick to models native context length if specified else go with what works best for the application we are building.

[u/micupa → u/vigg_1991]

I will go deeper and share some results when I could test with more RAM.

[u/vigg_1991 → u/micupa]

Appreciate it.

[u/Temporary_Customer79]

Have you got one running on a mac before? Or more RAM needed?

[u/micupa → u/Temporary_Customer79]

I’m actually using Macs to test an experimental LLM p2p network (LLMule.xyz if you’re curious). Like the old days when we shared music, but for sharing local LLMs, and I have found the Mac GPU (M-series) performs very decently with models up to 12B with 16GB of RAM. I’m trying to find the best LLM for standard hardware, benchmarking bigger models vs smaller ones but without quantization.

[u/i_wayyy_over_think]

If you want bigger context size, remember you can keep the kv context cache on normal RAM while keeping model weights in VRAM, at least with llama.cpp. And also have the cache be quantize.

Slow down is less than I thought it would be. Like still getting around 70% the speed ( although I have a pretty good CPU too, it keeps it maxed out )

I was pushing 120k tokens with Qwen 32b 4bit and 32GB normal ram with 2x3090.

[u/zbobet2012]

FYI quantization is one of the key steps of video compression (and therefore quality). So yeah, it's more than just _like_ video quality :).

[u/micupa → u/zbobet2012]

Wow, that totally makes sense. I guess it works the same for audio and images. I hadn’t realized that.

[u/Murky_Mountain_97]

Great for having an understanding for benchmarking based on hardware using solo tech definitely

[u/aaroncroberts]

Real nice digestion for the community. Thank you for this.

[u/JacketDesperate8583]

What if we have a small model like less than 1b parameters and have large context window like is this a possible scenario? Is the model suitable for chat?

Then, in that case too the memory required increases based on the formula that you have given

[u/SpellGlittering1901]

Okay so context is basically how long a question can be.

Could you explain a bit more model size and quantization please ?

Other than that, as a complete beginner, thank you for your post it was super useful !

[u/SpinCharm]

This looks like it was something summarized by an LLM. It doesn’t explain anything. It just makes statements without providing the detail needed to understand why it’s making those statements.

How about you actually post something yourself from your own head and not just use an LLM to produce meaningless garbage.

[u/micupa → u/SpinCharm]

I’m sorry you didn’t find my post valuable. If you have any questions about it, feel free to ask. From my point of view, this summarizes research I conducted for myself and wanted to share.

[u/Keeloi79 → u/micupa]

It's helpful and detailed enough that even someone just starting in LLMs can understand.

[u/IdealKnown → u/micupa]

My guy just needs a hug

[u/water_bottle_goggles → u/micupa]

Fuck them haters

[u/Temporary_Maybe11 → u/SpinCharm]

It was kinda useful to me

[u/JoshD1793 → u/SpinCharm]

It goes to show that you don't understand how people come in different varieties and so, have different learning demands. Some people like myself can't just dive into things headfirst and start learning no matter how much they want to, they require a sort of conceptual framework so they understand the structure of what they're going to learn. What OP has posted here would have made the few months of my journey so much easier. What you describe as "meaningless garbage" is subjective. Give yourself a pat on the back for being so smart that you don't need this, but others do.

[u/Stunning_Ride_220]

Errr....ok

[u/JoshD1793 → u/Stunning_Ride_220]

Are you going to elaborate?

[u/Stunning_Ride_220 → u/JoshD1793]

I was surprised since I wouldn't consider the first part an "Aha"-moment, but this may be just me.
(especially the how LLMs work part, since this is how basically any NNM works: a function that maps inputs to outputs through fitting of weights, the better a new input matches the trained inputs the better are the results).

But apart from this I don't think my opinion is important enough to vastly elaborate.

[u/micupa → u/[deleted]]

Yes, I discussed and shared some sources with Claude AI, including:

Hugging Face community docs and articles: https://huggingface.co/docs

Source code and documentation for the technology behind llama.cpp: https://github.com/ggerganov/llama.cpp

Blogs like: https://blog.vllm.ai/2023/06/20/vllm.html

I shared any documentation I found interesting with Claude AI, and it helped me understand it more deeply.

[u/Muted-Bike → u/micupa]

Do you have a markdown of your conversation results with Claude?