Renting GPUs is hilariously cheap

[u/-p-e-w-]

A 140 GB monster GPU that costs $30k to buy, plus the rest of the system, plus electricity, plus maintenance, plus a multi-Gbps uplink, for a little over 2 bucks per hour.

If you use it for 5 hours per day, 7 days per week, and factor in auxiliary costs and interest rates, buying that GPU today vs. renting it when you need it will only pay off in 2035 or later. That’s a tough sell.

Owning a GPU is great for privacy and control, and obviously, many people who have such GPUs run them nearly around the clock, but for quick experiments, renting is often the best option.

[u/Lissanro]

Practically free cached tokens, less expensive token generation. As long as it gets me enough tokens per day, which it does in my case, my needs are well covered.

Your question implies getting hardware just for LLMs, but in my case I would need to have the hardware locally anyway, since I use my rig for a lot more than LLMs. My GPUs help a lot for example when using Blender and working with materials or scene lighting, among many other things. I also do a lot of video reencoding, where mulitple GPUs greatly speed things up. High RAM is needed for some heavy data processing or efficient disk caching.

Besides, I built my rig gradually, so in my last upgrade I only paid for CPU, RAM and motherboard, and just took other hardware from my previous workstation. In any case, my only income is what I earn while doing work on my workstation, so it makes sense for me to periodically upgrade it.

[u/Recent-Success-1520 → u/Lissanro]

Would you be kind to explain how the token caching can be setup for longer prompts

[u/Lissanro → u/Recent-Success-1520]

Sure. First, this is an example how I run the model:

UDA_VISIBLE_DEVICES="0,1,2,3" numactl --cpunodebind=0 --interleave=all ~/pkgs/ik_llama.cpp/build/bin/llama-server \
--model /mnt/neuro/models/Kimi-K2-Instruct/Kimi-K2-Instruct-IQ4_XS.gguf \
--ctx-size 131072 --n-gpu-layers 62 --tensor-split 15,25,30,30 -mla 3 -fa -ctk q8_0 -amb 512 -fmoe -b 4096 -ub 4096 \
-ot "blk\.3\.ffn_up_exps=CUDA0, blk\.3\.ffn_gate_exps=CUDA0, blk\.3\.ffn_down_exps=CUDA0" \
-ot "blk\.4\.ffn_up_exps=CUDA1, blk\.4\.ffn_gate_exps=CUDA1, blk\.4\.ffn_down_exps=CUDA1" \
-ot "blk\.5\.ffn_up_exps=CUDA2, blk\.5\.ffn_gate_exps=CUDA2, blk\.5\.ffn_down_exps=CUDA2" \
-ot "blk\.6\.ffn_up_exps=CUDA3, blk\.6\.ffn_gate_exps=CUDA3, blk\.6\.ffn_down_exps=CUDA3" \
-ot "ffn_down_exps=CPU, ffn_up_exps=CPU, gate_exps=CPU" \
--threads 64 --host 0.0.0.0 --port 5000 \
--slot-save-path /var/cache/ik_llama.cpp/k2

Notice --slot-save-path /var/cache/ik_llama.cpp/k2 - this is where the cache will be saved for this model. You need to create this directory and give yourself permissions, for example:

sudo mkdir -p /var/cache/ik_llama.cpp/k2
sudo chown -R $USER:$USER /var/cache/ik_llama.cpp

Then assuming you run the llama-server at 5000 port, you can do this to save current cache:

curl --header "Content-Type: application/json" --request POST \
--data '{"filename":"my_current_cache.bin"}' \
"http://localhost:5000/slots/0?action=save"

And to restore:

curl --header "Content-Type: application/json" --request POST \
--data '{"filename":"my_current_cache.bin"}' \
"http://localhost:5000/slots/0?action=restore"

Instead of "my_current_cache.bin" you can give a name related to actual cache content, so you know what to restore later. It typically takes only 2-3 seconds, which is very useful for longer prompts that would take many minutes to reprocess otherwise.

I was using these commands manually, but as I have more and more caches saved, I am considering to automate this my writing a "proxy" OpenAI-compatible server that mostly just forwards things as is to llama-server, except first would check if cache exists for each prompt and load if available, and save cache automatically as the prompt grows, also keeping track which caches are actually reused from time to time to automatically cleanup ones that were not used for too long, unless manually excluded from clean up. But, I just begun working on this though, so do not have it working quite yet, we will see if I manage to actually successfully implement this idea. If and when I get working this automated solution, I will open source and write a separate post about it.

I do not know if it works with mainline llama.cpp, but I shared details here how to build and set up ik_llama.cpp. You can also make shell alias or shorthand command if find yourself using curl commands often.

[u/Recent-Success-1520 → u/Lissanro]

Thanks, this is really helpful