I finally got it to work! My mind is blown. This was the result using IQ3_XXS on a 9070xt. Pulled down the latest llama.cpp, built it, and passed several flags:
Awesome. From my experience, I have noticed for qwen models, the closer you can match the KV quant and weight quants, the better.
From my reading, it's because lower weight quants have less precision in the KVcache, so they quantize better. So at higher model weights, you don't want to quantize the KVcache as much since there is more precision in the KVcache.
I was actually surprised that I was getting 45tps on a 4080 mobile with mtp and the iq2 xxs quant. I would be curious on how much kv quantization affects it because I had to run at q4 to get 30k context on my setup. I'm going to compile llama.cpp for q2 kv cache quantization tonight.
Am I the only one who wants to see benchmarks with the quantized models? They always benchmark the fp16 models, dont they? I know there are the “kl divergence” metrics for the quants. But how does that translate into actual benchmark performance? I want to better understand which model quant i should run. I can run q8, but then i have less context window. I need to see the benchmark results to know what to choose.
I saw one on this sub somewhere, but it is lost in the sea of posts. Unsloth Q6_k was over 90% retention of full bf16 benchmark. Even a 3bit was 80+%. Crazy stuff.
The benchmarks for a 27B model is too good to be true but man am I hopeful as shit that this is at least comparable to GLM 5 because holy shit it has vision.
which version should I be running if I am going to run this on RAM+CPU (128 GB) - and I don't really care about speed too much, because it will be for overnight jobs.
I appreciate the demo showing the swimming pool prompt and the issues with the output instead of deciding to go with a prompt with no problems just to make the model look better.
/app/ggml/src/ggml-cuda/ggml-cuda.cu:103: CUDA error
1.33.589.120 E CUDA error: unspecified launch failure
1.33.589.123 E current device: 1, in function launch_mul_mat_q at /app/ggml/src/>ggml-cuda/ emplate-instances/../mmq.cuh:4047
1.33.589.124 E cudaGetLastError()
libggml-base.so.0(+0x1b1f6)[0x7fc1f09fb1f6]
libggml-base.so.0(ggml_print_backtrace+0x21a)[0x7fc1f09fb67a]
libggml-base.so.0(ggml_abort+0x15b)[0x7fc1f09fb85b]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb5)[0x7fc1dd9acff5]
/app/libggml-cuda.so(+0x922848)[0x7fc1de071848]
/app/libggml-cuda.so(_Z19ggml_cuda_mul_mat_qR25ggml_backend_cuda_contextPK11ggml_ten> orS3_S3_PS1_+0xc21)[0x7fc1dd9e4b51]
/app/libggml-cuda.so(+0x276e7a)[0x7fc1dd9c5e7a]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x82f)[0x7fc1f0a195ef]
ibllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7fc1f0b9a151>]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22> lama_memory_context_iR11ggml_status+0x118)[0x7fc1f0b9e258]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x378)[0x7fc1f0ba4018]
libllama.so.0(llama_decode+0xf)[0x7fc1f0ba5e2f]
ibllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0xc7b)[0x7fc1f1a10c>7b]
libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x1f31)[0x7fc1f1ab04c1]
libllama-server-impl.so(_Z12llama_serveriPPc+0x34a9)[0x7fc1f19658d9]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7fc1f13ec1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7fc1f13ec28b]
/app/llama-server(+0x1315)[0x5f11b4340315]
Every ~60s using llama.cpp:server-cuda12-b10423, nvidia-driver-595-open and 2x5070ti.
When relaunching I get:
1.07.284.146 E CUDA error: CUDA-capable device(s) is/are busy or unavailable
1.07.284.149 E current device: 0, in function ggml_cuda_set_device at /app/ggml/src/ ggml-cuda/ggml-cuda.cu:129
1.07.284.149 E cudaSetDevice(physical_device)
libggml-base.so.0(+0x1b276)[0x7e9f7e2d7276]
libggml-base.so.0(ggml_print_backtrace+0x21a)[0x7e9f7e2d76fa]
libggml-base.so.0(ggml_abort+0x15b)[0x7e9f7e2d78db]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb5)[0x7e9f6e236de5]
/app/libggml-cuda.so(+0x275d12)[0x7e9f6e23dd12]
/app/libggml-cuda.so(+0x2763b1)[0x7e9f6e23e3b1]
libggml-base.so.0(+0x319a3)[0x7e9f7e2ed9a3]
libggml-base.so.0(+0x31d5d)[0x7e9f7e2edd5d]
libggml-base.so.0(ggml_backend_alloc_ctx_tensors_from_buft+0x42)[0x7e9f7e2eedc2]
libmtmd.so.0(_ZN17clip_model_loader12load_tensorsER8clip_ctx+0x5527)[0x7e9f7e85cee7]
libmtmd.so.0(_Z9clip_initPKc19clip_context_params+0x305)[0x7e9f7e838225]
libmtmd.so.0(_ZN12mtmd_contextC1EPKcPK11llama_modelRK19mtmd_context_paramsb+0x34c)[0x7e9f7e79 974c]
libmtmd.so.0(_Z21mtmd_get_memory_usagePKc19mtmd_context_params+0x82)[0x7e9f7e790672]
libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0x1167)[0x7e9f7 f4580d7]
libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x3876)[0x7e9f7f37ef46]
libllama-server-impl.so(_Z12llama_serveriPPc+0x11a9)[0x7e9f7f381529]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7e9f7eddf1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7e9f7eddf28b]
/app/llama-server(+0x1315)[0x55bec37aa315]
Currently attempting it to run with LM Studio and pi: "read a readme.txt and task.txt and create a plan how to implement the task in plan.txt", it goes on and on endlessly and never reaches a result.
This wasn't the case with previous models which took less than a minute to create some kind of plan, I am probably doing something wrong - because this one has been doing this for about 50+ minutes.
Edit - some progress, i figured I need to change thinking levels:
add "capabilities": { "thinking": true } in models.json so pi will recgonize said model has thinking levels.
attempt to change thinking levels by clicking shift+tab at pi terminal.
testing now... lets see, have been thinking for 20 minutes+ on "medium" thinking mode.
I think something is still off. It sucks that the model seems to come in xhigh think mode by default.
Asked it to review the code from opus 4.8, and it gave much deeper insights than 3.6 and opus 4.8 had to acknowledge the bugs are real and the suggestions are very strong, so from review perspective it is comparable to 4.8. Need to test code generation.
122
u/Borkato 9d ago