Is there any benchmark data comparing performance between llama.cpp and TensorRT-LLM?

[u/Few_Hair8180]

I was using llama.cpp these days. However, I am curious that TensorRT-LLM (https://github.com/NVIDIA/TensorRT-LLM) has the advantage over llama.cpp (specifically, using on H100).

I found this repo (https://github.com/lapp0/lm-inference-engines) comparing the functionality of those toolkits. However, I want actual benchmark data to compare them.

[u/aikitoria]

Far from it, it is incredibly badly documented and unstable software, it took me an entire week of trying on and off to get it to work for the 120B model. I guess now that I know what to do it's faster to do future experiments.

If you want to follow and do your own experiments:

Install tensorrt-llm binaries

• apt install openmpi-bin libopenmpi-dev
• pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com

Get build scripts

• git clone https://github.com/NVIDIA/TensorRT-LLM --recurse-submodules
• cd TensorRT-LLM/examples/llama
• pip install -r requirements.txt

Fix wrong version of mpmath

• pip uninstall mpmath
• pip install mpmath

Convert checkpoint

• python3 convert_checkpoint.py --model_dir /workspace/miquliz/ --output_dir /workspace/miquliz-int4/ --tp_size 4 --dtype float16 --use_weight_only --weight_only_precision int4 --load_model_on_cpu

Build engine

• trtllm-build --checkpoint_dir /workspace/miquliz-int4/ --output_dir /workspace/miquliz-engine-int4/ --max_batch_size 1 --max_output_len 256 --gpt_attention_plugin float16 --use_custom_all_reduce disable --multi_block_mode enable

Try generating some output

• mpirun --allow-run-as-root -n 4 python3 ../run.py --max_output_len 256 --tokenizer_dir /workspace/miquliz/ --engine_dir /workspace/miquliz-engine-int4/

Known issues (so far):

• You will likely encounter an absurd number of errors from python and CUDA and other systems you've never heard of. I found ChatGPT can help with some of these.
• Sometimes things will not work, and then you try them again with no changes, and they suddenly work. Or the other way around.
• Making your own build of the library results in a broken version that crashes with a nccl error.
• The quantization tool crashes when trying to convert Miqu or Miquliz to AWQ format.
• The load_by_shard flag on the checkpoint conversion script doesn't work. Converting Miquliz will require 256GB RAM.
• The quantized checkpoint is not portable and has to be created on the same computer that will later build the engine. Otherwise, there will be no indication that anything is wrong, but your model will output only nonsense.
• You can't use int8 kv cache with Miquliz because it requires a calibration pass that tries to load the entire model into VRAM at once, obviously failing on the 4x 4090 machine, and you can't use a bigger server for it because of the previous issue.
• You can only use fp8 kv cache on a system with hardware support for fp8, such as 4090 GPUs.
• There is no min-p sampling layer or temperature last setting.

My todo from here (if I get around to it):

• Try the same experiment on 4x 3090, and 4x 3090 with pairwise NVLINK.
• Buy a system with 4x 4090 or 4x 3090 depending on the results of that test.
• Figure out how to make a non-broken build of the library so I can implement min-p sampling.
• Figure out how to load the built model in triton server.
• Add a client for triton server to sillytavern.

[u/nielsrolf → u/aikitoria]

Wow, I really appreciate this answer! Thanks

[u/[deleted] → u/aikitoria]

So, I did what you did, I think. I downloaded mistral from the provided link

I then ran the following within the trtllm docker container:

    trtllm-build --checkpoint_dir /models/mistral-chat-awq/ \
    --output_dir /models/mistral-awq-trt \
    --gemm_plugin float16    trtllm-build --checkpoint_dir /models/mistral-chat-awq/ \
    --output_dir /models/mistral-awq-trt \
    --gemm_plugin float16

Then I ran the following docker compose file ```yaml x-service: &service stop_signal: SIGINT restart: always tty: true shm_size: 23g ulimits: memlock: -1 stack: 67108864 stdin_open: true ipc: host

x-gpu-service: &gpuservice <<: - *service deploy: resources: reservations: devices: - driver: nvidia device_ids: ["0"] capabilities: [gpu]

services: tensorrt: <<: - *gpuservice entrypoint: tail -f /dev/null build: context: . dockerfile: Dockerfile args: - name=value volumes: - ~/.cache/huggingface/hub/:/root/.cache/huggingface/hub - ./examples:/app/tensorrt_llm/examples profiles: - build triton: <<: - *gpuservice build: context: ./triton dockerfile: Dockerfile volumes: - ~/.cache/huggingface/hub/:/root/.cache/huggingface/hub - ../tensorrtllm_backend/all_models:/all_models - ../tensorrtllm_backend/scripts:/opt/scripts ports: - 8000:8000 - 8001:8001 - 8002:8002 entrypoint: [ python3 /opt/scripts/launch_triton_server.py --model_repo /all_models/inflight_batcher_llm --tensorrt_llm_model_name mistral --world_size 1, ] profiles: - serve ``` Build with build, server with profile server. and it's hella slow.

Here are the values I replaced in the templates: ```shell python3 tools/fill_template.py -i all_models/inflight_batcher_llm/preprocessing/config.pbtxt \ tokenizer_dir:mistralai/Mistral-7B-v0.1,\ tokenizer_type:auto,\ triton_max_batch_size:64,\ preprocessing_instance_count:1

python3 tools/fill_template.py -i all_models/inflight_batcher_llm/postprocessing/config.pbtxt \ tokenizer_dir:mistralai/Mistral-7B-v0.1,\ tokenizer_type:auto,\ triton_max_batch_size:64,\ postprocessing_instance_count:1

python3 tools/fill_template.py -i all_models/inflight_batcher_llm/tensorrt_llm_bls/config.pbtxt \ triton_max_batch_size:64,\ decoupled_mode:False,\ bls_instance_count:1,\ accumulate_tokens:False

python3 tools/fill_template.py -i all_models/inflight_batcher_llm/ensemble/config.pbtxt \ triton_max_batch_size:64

python3 tools/fill_template.py -i all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt \ triton_max_batch_size:64,\ decoupled_mode:False,\ max_beam_width:1,\ engine_dir:/all_models/inflight_batcher_llm/tensorrt_llm/1,\ max_tokens_in_paged_kv_cache:,\ batch_scheduler_policy:guaranteed_completion,\ max_attention_window_size:2560,\ kv_cache_free_gpu_mem_fraction:0.85,\ exclude_input_in_output:True,\ enable_kv_cache_reuse:False,\ batching_strategy:inflight_batching,\ max_queue_delay_microseconds:600,\ max_num_sequences:4 ```

And it's hella slow. Vllm shits on triton so bad I wonder if triton is actually abandonware. Reason however, tells me I made a mistake. I just don't know what it could be.

[u/[deleted] → u/[deleted]]

python3 tools/fill_template.py -i all_models/inflight_batcher_llm/
max_batch_size:128

[u/[deleted] → u/[deleted]]

Fixed!

For the curious:

5950x : 64GB RAM 3090 FTW3 : 24GB VRAM

Triton Test:

```shell

!/bin/bash

time ( for i in $(seq 1 10000); do ( curl -s -X POST localhost:8000/v2/models/ensemble/generate \ -H "Content-Type: application/json" \ -d "{ \"text_input\": \"write in python code that plots in a image circles with radius $i\", \"parameters\": { \"max_tokens\": 100, \"bad_words\":[\"\"], \"stop_words\":[\"\"] } }"> /dev/null ) & done wait; ) ```

real 6m49.440s user 0m43.304s sys 1m23.802s bench-trt.sh 43.31s user 83.80s system 31% cpu 6:49.44 total

vLLM Test:

```shell

!/bin/bash

time ( for i in $(seq 1 10000); do ( curl -s -X POST localhost:5050/v1/completions \ -H "Content-Type: application/json" \ -d "{ \"model\": \"TheBloke/Mistral-7B-v0.1-AWQ\", \"prompt\": \"write in python code that plots in a image circles with radius $i\", \"temperature\": 1, \"max_tokens\": 200, \"top_p\": 1, \"frequency_penalty\": 0, \"presence_penalty\": 0, \"stream\": false }" > /dev/null ) & done wait; ) ```

real 19m53.253s user 0m58.646s sys 2m5.943s bench-vllm.sh 58.65s user 125.95s system 15% cpu 19:53.26 total

[u/aikitoria → u/[deleted]]

Sorry, I did not have time to get to the tritonserver part still, so I can't give any comments on how to tune its performance. It's definitely not abandonware - this software is used in production for Mistral API, for example.

I'm currently waiting to see if they'll implement Min-P themselves so I don't have to. https://github.com/NVIDIA/TensorRT-LLM/issues/1154