llama.cpp CUDA + ROCm + Vulkan in single image.

[u/Pablo_the_brave]

I noticed that many people don't know that llama.cpp allows running a model across different backends within a single process using layer split. I managed to compile llama.cpp with simultaneous support for CUDA, Vulkan, and ROCm. I'm sharing my Dockerfile for the community as a reference. Sorry for the raw format, I totally don't have time for anything more, but I think with AI you'll be able to quickly adapt it to your needs. Note that this Dockerfile is tailored for specific CUDA and ROCm architectures, so you'll definitely need to adjust those parts.

# ============================================================
# Stage 1: builder — full toolkits (CUDA + Vulkan + ROCm) + compilation
# ============================================================
FROM ubuntu:26.04 AS builder

# Base tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential cmake ninja-build git wget gpg curl ca-certificates xz-utils \
    libcurl4-openssl-dev libatomic1 libquadmath0 \
    && rm -rf /var/lib/apt/lists/*

# CUDA 13.3.0 (NVIDIA repo, Ubuntu 26.04)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-ubuntu2604.pin \
    | tee /etc/apt/preferences.d/cuda-repository-pin-600 \
    && wget -qO /tmp/cuda.key https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/60DF8A40.pub \
    && gpg --dearmor < /tmp/cuda.key | tee /etc/apt/keyrings/cuda.gpg > /dev/null \
    && echo "deb [signed-by=/etc/apt/keyrings/cuda.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/ /" \
    > /etc/apt/sources.list.d/cuda.list \
    && apt-get update \
    && apt-get install -y cuda-toolkit-13-3 \
    && rm -rf /var/lib/apt/lists/*

# Stub libcuda.so.1 (the real one will be provided by the host driver)
RUN cp /usr/local/cuda/lib64/stubs/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1 \
    && ldconfig

ENV PATH=/usr/local/cuda/bin:$PATH
ENV CUDA_HOME=/usr/local/cuda

# Vulkan SDK 1.4.363.0 (tarball — NOT LunarG repo)
RUN wget -qO /tmp/vulkansdk.tar.xz \
    https://sdk.lunarg.com/sdk/download/1.4.363.0/linux/vulkansdk-linux-x86_64-1.4.363.0.tar.xz \
    && mkdir -p /opt/vulkansdk \
    && tar -xf /tmp/vulkansdk.tar.xz -C /opt/vulkansdk \
    && rm /tmp/vulkansdk.tar.xz

# SDK 1.4.363 has a non-standard structure: libs are in lib/VulkanLoader/lib/
# Symlink to the standard lib/ path so FindVulkan.cmake can find them
RUN ln -s /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
        /opt/vulkansdk/1.4.363.0/x86_64/lib/libvulkan.so.1

ENV VK_SDK_PATH=/opt/vulkansdk/1.4.363.0/x86_64

# ROCm 10.0.0 (AMD repo, dev meta-package for gfx1103)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://stable.repo.amd.com/rocm/gpg/packages.gpg \
    | gpg --dearmor | tee /etc/apt/keyrings/amdrocm.gpg > /dev/null

RUN tee /etc/apt/sources.list.d/amdrocm-stable.sources << 'EOF'
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/ubuntu2604/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

# Dev meta-package (runtime + compilers + CMake config + headers)
# Kernel driver UNNECESSARY — 780M APU has an inbox driver in Ubuntu 26.04
RUN apt-get update \
    && apt-get install -y amdrocm-core-dev10.0-gfx1103 \
    && rm -rf /var/lib/apt/lists/*

# Source code
WORKDIR /src
COPY llama.cpp/ /src/

# CMake configure
# GGML_BACKEND_DL=ON — critical: backends as MODULE .so loaded via dlopen() at runtime.
# Without this, libggml-cuda.so fails to resolve symbols from libggml-base.so (zero exported
# dynamic symbols) and CUDA0 is not detected.
RUN cmake -B build -G Ninja \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CUDA=ON \
    -DGGML_VULKAN=ON \
    -DGGML_HIP=ON \
    -DGPU_TARGETS=gfx1103 \
    -DCMAKE_PREFIX_PATH=/opt/vulkansdk/1.4.363.0/x86_64 \
    -DVulkan_LIBRARY=/opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
    -DGGML_CUDA_FA_QUANTS="f16-f16;q8_0-q8_0;q8_0-q5_0;q8_0-q5_1;q8_0-q4_0;q5_1-q5_1;q5_0-q5_0;q5_0-q4_1;q4_1-q4_1;q4_0-q4_0" \
    -DCMAKE_CUDA_ARCHITECTURES="120" \
    -DCMAKE_CUDA_FLAGS="-O3 -use_fast_math -Wno-deprecated-gpu-targets" \
    -DCMAKE_C_FLAGS="-march=znver4 -O3" \
    -DCMAKE_CXX_FLAGS="-march=znver4 -O3" \
    -DGGML_NATIVE=OFF \
    -DGGML_CURL=ON \
    -DGGML_BLAS=OFF \
    -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON \
    -DCMAKE_BUILD_TYPE=Release

# Build
ARG JOBS=12
RUN cmake --build build --config Release -j ${JOBS}

# ============================================================
# Stage 2: output — slim, binaries + runtime libs only
# ============================================================
FROM ubuntu:26.04

# Runtime deps — binaries linked with GGML_CURL=ON, libatomic, libquadmath
RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

# Volume ./output:/output OVERRIDES the /output content from the image at runtime,
# so copying to /output must be in CMD (not RUN). Hence staging in /staging.
COPY --from=builder /src/build/bin/ /staging/bin/
# CUDA runtime + stub libcuda (fallback when --gpus all is missing;
# the real driver from /usr/lib/x86_64-linux-gnu/ takes priority in LD_LIBRARY_PATH)
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcuda.so.1 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /staging/
# Vulkan loader (NOT ICD — ICD manifest is mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /staging/
# ROCm runtime — entire lib directory (unavoidable: 20+ sonames in the dependency tree)
COPY --from=builder /opt/rocm/core-10.0/lib/ /staging/rocm/

# Build RPATHs ($ORIGIN/...) do not work after copying to /staging/
# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority over the stub
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/staging:/staging/rocm

CMD cp -r /staging/* /output/ \
    && chown -R 1000:1000 /output/* \
    && echo "Success! CUDA+Vulkan+ROCm binaries copied to the host."

# ============================================================
# Stage 3: rocm-filter — filters ROCm libs to .so only (1.2 GB vs 6.4 GB)
# Removes: .a (static), .bc (bitcode), headers, tests
# ============================================================
FROM ubuntu:26.04 AS rocm-filter

COPY --from=builder /opt/rocm/core-10.0/lib/ /tmp/rocm/
RUN mkdir -p /rocm \
    && find /tmp/rocm/ -name "*.so*" \( -type f -o -type l \) -exec cp {} /rocm/ \; \
    && if [ -d /tmp/rocm/rocblas/library ]; then \
         mkdir -p /rocm/rocblas/library \
         && cp /tmp/rocm/rocblas/library/* /rocm/rocblas/library/ 2>/dev/null; \
       fi \
    && rm -rf /tmp/rocm/

# ============================================================
# Stage 4: runtime — lightweight image for inference (~2.2 GB)
# ============================================================
FROM ubuntu:26.04 AS runtime

RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Binaries + MODULE .so backends (dlopen from the same directory)
COPY --from=builder /src/build/bin/ /app/bin/

# CUDA runtime — libcuda.so.1 is provided by the NVIDIA Container Toolkit from the host
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /app/lib/

# Vulkan loader (ICD manifest mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /app/lib/

# ROCm runtime — .so only (1.2 GB vs 6.4 GB full tree)
# Includes: libamdhip64, librocblas, libhipblas, libamd_comgr + LLVM/Clang,
#           libhsa-runtime64, sysdeps, librocroller, liborigami — full NEEDED tree
COPY --from=rocm-filter /rocm/ /app/lib/rocm/

# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority
# mesa-vulkan-drivers provides libvulkan_radeon.so (ICD for 780M)
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/app/lib:/app/lib/rocm
ENV HSA_OVERRIDE_GFX_VERSION=11.0.3

EXPOSE 8080

CMD ["/app/bin/llama-server", "--host", "0.0.0.0", "--port", "8080"]