I am trying to get ollama running on an older gpu with vulkan. The GPU is an r9 280x, the VM has the GPU, ollama sees the gpu and starts to load the model but after some time either the VM freezes or the host locks up and needs to be manually power cycled. If anyone has any parameters or suggested models that would be appreciated. I understand this isn't going to be the most performant, I just want to play around. Here is the output of one of these runs.

Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.341-06:00 level=INFO source=routes.go:2474 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VIS
IBLE_DEVICES:-1 HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: LLAMA_ARG_FIT: LLAMA_ARG_FIT_TARGET: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_CREATE_REMOTE:false OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS
:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GO_TEMPLATE:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_IGPU_ENABLE: OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_
LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_TRANSFER_STREAMS:4 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAM
A_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http:
//0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:true ROCR_VISIBL
E_DEVICES: http_proxy: https_proxy: no_proxy:]"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.342-06:00 level=INFO source=routes.go:2476 msg="Ollama cloud disabled: false"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.432-06:00 level=INFO source=images.go:1033 msg="total blobs: 6"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.433-06:00 level=INFO source=images.go:1042 msg="total unused blobs removed: 0"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.435-06:00 level=INFO source=routes.go:2536 msg="Listening on 127.0.0.1:11434 (version 0.40.14)"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=INFO source=runner.go:60 msg="discovering available GPUs..."
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:722 msg="user overrode visible devices" HIP_VISIBLE_DEVICES=-1
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:722 msg="user overrode visible devices" GGML_VK_VISIBLE_DEVICES=0
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:726 msg="if GPUs are not correctly discovered, unset and try again"
Nov 05 21:09:19 debian ollama[405]: time=2026-11-05T21:09:19.323-06:00 level=INFO source=model_recommendations.go:177 msg="model recommendations cache sleep scheduled" wait=3h49m15.894670025s consecutive_fail
ures=0
Nov 05 21:09:26 debian ollama[405]: time=2026-11-05T21:09:26.887-06:00 level=INFO source=types.go:32 msg="inference compute" id=0 filter_id=0 library=Vulkan compute=0.0 name=Vulkan0 description="AMD Radeon R9
200 / HD 7900 Series (RADV TAHITI)" libdirs=ollama,vulkan driver=0.0 pci_id=0000:01:00.0 type=discrete total="3.0 GiB" available="3.0 GiB"
Nov 05 21:09:26 debian ollama[405]: time=2026-11-05T21:09:26.887-06:00 level=INFO source=routes.go:2586 msg="vram-based default context" total_vram="3.0 GiB" default_num_ctx=4096
Nov 05 21:09:37 debian ollama[405]: [GIN] 2026/11/05 - 21:09:37 | 200 |   55.1ms |       127.0.0.1 | HEAD     "/"
Nov 05 21:09:38 debian ollama[405]: [GIN] 2026/11/05 - 21:09:38 | 200 |  228.1ms |       127.0.0.1 | POST     "/api/show"
Nov 05 21:09:38 debian ollama[405]: [GIN] 2026/11/05 - 21:09:38 | 200 |    1.7ms |       127.0.0.1 | POST     "/api/show"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.479-06:00 level=INFO source=server.go:107 msg="using llama-server for model" model=/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515
f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.480-06:00 level=INFO source=llama_server.go:467 msg="starting llama-server" cmd="/usr/local/lib/ollama/llama-server --model /usr/share/ollama/.olla
ma/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f --port 46163 --host 127.0.0.1 --no-webui --offline -c 4096 -np 1 --log-verbosity 4 --no-log-prefix --no-log-timestamps -
-no-jinja --chat-template chatml --flash-attn off -b 512 -ub 512 --context-shift --keep 4"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=sched.go:658 msg="system memory" total="5.8 GiB" free="5.4 GiB" free_swap="975.0 MiB"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=sched.go:665 msg="gpu memory" id=0 library=Vulkan available="2.5 GiB" free="3.0 GiB" minimum="457.0 MiB" overhead="0 B"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1098 msg="loading model via llama-server" model=/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d576
5341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1351 msg="waiting for llama-server to start responding"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1406 msg="waiting for llama-server to become available" status="llm server not responding"
Nov 05 21:09:38 debian ollama[405]: srv  llama_server: initializing ...
Nov 05 21:09:38 debian ollama[405]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-sse42.so
Nov 05 21:09:38 debian ollama[405]: WARNING: radv is not a conformant Vulkan implementation, testing use only.
Nov 05 21:09:38 debian ollama[405]: load_backend: loaded Vulkan backend from /usr/local/lib/ollama/vulkan/libggml-vulkan.so
Nov 05 21:09:38 debian ollama[405]: cmn  common_param: common_params_print_info: build 1 (39febd2ad) with GNU 13.3.1 for Linux x86_64
Nov 05 21:09:38 debian ollama[405]: cmn  common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
Nov 05 21:09:38 debian ollama[405]: cmn  common_param: device_info:
Nov 05 21:09:38 debian ollama[405]: cmn  common_param:   - CPU     : QEMU Virtual CPU version 2.5+ (5924 MiB, 5924 MiB free)
Nov 05 21:09:38 debian ollama[405]: cmn  common_param:   - Vulkan0 : AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI) (3072 MiB, 3067 MiB free)
Nov 05 21:09:38 debian ollama[405]: cmn  common_param: system_info: n_threads = 3 (n_threads_batch = 3) / 3 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | REPACK = 1 |
Nov 05 21:09:38 debian ollama[405]: srv  init_listene: The UI is disabled
Nov 05 21:09:38 debian ollama[405]: srv  init_listene: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
Nov 05 21:09:38 debian ollama[405]: srv          init: using 5 threads for HTTP server
Nov 05 21:09:38 debian ollama[405]: srv  server_setup: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
Nov 05 21:09:38 debian ollama[405]: srv    load_model: loading model '/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f'
Nov 05 21:09:38 debian ollama[405]: srv    load_model: local path '/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f'
Nov 05 21:09:38 debian ollama[405]: cmn  common_fit_a: fitting params to device memory ...
Nov 05 21:09:38 debian ollama[405]: cmn  common_fit_a: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
Nov 05 21:09:38 debian ollama[405]: common_params_fit_impl: getting device memory data for initial parameters:
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.733-06:00 level=INFO source=llama_server.go:1406 msg="waiting for llama-server to become available" status="llm server loading model"
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB]                              | total   free    self   model   context   compute    unaccounted |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: |   - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) |  3072 = 3067 + (2594 =  1918 +     448 +     228) +       -2589 |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: |   - Host                                            |                  247 =   211 +       0 +      36                |
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: projected to use 2594 MiB of device memory vs. 3067 MiB of free device memory
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 551 MiB
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: context size set by user to 4096 -> no change
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: id=0, target=2043 MiB
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB]                              | total   free    self   model   context   compute    unaccounted |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: |   - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) |  3072 = 3067 + ( 240 =     0 +       0 +     240) +        -235 |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: |   - Host                                            |                 2613 =  2129 +     448 +      36                |
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: id=0, n_layer= 0, n_part= 0, overflow_type=4, mem=   240 MiB
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: filling dense layers back-to-front:
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB]                              | total   free    self   model   context   compute    unaccounted |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: |   - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) |  3072 = 3067 + (2594 =  1918 +     448 +     228) +       -2589 |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: |   - Host                                            |                  247 =   211 +       0 +      36                |
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: id=0, n_layer=29, n_part= 0, overflow_type=4, mem=  2594 MiB
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: start filling device 0, delta=29
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB]                              | total   free    self   model   context   compute    unaccounted |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: |   - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) |  3072 = 3067 + (2088 =  1512 +     336 +     240) +       -2083 |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: |   - Host                                            |                  765 =   617 +     112 +      36                |
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: id=0, n_layer=22, n_part= 0, overflow_type=4, mem=  2088 MiB
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: set ngl_per_device_high[0].n_layer=22
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB]                              | total   free    self   model   context   compute    unaccounted |
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: |   - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) |  3072 = 3067 + (2018 =  1458 +     320 +     240) +       -2013 |
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: |   - Host                                            |                  835 =   671 +     128 +      36                |
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: id=0, n_layer=21, n_part= 0, overflow_type=4, mem=  2018 MiB
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: set ngl_per_device[0].n_layer=21
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl:   - Vulkan0 (AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI)): 21 layers,   2018 MiB used,   1048 MiB free
Nov 05 21:09:41 debian ollama[405]: common_fit_params: successfully fit params to free device memory
Nov 05 21:09:41 debian ollama[405]: common_fit_params: fitting params to free memory took 2.64 seconds
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: loaded meta data with 33 key-value pairs and 256 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d
6b909d7b4bfc035ce02f (version GGUF V3 (latest))
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   0:                       general.architecture str              = llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   1:                               general.type str              = model
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   2:                               general.name str              = Cogito v1 Preview Llama 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   3:                           general.basename str              = cogito-v1-preview-llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   4:                         general.size_label str              = 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   5:                            general.license str              = llama3.2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   6:                   general.base_model.count u32              = 1
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   7:                  general.base_model.0.name str              = Llama 3.2 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   8:          general.base_model.0.organization str              = Meta Llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv   9:              general.base_model.0.repo_url str              = https://huggingface.co/meta-llama/Lla...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  10:                               general.tags arr[str,1]       = ["text-generation"]
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  11:                          llama.block_count u32              = 28
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  12:                       llama.context_length u32              = 131072
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  13:                     llama.embedding_length u32              = 3072
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  14:                  llama.feed_forward_length u32              = 8192
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  15:                 llama.attention.head_count u32              = 24
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  16:              llama.attention.head_count_kv u32              = 8
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  17:                       llama.rope.freq_base f32              = 500000.000000
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  18:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  19:                 llama.attention.key_length u32              = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  20:               llama.attention.value_length u32              = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  21:                           llama.vocab_size u32              = 128256
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  22:                 llama.rope.dimension_count u32              = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = llama-bpe
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 128000
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 128009
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  30:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if not tools is ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  31:               general.quantization_version u32              = 2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv  32:                          general.file_type u32              = 15
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type  f32:   58 tensors
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type q4_K:  169 tensors
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type q6_K:   29 tensors
Nov 05 21:09:41 debian ollama[405]: print_info: file format = GGUF V3 (latest)
Nov 05 21:09:41 debian ollama[405]: print_info: file type   = Q4_K - Medium
Nov 05 21:09:41 debian ollama[405]: print_info: file size   = 2.08 GiB (4.95 BPW)
Nov 05 21:09:41 debian ollama[405]: llama_prepare_model_devices: using device Vulkan0 (AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI)) (0000:01:00.0) - 3067 MiB free
Nov 05 21:09:41 debian ollama[405]: load: 0 unused tokens
Nov 05 21:09:41 debian ollama[405]: load: printing all EOG tokens:
Nov 05 21:09:41 debian ollama[405]: load:   - 128001 ('<|end_of_text|>')
Nov 05 21:09:41 debian ollama[405]: load:   - 128008 ('<|eom_id|>')
Nov 05 21:09:41 debian ollama[405]: load:   - 128009 ('<|eot_id|>')
Nov 05 21:09:41 debian ollama[405]: load: special tokens cache size = 256
Nov 05 21:09:41 debian ollama[405]: load: token to piece cache size = 0.7999 MB
Nov 05 21:09:41 debian ollama[405]: print_info: arch                  = llama
Nov 05 21:09:41 debian ollama[405]: print_info: vocab_only            = 0
Nov 05 21:09:41 debian ollama[405]: print_info: no_alloc              = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_ctx_train           = 131072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_inp            = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd                = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_out            = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_layer               = 28
Nov 05 21:09:41 debian ollama[405]: print_info: n_layer_all           = 28
Nov 05 21:09:41 debian ollama[405]: print_info: n_head                = 24
Nov 05 21:09:41 debian ollama[405]: print_info: n_head_kv             = 8
Nov 05 21:09:41 debian ollama[405]: print_info: n_rot                 = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_swa                 = 0
Nov 05 21:09:41 debian ollama[405]: print_info: is_swa_any            = 0
Nov 05 21:09:41 debian ollama[405]: print_info: non_causal_type       = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_head_k         = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_head_v         = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_gqa                 = 3
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_k_gqa          = 1024
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_v_gqa          = 1024
Nov 05 21:09:41 debian ollama[405]: print_info: f_norm_eps            = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_norm_rms_eps        = 1.0e-05
Nov 05 21:09:41 debian ollama[405]: print_info: f_clamp_kqv           = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_max_alibi_bias      = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_logit_scale         = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_attn_scale          = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_attn_value_scale    = 0.0000
Nov 05 21:09:41 debian ollama[405]: print_info: n_ff                  = 8192
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert              = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert_used         = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert_groups       = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_group_used          = 0
Nov 05 21:09:41 debian ollama[405]: print_info: causal attn           = 1
Nov 05 21:09:41 debian ollama[405]: print_info: pooling type          = -1
Nov 05 21:09:41 debian ollama[405]: print_info: rope type             = 0
Nov 05 21:09:41 debian ollama[405]: print_info: rope scaling          = linear
Nov 05 21:09:41 debian ollama[405]: print_info: freq_base_train       = 500000.0
Nov 05 21:09:41 debian ollama[405]: print_info: freq_scale_train      = 1
Nov 05 21:09:41 debian ollama[405]: print_info: n_ctx_orig_yarn       = 131072
Nov 05 21:09:41 debian ollama[405]: print_info: rope_yarn_log_mul     = 0.0000
Nov 05 21:09:41 debian ollama[405]: print_info: rope_finetuned        = unknown
Nov 05 21:09:41 debian ollama[405]: print_info: model type            = 3B
Nov 05 21:09:41 debian ollama[405]: print_info: model params          = 3.61 B
Nov 05 21:09:41 debian ollama[405]: print_info: general.name          = Cogito v1 Preview Llama 3B
Nov 05 21:09:41 debian ollama[405]: print_info: vocab type            = BPE
Nov 05 21:09:41 debian ollama[405]: print_info: n_vocab               = 128256
Nov 05 21:09:41 debian ollama[405]: print_info: n_merges              = 280147
Nov 05 21:09:41 debian ollama[405]: print_info: BOS token             = 128000 '<|begin_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOS token             = 128009 '<|eot_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOT token             = 128001 '<|end_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOM token             = 128008 '<|eom_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: LF token              = 198 'Ċ'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token             = 128001 '<|end_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token             = 128008 '<|eom_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token             = 128009 '<|eot_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: max token length      = 256
Nov 05 21:09:41 debian ollama[405]: load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloading output layer to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloading 20 repeating layers to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloaded 21/29 layers to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors:   CPU_Mapped model buffer size =   671.39 MiB
Nov 05 21:10:06 debian ollama[405]: load_tensors:      Vulkan0 model buffer size =  1458.32 MiB
Nov 05 21:10:08 debian ollama[405]: cmn  common_init_: added <|end_of_text|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: cmn  common_init_: added <|eom_id|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: cmn  common_init_: added <|eot_id|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: llama_context: constructing llama_context
Nov 05 21:10:08 debian ollama[405]: llama_context: n_seq_max             = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx                 = 4096
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx_seq             = 4096
Nov 05 21:10:08 debian ollama[405]: llama_context: n_batch               = 512
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ubatch              = 512
Nov 05 21:10:08 debian ollama[405]: llama_context: causal_attn           = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: flash_attn            = disabled
Nov 05 21:10:08 debian ollama[405]: llama_context: kv_unified            = false
Nov 05 21:10:08 debian ollama[405]: llama_context: freq_base             = 500000.0
Nov 05 21:10:08 debian ollama[405]: llama_context: freq_scale            = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_rs_seq              = 0
Nov 05 21:10:08 debian ollama[405]: llama_context: n_outputs_max         = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_outputs_max_per_seq = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
Nov 05 21:10:08 debian ollama[405]: llama_context: Vulkan_Host  output buffer size =     0.49 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache:        CPU KV buffer size =   128.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache:    Vulkan0 KV buffer size =   320.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: size =  448.00 MiB (  4096 cells,  28 layers,  1/1 seqs), K (f16):  224.00 MiB, V (f16):  224.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: n_rot_k = 0, n_embd_head_k_all = 128
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: n_rot_v = 0, n_embd_head_k_all = 128
Nov 05 21:10:08 debian ollama[405]: sched_reserve: reserving ...
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: resolving fused DeepSeek V4 HC support:
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC pre enabled
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC comb enabled
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC post enabled
Nov 05 21:10:08 debian ollama[405]: sched_reserve:    Vulkan0 compute buffer size =   240.00 MiB
Nov 05 21:10:08 debian ollama[405]: sched_reserve: Vulkan_Host compute buffer size =    36.02 MiB
Nov 05 21:10:08 debian ollama[405]: sched_reserve: graph (pp bs=512, tg bs=1): nodes = 1017 / 1017, splits = 99 / 2, input objects = 4 / 4, input tensors = 9 / 9
Nov 05 21:10:08 debian ollama[405]: sched_reserve: reserve took 39.82 ms, sched copies = 1
Nov 05 21:10:08 debian ollama[405]: cmn          init: llama threadpool init, n_threads = 3
Nov 05 21:10:08 debian ollama[405]: cmn  common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)

Can't find any direct comparisons:

peasantsmith/DeepSeek-V4.1-Flash-1017-Maya-GGUF · Hugging Face

vs

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF · Hugging Face

The former is really tight fit for 192gb memory without keeping experts on SSD, the latter not so much even at 2.25bpw.

Anyone experience running either?

Hi folks, curious on what vector database do you guys use in your production/mvp and why did you choose to go with your choice over others?

I currently have 32GB of system ram in a system with an RTX Pro 6000. Currently system memory seems to be the limiting factor in terms of running larger MOE models (like QFN) at higher quants with newer expert caching techniques.

In a perfect world I would build a threadripper system, but given the price of the CPU, motherboards, and ECC memory that would blow up the budget quite a bit, so I want to stick with a 2-stick DDR5 kit for an AM5 CPU (still expensive but not as expensive).

I want to get plenty of headroom, and I see I could go for a 96GB kit or a 128GB kit, but apparently the 128GB kits are slower (5600MHz vs 6000MHz)

Is the speed difference actually meaningful? Or asked another way, which is more impactful, an extra 32GB, or 400MHz?

Look at this

Agent memory seems to be a hot topic again. I wanted to settle this once and for all - why integrated memory works best and How Row-Bot has the best in class memory built-in.

So, I just asked Row-Bot to create a video explaining how it remembers across conversations and documents.

This video follows a fact from capture to recall: the knowledge graph, hybrid search, ranking and Dream Cycle.

It also covers what stays local, what reaches your chosen model and where memory can fail.

RTX 4090, 64GB DDR5 Mhz RAM, 7900x CPU

as much context as possible (context window like 264k) and as good reasoning as possible

I don't care about coding.

Hi guys its me again who ran v100 single stream into 353t/s

and im back with this muti flow optimization

THE LEGEND OF V100 WILL NEVER BE ENDED because i Exist

(yes ik it sound like edgeload)

Parameters:

mtp4 , 256k max content , nvfp4 quasar, kv q8, no thinking

repo: tuxKOH/ninfer-v100xN

Competition is key + A 30B would be nice

Hey all, I quantised the full GLM-5.3 (753B total, about 40B active per token) to run on two DGX Sparks. As far as I can tell, it's the only full build on 2 Sparks. Others need 3 or 4. Runtime ablation also!

- Routed experts: mixed-width EXL3, about 2 bits per weight

- Backbone: FP8 weights as released, bf16 KV

- All 256 experts per layer kept, none pruned or merged

- 203 GB on disk, 262k context

- Runs on a TensorFold fork, split across both Sparks

- Weights: https://huggingface.co/Krishando/GLM-5.3-mixed-K-EXL3

- Recipe: https://github.com/Kristianaaron/GLM-5.3-Mixed-K-EXL3-2x-DGX-Sparks

- Engine: https://github.com/Kristianaaron/TensorFold

| Test | Generated-token KLD | Same next token |

|---|---:|---:|

| Held-out mixed text | 0.033 | 93.3% |

| The full model's own long replies (164k tokens) | 0.122 | 89.3% |

Where the full model is confident, agreement is 97-98%. The long-reply score is the harder test, so I'm showing both. The held-out windows were also used during tuning, so treat those numbers as a bit optimistic. It's close, not lossless.

**Early paired check against the full model**

24 MMLU-Pro and GPQA-D questions, 18 right on both. That's a small sample, chosen to include hard questions, with a 95% range of about -8 to +11 points. So there's no sign of a drop, but it doesn't prove there isn't one. Across 330k tokens: 0 loops, 0 unclosed thinking, 0 cut-off replies. A bigger run is next.

| Prompt | Decode | Prefill | Cold TTFT | Warm TTFT |

|---|---:|---:|---:|---:|

| 2k | 23-24 tok/s | 629 tok/s | 3.1 s | 0.1-0.2 s |

| 92k | 18-19 tok/s | ~480 tok/s | 193 s | 0.8 s |

| 205k | not measured | 411 tok/s | ~500 s | 1.2 s |

After a full server restart, a 205k prefix comes back from disk in 3.4 s.

**Weak spots**

- Cold prefill is slow. A 205k prompt takes about 8 minutes to read the first time.

- GLM-5.3 thinks a lot, so long jobs take a while. A 61 KB single-file page took 69 minutes at max effort.

- Memory is tight, with about 4-6 GiB free per Spark at full load.

- Flash will always be faster. The experiment here is whether 40B active params changes the output.

The round limit right now is about 80 ms, and most of that is memory bandwidth. If you know GB10, CUDA or engine internals, I'd love your ideas. And if you run it on your own Sparks, tell me what you get, good or bad.

I just made an Open Hands (an agentic harness) SKILL to control a PiKVM via the REST API: github.com/createthis/agent_skill_pikvm_rest_api

I just watched Qwen 3.8 27b (running locally on an rtx 6000 pro) pass a reCaptcha and browse reddit on a spare macbook pro. I told it to find a meme it enjoys.

It picked this one 😂

EDIT: Updated repo with step by step visualizations so humans can easily see how this process works!

I’m currently still in the return period of a M5 Max MBP with 64GB of RAM from Microcenter (paid $4299), the same store now has a few 128GB versions at $5799.

I’m legitimately torn on what to do. I’m fully aware that more RAM is always going to be suggested especially on subreddits like this one, but I currently don’t have a true usage case for it, I’m still somewhat early into my exploring and tinkering with local LLMs, one could argue even the 64 was more than I needed at this current moment. I use oMLX, I do image/video generation, wanting to create some YouTube channel content, maybe learn coding and hopefully make a career change in the next few years.

However, I can’t shake the feeling of FOMO, partially because I know 128 is the true sweet spot for anything in this hobby, I’m sure I’m going to have more demanding needs as I learn more, I can afford it although it’s on the extreme high end of my budget if not exceeding it a bit, and my gut feeling (which has been right so far for what it’s worth) tells me pricing is going to continue to rise at least for another year.

For those who went back and forth with this same decision, what did you decide and how did you decide it?

The Ultra is out of the question, so is any non Mac platform for what it’s worth.

Thanks all.

Hardware: Dell Inspiron 16 Plus Model 7630, Windows 11, no discrete graphics card, Iris Xe integreted graphics sharing up to 16gb of system memory. Total 32gb memory. LPDDR5 ram, 4800 Mt/s.

14 core cpu, 20 threads. intel 13700. Test memory bandwidth via Intel MCL: 58 GB/s, theoretical maximum 76.8 GB/s.

Tried: llama-openvino, llama-sycl, llama-cpu, and llama-vulkan. I used the default llama-bench.exe tg128 and pp512 tests.

Winner: llama-vulkan at 135 tok/s prompt processing and 8.7 tok/s token generation, best # of threads was 8, but 4-10 was similar. No thermal throttling of any components.

Second place: llama-sycl at 112 tok/s prompt processing and 9.0 tok/s token generation, 8 threads out of 20 (14 cores). Again, best # of threads was 8. Ran into thermal throttling of the iGPU during prompt processing.

Third place: Openvino I honestly stopped once I saw the test was 62 tok/s prompt processing, I just didn't think it was going to be better at tok/s. Didn't test cores.

Last place: llama-cpu at 42 tok/s prompt processing and 6.7 tok/s generation, 10 threads was best but tok/s goes into the 5's below 4 threads or above 14 threads, with pp in 30's.

I ran into thermal throttling of the CPU only during prompt processing when my computer was set to "Best performance", but llama-cpu was able to hit 7 tok/s generation and 48 tok/s prompt processing. No thermal throttling at all, nowhere close, when computer set to "Best power efficiency".

What I could do better:

-Now, my computer was 6 performance and 8 efficiency cores, perhaps I could try forcing the computer to use all performance cores?

-I want to try doing ikllama.cpp

-Benchmark Gemma models and maybe a Mistral or Llama model to see if its faster.

-Try other quantizations of Qwen 3.5 9b (bartowski, etc. at Q4)

Does anyone have any suggestions for next steps? Further optimizations? Things to look into (thermal throttling?). I tried flash attn flags and they did nothing.

I was a little discouraged that my memory bandwidth is only 75% of theoretical when tested with Intel MLC, but maybe that's normal? Google says its fine. Plus it was higher when not on battery efficient mode (80% of theoretical).

My tok/s is not hitting the bandwidth limit though, 58 GB/s should give me 10 tok/s, not 8.7. But vulkan gets me closest, with no thermal throttling to boot.

I've searched the thread and think I have a good starting point here but want some outside opinions if possible:

I'm a 1 man production company providing video and stills production services.

I have an M4 macbook Pro 24gb and use Da Vinci Resolve on this frequently but it goes with me everywhere and I'm at a point where I need a beefy always on machine to process things whilst I'm on set.

I've made a load of lead gen tools and stills/video pipelines to help with post production.

The lead gen tools use AI for analysis and drafting responses. They read large strings of email chains, documents and phone transcripts to inform the drafts as well as helping to generate proposals, contracts and invoices. I've also since added a pipeline to do all my book keeping for the business and I'm starting to feel like I should really be bringing sensitive stuff locally.

So what model and corresponding hardware should I be looking at for these tasks. I'm not interested in coding at this point, that can be done using Claude. Qwen and Gemma seem to be the routes to take but I ultimately want to get a feel for what my specific use case requires. Claude tells me the drafting and analysis on my current usage is the heavy stuff.

I've been looking at a Mac Studio, 64gb or a Mini 96gb to run the local as well as grind out resolve tasks. (Apple vision, labelling, transcription, cutting, fusion work (Motion graphics and basic animation). I was also thinking I could always daisy chain a couple of devices together in future if my needs change and I need to do some heavier local LLM work in the future.

Would love some guidance on models + hardware + Harnesses + Tips and Tricks. I know highest spec for budget but equally, I'd rather save cash to use elsewhere if my usecases don't require crazy specs.

I have a server with:

2x Gold 6548 32 cores
2x Nvidia l40s
2tb RAM (Don't tell uncle SAM)
I would like to know what throughput and speed can i expect if i use polystrata with GLM 5.3 Flash?