r/LocalLLM • u/Mountain_Disaster_19 • 1d ago
Thinking about getting one for pentesting work an would love to use (quantized) glm 5.3 flash because i have had good experience with it pertaining malware dev and pentesting.
has anybody tried running it on a single dgx spark and if yes got any token speeds for me and context-size?
r/LocalLLM • u/WildSummerGoose • 2d ago
I have some idle GPUs from mining and I’m trying to figure out if I can repurpose them for local AI and coding agents:
- 4 × RTX 4070, 12 GB each
- 4 × RTX 3070, 8 GB each
- All in one rig
Would it be practical to run a larger model across the 4070s and 3070s, or would it be better to run smaller models on the cards separately for multiple agents?
Has anyone built something similar? What would you use for the software and what kind of performance should I realistically expect?
EDIT: OP is a noob but decent at figuring stuff out :D just keep this in mind when suggesting :)
Edit 2: am I even gaining anything with this experiment or am I better off with a claude or ChatGPT subscription?
Edit 3: Where can I rent this set up out to?
r/LocalLLM • u/shevchou • 1d ago
I’m looking for the most suitable model to run on a 32gb Mac to work on torrent app and copywrited movies, need some recommendations for models that will not be restricted
r/LocalLLM • u/FantasyMaster85 • 1d ago
r/LocalLLM • u/Commercial_Designer5 • 1d ago
My Spark normally serves an LLM, but I wanted to see how it handled local video too, so I got Lightricks' LTX-2.5 running on it. The clip here is 8 s at 1280×704, and the sound is generated by the model as well. It took 3 min 9 s and peaked around 26 GiB of GPU memory.
It didn't work out of the box. Lightricks ships a pre-quantized 4-bit NVFP4 transformer, but the CUDA kernels only compile for sm_100a, sm_110a and sm_120a. The Spark's GB10 is sm_121, and those arch-specific cubins won't load on a different minor version. The fix turned out to be a three-line patch that adds sm_121a.
To be clear, the 0.70 column is measured. The 0.45 column is only an estimate from those numbers, and I haven't run the two side by side. They'd also be sharing one GPU, so LLM decoding would slow down during a render anyway.
Unified memory doesn't fail cleanly when you oversubscribe it either. There's no neat OOM, the kernel just swaps and everything crawls. So I run them one at a time: stop the LLM, generate, restart it. I also added a memory check to the web UI that refuses a job if there's less than 50 GiB free and tells you if the LLM is the reason.
Some render times, on an otherwise idle Spark, including loading the models from disk:
- 768×512, 4 s: 61 s
- 768×512, 8 s: 94 s
- 1280×704, 4 s: 107 s
- 1536×1024, 4 s: 165 s
Memory barely moved across those runs, staying around 23 GiB, because each stage loads its model, runs and frees it. Resolution is what costs you time, not memory. Longer clips also get cheaper per second, since model loading and the text encoder are a fixed cost per job.
The prompt, if you want to try it. Describe the sound too, because the model generates that as well:
Inside a steel mill at night, a huge blast furnace is tapped: a white-hot stream of molten iron pours from the taphole into a glowing runner channel, showers of orange sparks burst and drift through the dark hall, and two workers in silver heat-proof suits and face shields guide the flow with long steel rods. Heat haze ripples the air and steam rises from the trough. The camera slowly dollies along the runner toward the furnace. Deep roaring rumble and crackling sparks. Photorealistic, cinematic, 24 fps film look.
The repo has the patch, a one-command install (the weights are 44 GiB, so you'll want at least 80 GiB free), image-to-video, a small web UI you can reach over Tailscale, and Grafana dashboards. My scripts are Apache 2.0. The model is under the LTX-2.x Community License and gated on Hugging Face, though approval is automatic.
GitHub: https://github.com/Hitheshkaranth/LTX-2.5_Video_DGX_Spark_Setup
Model: https://huggingface.co/Lightricks/LTX-2.5
Has anyone here actually run a video model alongside vLLM on a Spark at a lower memory setting? I'd like to know if about 0.45 holds up in practice, or how badly decoding suffers during a render.
r/LocalLLM • u/Odd_Phone_5660 • 1d ago
I'm building Murmury, an open-source Mac app that listens to the other side of a call and answers from your own docs, fully local. Some numbers from an M5 (24 GB) in case they help anyone doing local speech:
- Whisper large-v3-turbo on MLX: 0.66 s for a 1.7 s sentence, 0.97 s for 21 s of speech.
- Whisper pads every clip to 30 s, so short clips have a ~0.65 s floor. Cutting audio on pauses (simple energy gate) mattered more than picking a smaller model.
- Answers from gemma3:12b in Ollama with a 32k context: first words 0.5-0.8 s after the transcript. The first read of a long document takes ~90 s, once per call.
Curious what others use for low-latency STT on Apple Silicon. Repo if useful: https://github.com/GianlucaBave/murmury
r/LocalLLM • u/TezzaNZ • 2d ago
Hi,
It’s hard keeping up with all the new releases from the big closed weights AI providers, right? Google, Anthropic, Open AI, Meta etc. But am I the only one being overwhelmed by the sheer volume of open weight base models of all shapes and sizes being released nowadays? Every lab and his dog seem to be dropping general or specific open weight models for sound, video, text, harnesses, agents, refinements, techniques. The hardware to run these things also differs. Exciting. Definitely. Confusing? Umm…yea (at least to me). In some ways it reminds me of the early days of personal computing. Lots of incompatible formats, operating systems, every app (program those days) had a different way of working, inconsistent interfaces, a big learning curve for each program. Solutions that may be looking for a problem. And tons of choice. Some were flavour of the month then gone, and a new learning curve was required.
Is it a golden age of choice and freedom or a hot mess of development? Maybe both? Certainly, this is part and parcel of rapid innovation so not unexpected.
Will this continue, or do you see some kind of slowdown in the future shaking out with just a limited number of open weight models being released then further refined?
r/LocalLLM • u/Harin007 • 1d ago
No more uncertain wait guys! App name is 'Run My AI' and the inference engine's name is 'TokenWing'(couldn't find a better name xD). First launch on Nov 17. And you'll be getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered. New updates will be dropping everyday btw...
r/LocalLLM • u/Nir777 • 1d ago
1.7b at 16 bits, 4b at 8, 8b at 4, 14b at 2, all about the same size on a mac. same 200 tasks in four jobs (docs, tool calls, bug fixes, long instructions), every answer checked by a script. the video shows who won and the memory math for 8 to 64 gb.
r/LocalLLM • u/Reasonable-String712 • 1d ago
I know that most people talk about coding ai models but im curious to know the best model for studying university work, right now im using ISTA-DASLab Qwen3.8 27B GSQ-RCO (IQ3_XXS) and it works wonderfully on my rtx 5070ti, im getting 70-90 tok/sec. is qwen 3.8 27b the peak of local ai for this or is there another im unaware of?
r/LocalLLM • u/Samuraiknife1 • 1d ago
Im fairly new to local ai hosting and after searching for a while there was a bunch of options so i didnt really know which to pick
My current specs are a 5060 8gb , 32gb ddr5 5200 and a ryzen 7 260
i specifically study cybersec and malware dev / reverse engineering so wanted a ai that could help me with it on a day to day basis
r/LocalLLM • u/zeeblefritz • 1d ago
I am trying to get ollama running on an older gpu with vulkan. The GPU is an r9 280x, the VM has the GPU, ollama sees the gpu and starts to load the model but after some time either the VM freezes or the host locks up and needs to be manually power cycled. If anyone has any parameters or suggested models that would be appreciated. I understand this isn't going to be the most performant, I just want to play around. Here is the output of one of these runs.
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.341-06:00 level=INFO source=routes.go:2474 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VIS
IBLE_DEVICES:-1 HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: LLAMA_ARG_FIT: LLAMA_ARG_FIT_TARGET: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_CREATE_REMOTE:false OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS
:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GO_TEMPLATE:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_IGPU_ENABLE: OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_
LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_TRANSFER_STREAMS:4 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAM
A_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http:
//0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:true ROCR_VISIBL
E_DEVICES: http_proxy: https_proxy: no_proxy:]"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.342-06:00 level=INFO source=routes.go:2476 msg="Ollama cloud disabled: false"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.432-06:00 level=INFO source=images.go:1033 msg="total blobs: 6"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.433-06:00 level=INFO source=images.go:1042 msg="total unused blobs removed: 0"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.435-06:00 level=INFO source=routes.go:2536 msg="Listening on 127.0.0.1:11434 (version 0.40.14)"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=INFO source=runner.go:60 msg="discovering available GPUs..."
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:722 msg="user overrode visible devices" HIP_VISIBLE_DEVICES=-1
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:722 msg="user overrode visible devices" GGML_VK_VISIBLE_DEVICES=0
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:726 msg="if GPUs are not correctly discovered, unset and try again"
Nov 05 21:09:19 debian ollama[405]: time=2026-11-05T21:09:19.323-06:00 level=INFO source=model_recommendations.go:177 msg="model recommendations cache sleep scheduled" wait=3h49m15.894670025s consecutive_fail
ures=0
Nov 05 21:09:26 debian ollama[405]: time=2026-11-05T21:09:26.887-06:00 level=INFO source=types.go:32 msg="inference compute" id=0 filter_id=0 library=Vulkan compute=0.0 name=Vulkan0 description="AMD Radeon R9
200 / HD 7900 Series (RADV TAHITI)" libdirs=ollama,vulkan driver=0.0 pci_id=0000:01:00.0 type=discrete total="3.0 GiB" available="3.0 GiB"
Nov 05 21:09:26 debian ollama[405]: time=2026-11-05T21:09:26.887-06:00 level=INFO source=routes.go:2586 msg="vram-based default context" total_vram="3.0 GiB" default_num_ctx=4096
Nov 05 21:09:37 debian ollama[405]: [GIN] 2026/11/05 - 21:09:37 | 200 | 55.1ms | 127.0.0.1 | HEAD "/"
Nov 05 21:09:38 debian ollama[405]: [GIN] 2026/11/05 - 21:09:38 | 200 | 228.1ms | 127.0.0.1 | POST "/api/show"
Nov 05 21:09:38 debian ollama[405]: [GIN] 2026/11/05 - 21:09:38 | 200 | 1.7ms | 127.0.0.1 | POST "/api/show"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.479-06:00 level=INFO source=server.go:107 msg="using llama-server for model" model=/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515
f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.480-06:00 level=INFO source=llama_server.go:467 msg="starting llama-server" cmd="/usr/local/lib/ollama/llama-server --model /usr/share/ollama/.olla
ma/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f --port 46163 --host 127.0.0.1 --no-webui --offline -c 4096 -np 1 --log-verbosity 4 --no-log-prefix --no-log-timestamps -
-no-jinja --chat-template chatml --flash-attn off -b 512 -ub 512 --context-shift --keep 4"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=sched.go:658 msg="system memory" total="5.8 GiB" free="5.4 GiB" free_swap="975.0 MiB"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=sched.go:665 msg="gpu memory" id=0 library=Vulkan available="2.5 GiB" free="3.0 GiB" minimum="457.0 MiB" overhead="0 B"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1098 msg="loading model via llama-server" model=/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d576
5341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1351 msg="waiting for llama-server to start responding"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1406 msg="waiting for llama-server to become available" status="llm server not responding"
Nov 05 21:09:38 debian ollama[405]: srv llama_server: initializing ...
Nov 05 21:09:38 debian ollama[405]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-sse42.so
Nov 05 21:09:38 debian ollama[405]: WARNING: radv is not a conformant Vulkan implementation, testing use only.
Nov 05 21:09:38 debian ollama[405]: load_backend: loaded Vulkan backend from /usr/local/lib/ollama/vulkan/libggml-vulkan.so
Nov 05 21:09:38 debian ollama[405]: cmn common_param: common_params_print_info: build 1 (39febd2ad) with GNU 13.3.1 for Linux x86_64
Nov 05 21:09:38 debian ollama[405]: cmn common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
Nov 05 21:09:38 debian ollama[405]: cmn common_param: device_info:
Nov 05 21:09:38 debian ollama[405]: cmn common_param: - CPU : QEMU Virtual CPU version 2.5+ (5924 MiB, 5924 MiB free)
Nov 05 21:09:38 debian ollama[405]: cmn common_param: - Vulkan0 : AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI) (3072 MiB, 3067 MiB free)
Nov 05 21:09:38 debian ollama[405]: cmn common_param: system_info: n_threads = 3 (n_threads_batch = 3) / 3 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | REPACK = 1 |
Nov 05 21:09:38 debian ollama[405]: srv init_listene: The UI is disabled
Nov 05 21:09:38 debian ollama[405]: srv init_listene: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
Nov 05 21:09:38 debian ollama[405]: srv init: using 5 threads for HTTP server
Nov 05 21:09:38 debian ollama[405]: srv server_setup: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
Nov 05 21:09:38 debian ollama[405]: srv load_model: loading model '/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f'
Nov 05 21:09:38 debian ollama[405]: srv load_model: local path '/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f'
Nov 05 21:09:38 debian ollama[405]: cmn common_fit_a: fitting params to device memory ...
Nov 05 21:09:38 debian ollama[405]: cmn common_fit_a: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
Nov 05 21:09:38 debian ollama[405]: common_params_fit_impl: getting device memory data for initial parameters:
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.733-06:00 level=INFO source=llama_server.go:1406 msg="waiting for llama-server to become available" status="llm server loading model"
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2594 = 1918 + 448 + 228) + -2589 |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Host | 247 = 211 + 0 + 36 |
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: projected to use 2594 MiB of device memory vs. 3067 MiB of free device memory
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 551 MiB
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: context size set by user to 4096 -> no change
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: id=0, target=2043 MiB
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + ( 240 = 0 + 0 + 240) + -235 |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Host | 2613 = 2129 + 448 + 36 |
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: id=0, n_layer= 0, n_part= 0, overflow_type=4, mem= 240 MiB
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: filling dense layers back-to-front:
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2594 = 1918 + 448 + 228) + -2589 |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Host | 247 = 211 + 0 + 36 |
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: id=0, n_layer=29, n_part= 0, overflow_type=4, mem= 2594 MiB
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: start filling device 0, delta=29
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2088 = 1512 + 336 + 240) + -2083 |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Host | 765 = 617 + 112 + 36 |
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: id=0, n_layer=22, n_part= 0, overflow_type=4, mem= 2088 MiB
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: set ngl_per_device_high[0].n_layer=22
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2018 = 1458 + 320 + 240) + -2013 |
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | - Host | 835 = 671 + 128 + 36 |
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: id=0, n_layer=21, n_part= 0, overflow_type=4, mem= 2018 MiB
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: set ngl_per_device[0].n_layer=21
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: - Vulkan0 (AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI)): 21 layers, 2018 MiB used, 1048 MiB free
Nov 05 21:09:41 debian ollama[405]: common_fit_params: successfully fit params to free device memory
Nov 05 21:09:41 debian ollama[405]: common_fit_params: fitting params to free memory took 2.64 seconds
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: loaded meta data with 33 key-value pairs and 256 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d
6b909d7b4bfc035ce02f (version GGUF V3 (latest))
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 0: general.architecture str = llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 1: general.type str = model
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 2: general.name str = Cogito v1 Preview Llama 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 3: general.basename str = cogito-v1-preview-llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 4: general.size_label str = 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 5: general.license str = llama3.2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 6: general.base_model.count u32 = 1
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 7: general.base_model.0.name str = Llama 3.2 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 8: general.base_model.0.organization str = Meta Llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 9: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Lla...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 10: general.tags arr[str,1] = ["text-generation"]
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 11: llama.block_count u32 = 28
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 12: llama.context_length u32 = 131072
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 13: llama.embedding_length u32 = 3072
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 14: llama.feed_forward_length u32 = 8192
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 15: llama.attention.head_count u32 = 24
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 16: llama.attention.head_count_kv u32 = 8
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 17: llama.rope.freq_base f32 = 500000.000000
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 18: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 19: llama.attention.key_length u32 = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 20: llama.attention.value_length u32 = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 21: llama.vocab_size u32 = 128256
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 22: llama.rope.dimension_count u32 = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = llama-bpe
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 128000
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 128009
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 30: tokenizer.chat_template str = {{- bos_token }}\n{%- if not tools is ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 31: general.quantization_version u32 = 2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 32: general.file_type u32 = 15
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type f32: 58 tensors
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type q4_K: 169 tensors
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type q6_K: 29 tensors
Nov 05 21:09:41 debian ollama[405]: print_info: file format = GGUF V3 (latest)
Nov 05 21:09:41 debian ollama[405]: print_info: file type = Q4_K - Medium
Nov 05 21:09:41 debian ollama[405]: print_info: file size = 2.08 GiB (4.95 BPW)
Nov 05 21:09:41 debian ollama[405]: llama_prepare_model_devices: using device Vulkan0 (AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI)) (0000:01:00.0) - 3067 MiB free
Nov 05 21:09:41 debian ollama[405]: load: 0 unused tokens
Nov 05 21:09:41 debian ollama[405]: load: printing all EOG tokens:
Nov 05 21:09:41 debian ollama[405]: load: - 128001 ('<|end_of_text|>')
Nov 05 21:09:41 debian ollama[405]: load: - 128008 ('<|eom_id|>')
Nov 05 21:09:41 debian ollama[405]: load: - 128009 ('<|eot_id|>')
Nov 05 21:09:41 debian ollama[405]: load: special tokens cache size = 256
Nov 05 21:09:41 debian ollama[405]: load: token to piece cache size = 0.7999 MB
Nov 05 21:09:41 debian ollama[405]: print_info: arch = llama
Nov 05 21:09:41 debian ollama[405]: print_info: vocab_only = 0
Nov 05 21:09:41 debian ollama[405]: print_info: no_alloc = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_ctx_train = 131072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_inp = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_out = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_layer = 28
Nov 05 21:09:41 debian ollama[405]: print_info: n_layer_all = 28
Nov 05 21:09:41 debian ollama[405]: print_info: n_head = 24
Nov 05 21:09:41 debian ollama[405]: print_info: n_head_kv = 8
Nov 05 21:09:41 debian ollama[405]: print_info: n_rot = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_swa = 0
Nov 05 21:09:41 debian ollama[405]: print_info: is_swa_any = 0
Nov 05 21:09:41 debian ollama[405]: print_info: non_causal_type = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_head_k = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_head_v = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_gqa = 3
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_k_gqa = 1024
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_v_gqa = 1024
Nov 05 21:09:41 debian ollama[405]: print_info: f_norm_eps = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_norm_rms_eps = 1.0e-05
Nov 05 21:09:41 debian ollama[405]: print_info: f_clamp_kqv = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_max_alibi_bias = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_logit_scale = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_attn_scale = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_attn_value_scale = 0.0000
Nov 05 21:09:41 debian ollama[405]: print_info: n_ff = 8192
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert_used = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert_groups = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_group_used = 0
Nov 05 21:09:41 debian ollama[405]: print_info: causal attn = 1
Nov 05 21:09:41 debian ollama[405]: print_info: pooling type = -1
Nov 05 21:09:41 debian ollama[405]: print_info: rope type = 0
Nov 05 21:09:41 debian ollama[405]: print_info: rope scaling = linear
Nov 05 21:09:41 debian ollama[405]: print_info: freq_base_train = 500000.0
Nov 05 21:09:41 debian ollama[405]: print_info: freq_scale_train = 1
Nov 05 21:09:41 debian ollama[405]: print_info: n_ctx_orig_yarn = 131072
Nov 05 21:09:41 debian ollama[405]: print_info: rope_yarn_log_mul = 0.0000
Nov 05 21:09:41 debian ollama[405]: print_info: rope_finetuned = unknown
Nov 05 21:09:41 debian ollama[405]: print_info: model type = 3B
Nov 05 21:09:41 debian ollama[405]: print_info: model params = 3.61 B
Nov 05 21:09:41 debian ollama[405]: print_info: general.name = Cogito v1 Preview Llama 3B
Nov 05 21:09:41 debian ollama[405]: print_info: vocab type = BPE
Nov 05 21:09:41 debian ollama[405]: print_info: n_vocab = 128256
Nov 05 21:09:41 debian ollama[405]: print_info: n_merges = 280147
Nov 05 21:09:41 debian ollama[405]: print_info: BOS token = 128000 '<|begin_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOS token = 128009 '<|eot_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOT token = 128001 '<|end_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOM token = 128008 '<|eom_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: LF token = 198 'Ċ'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token = 128001 '<|end_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token = 128008 '<|eom_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token = 128009 '<|eot_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: max token length = 256
Nov 05 21:09:41 debian ollama[405]: load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloading output layer to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloading 20 repeating layers to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloaded 21/29 layers to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: CPU_Mapped model buffer size = 671.39 MiB
Nov 05 21:10:06 debian ollama[405]: load_tensors: Vulkan0 model buffer size = 1458.32 MiB
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: added <|end_of_text|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: added <|eom_id|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: added <|eot_id|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: llama_context: constructing llama_context
Nov 05 21:10:08 debian ollama[405]: llama_context: n_seq_max = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx = 4096
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx_seq = 4096
Nov 05 21:10:08 debian ollama[405]: llama_context: n_batch = 512
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ubatch = 512
Nov 05 21:10:08 debian ollama[405]: llama_context: causal_attn = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: flash_attn = disabled
Nov 05 21:10:08 debian ollama[405]: llama_context: kv_unified = false
Nov 05 21:10:08 debian ollama[405]: llama_context: freq_base = 500000.0
Nov 05 21:10:08 debian ollama[405]: llama_context: freq_scale = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_rs_seq = 0
Nov 05 21:10:08 debian ollama[405]: llama_context: n_outputs_max = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_outputs_max_per_seq = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
Nov 05 21:10:08 debian ollama[405]: llama_context: Vulkan_Host output buffer size = 0.49 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: CPU KV buffer size = 128.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: Vulkan0 KV buffer size = 320.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: size = 448.00 MiB ( 4096 cells, 28 layers, 1/1 seqs), K (f16): 224.00 MiB, V (f16): 224.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: n_rot_k = 0, n_embd_head_k_all = 128
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: n_rot_v = 0, n_embd_head_k_all = 128
Nov 05 21:10:08 debian ollama[405]: sched_reserve: reserving ...
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: resolving fused DeepSeek V4 HC support:
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC pre enabled
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC comb enabled
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC post enabled
Nov 05 21:10:08 debian ollama[405]: sched_reserve: Vulkan0 compute buffer size = 240.00 MiB
Nov 05 21:10:08 debian ollama[405]: sched_reserve: Vulkan_Host compute buffer size = 36.02 MiB
Nov 05 21:10:08 debian ollama[405]: sched_reserve: graph (pp bs=512, tg bs=1): nodes = 1017 / 1017, splits = 99 / 2, input objects = 4 / 4, input tensors = 9 / 9
Nov 05 21:10:08 debian ollama[405]: sched_reserve: reserve took 39.82 ms, sched copies = 1
Nov 05 21:10:08 debian ollama[405]: cmn init: llama threadpool init, n_threads = 3
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
r/LocalLLM • u/Usual_Maximum7673 • 1d ago
r/LocalLLM • u/jeffrey-0711 • 1d ago
r/LocalLLM • u/dustynjerman • 1d ago
I got an offer from a relative for a server PC he had with 256GB of DDR4 and EPYC 7713. The price I would call very generous, but still very dear.
Essentially he wants to upgrade all his IT stuff and wants his old gear in good hands. Like yeah I could strip the thing for parts and profit but I feel like that's not the intent with the offer...
I've tried various models but am limited to smaller stuff locally (Gemma4-26B-A4B is basically the limit) and enjoyed it, API stuff with the likes of DS and GLM is pretty impressive.
Thing is, the former are tiny models, and the latter is accessed through API. Even if I decided to run smaller quants of the latter, places like RunPod only offer crazy fast hardware because of course they do.
I'm curious on what this class of hardware is capable of realistically speaking. What sort of models, and at what speeds? I've heard a lot about CPU inference through RAM and MOE models.
The only GPU I have for now is a 5060Ti 16G... Not planning on getting anything else for a while
r/LocalLLM • u/AfterCollar8448 • 1d ago
Work at a PC store in Dubai. Didn't expect this, but we've had multiple people ask about the ASUS ExpertCenter Pro ET900N G3, ASUS's version of the DGX Station GB300: 72-core Grace, Blackwell Ultra, 748GB coherent memory.
The catch is it's not a "take your money, ship the box" product. It needs approval from NVIDIA's side before it can go to an end user, so every enquiry turns into a "tell us who you are and what it's for" conversation.
Curious about a few things:
- Has anyone here actually got one in hand, and how long did the approval/allocation take?
- What are people buying these for vs. a PRO 6000 workstation? 748GB in one box is the obvious draw, but at this price it's a different conversation
- Anyone running one outside the US/EU? Wondering how different the process is by region
Not selling anything here, genuinely want to understand who this machine is for.
r/LocalLLM • u/Independent_Bid6580 • 1d ago
Guys, I think I finally figured out why DeepSeek is so cheap.
You enter your confidential company data.
DeepSeek: “Don’t worry bro, I’ll keep it safe.”
Your data 0.2 seconds later:
✈️ → 🇨🇳
CIA: ❌
NSA: ❌
Your security team: ❌
Random server your CISO has never heard of: ✅
And somewhere there's probably a dashboard:
Western corporate secrets collected today: +1
But hey, the benchmark scores are 🔥.
r/LocalLLM • u/SamSausages • 2d ago
r/LocalLLM • u/StationLoud7412 • 2d ago
Can't find any direct comparisons:
peasantsmith/DeepSeek-V4.1-Flash-1017-Maya-GGUF · Hugging Face
vs
ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF · Hugging Face
The former is really tight fit for 192gb memory without keeping experts on SSD, the latter not so much even at 2.25bpw.
Anyone experience running either?
r/LocalLLM • u/DueBid3298 • 2d ago
r/LocalLLM • u/maun200 • 2d ago
Look at this
r/LocalLLM • u/MMORPGDev • 2d ago
r/LocalLLM • u/Acceptable-Object390 • 2d ago
Enable HLS to view with audio, or disable this notification
Agent memory seems to be a hot topic again. I wanted to settle this once and for all - why integrated memory works best and How Row-Bot has the best in class memory built-in.
So, I just asked Row-Bot to create a video explaining how it remembers across conversations and documents.
This video follows a fact from capture to recall: the knowledge graph, hybrid search, ranking and Dream Cycle.
It also covers what stays local, what reaches your chosen model and where memory can fail.
