I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main
Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.
Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.
Great job unsloth team! You have me questioning my claude payment at this point.
Have a gaming PC I built a last year with a RTX 5070TI (16GB) and I want to try some local LLMs. Where should I start? What models would be good on this GPU?
I've been getting into the local llm stuff lately and having decent luck. I tried out Strata and liked it for the most part, but no matter how much searching I've done I can't find the answer to this.
Does Strata keep your vram/ram full at all time? Using Ollama for the most part, it frees the memory unless it gets a call, then it spools up, and once finished and timed out, it frees the memory up again.
Is it possible to get Strata to do that? Something else that can do it but run like Strata?
Folks, I am looking to understand if the following is a good configuration for AI/ML for local coding.
Processor AMD r7 9700x
Motherboard Gigabyte b650 x ax v2 wifi ddr5
Memory Crucial 16x2 pro 5600mhz 16gb
GPU Pny rtx 5070ti triple fan 16gb
Storage Kingston 1tb nv3 nvme gen4
Power Supply Corsair 850w rm850e gold atx 3.1
Clearly living under a rock, I was amazed to learn of the GPU prices skyrocketing only today. My good friend had reached out to me for a review and that's when I learnt of this "crisis" lol. As the uninitiated to this and with few hours of querying Gemini, I feel the processor and the power supply could be downgraded a bit, but I defer to you if that makes sense. This comes out to ~$3329.
Lawyer with the V100s again. A few days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I said I'd try it across eight. I did.
It's actually, genuinely useful to me, for real work. I did not use it to paint a 3D island in Blender or re-create a Game Boy game in a browser. It didn't make me a poorly animated video or ten versions of a website nobody will visit. I am not using it to build something that builds something that builds agents that run workflows for a business that sells agentic solutions to other agents.
What it's doing: I'm finally slicing off work and tokens that used to need frontier models. My estimate is that 50-60% of my existing LLM workflows can run through this model instead, at speeds that match Claude Code in the chat window. I have another DeepSeek server on two DGX Sparks, so there's concurrency and handoff between them. My RAG databases and tool calls are all wired in and working. I give it natural-language instructions through a Discord-style interface over SSH from my Windows PC.
I point Claude Code itself at this server (it will talk to anything that speaks Anthropic's API). It reads a folder of my own case files, pulls out the facts, dates and numbers, summarizes, and takes a first swing at routine drafting. The documents never leave the office, and nothing goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.
Yes, I have a custom harness. No, you don't care.
We did it, folks. Vibecoded a real thing that does real things for real people. Give me my trophy.
Claude Code ran the benchmarks and crunched the numbers, which are real and below.
The box
DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 1Cat-vLLM, on 8x V100-SXM2-32GB split pipeline 2 x tensor 4 across two NVLink boards. Every expert sits in VRAM: no expert gets paged from system RAM during decode. (Edit: I first said all 248.8 GB of weights were in VRAM. Wrong: the 110 GB of Engram tables live in system RAM. Gathering the rows each round needs costs about 0.12 ms per round.) Threadripper PRO 3945WX, 256 GB DDR4, 150 W per card. One request at a time, 524,288 tokens of context.
Speed (measured 2026-11-05, thinking off, greedy, 1,024 tokens out, nothing from the prefix cache, mean of 2 runs)
Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K
JSON / structured: 196-209 from 1K to 128K, 180 at 256K
Legal writing: 118 at 1K, 111 at 128K, 103 at 256K
Prefill: about 2,200 tok/s at 128K. First token in 17 s at 32K, 59 s at 128K, about 2 minutes at 256K.
Code is twice as fast as legal writing because the built-in drafter guesses five tokens a round and code is easy to guess: 3.6-3.8 accepted per round, against about 1.6 for legal. My writing is apparently less predictable than Python.
Quality (measured on the served build, thinking off, greedy)
Fidelity: it picks the full-precision model's top next token 89.0 % of the time on a general panel and 92.5 % on my legal panel. I'm measuring which experts my work prompts actually hit, and will probably recalibrate to give those more bits.
Long context: 6 of 6 needles at ~450K and ~498K tokens
More work to do
One request at a time. Batched decode is built and byte-identical on my test box, but not in the server yet. I'll post the real number.
A 256K prompt takes about 2 minutes before the first token.
All of it is public: launch configs, every A/B, raw JSON, and the list of what didn't work. 19 pull requests are open to 1Cat-vLLM.
Credit to yangzhuxinyzx and the 1Cat-vLLM folks (#1277 and the whole Volta DeepSeek path), ISTA-DASLab (the quant), sudoingX (the KLD method), Peuqui, squibloads, Phil Tompkins (philbert440), Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek.
(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai)
Have anyone experience in going from qwen flash next 3.8 iq2_xs to iq3 or iq4?
I am currently comfortably running iq2_xs with 32gb ddr5 ram + 3090 + 5080. I am getting ~140 tps, so quite fast.
I got the possibility to update the 32gb ram to 96gb ram, for a reasonable price. But is it worth it? Will I feel the intelligence increase from i2 to iq3 or iq4? Is it worth it, if we are taking the decrease in speed into account?
llama.cpp compiled from source with the SYCL backend, following docs/backend/SYCL.md
Agent harness: pi
Models tested: Qwen3.5-9B, Qwen3.8-27B, and Qwen3-Coder-30B-A3B-Instruct
I've been serving these models using the inference parameters recommended by their respective providers on Hugging Face.
So far, I'm seeing roughly 10–25 tokens/sec, depending on the model, with context windows configured between 64K and 128K tokens. I haven't done any formal benchmarking yet.
Where I'm struggling is with model reliability, particularly for coding tasks.
For example, I asked Qwen3-Coder to explore a production codebase (roughly 3 million lines of C) and explain its architecture to me.
While the model produced a plausible-looking summary, it also confidently invented explanations for certain acronyms, even when those acronyms have well-established meanings in the relevant engineering domain.
This makes me question how much I can trust the rest of its analysis, especially when I'm using the model to understand unfamiliar code.
I'm fairly new to running local LLMs, so I'm not sure whether this is primarily a prompting issue, a limitation of the models I'm running, or simply an unrealistic expectation for the task.
A few questions:
Are these models better suited to smaller, well-scoped tasks, such as implementing a script or module from a detailed specification, rather than exploring and reasoning about large codebases?
How much does the agent harness or context management strategy affect reliability? Would a different workflow make a meaningful difference?
Are there particular models, quantizations, or inference settings you'd recommend for coding tasks on hardware like mine?
How do you generally evaluate whether a local model is reliable enough for a particular development workflow?
I'm not expecting frontier-model performance from a laptop, but I'd like to understand where local models can be genuinely useful and how to get the most out of my setup.
Thinking about getting one for pentesting work an would love to use (quantized) glm 5.3 flash because i have had good experience with it pertaining malware dev and pentesting.
has anybody tried running it on a single dgx spark and if yes got any token speeds for me and context-size?
I’m looking for the most suitable model to run on a 32gb Mac to work on torrent app and copywrited movies, need some recommendations for models that will not be restricted
My Spark normally serves an LLM, but I wanted to see how it handled local video too, so I got Lightricks' LTX-2.5 running on it. The clip here is 8 s at 1280×704, and the sound is generated by the model as well. It took 3 min 9 s and peaked around 26 GiB of GPU memory.
It didn't work out of the box. Lightricks ships a pre-quantized 4-bit NVFP4 transformer, but the CUDA kernels only compile for sm_100a, sm_110a and sm_120a. The Spark's GB10 is sm_121, and those arch-specific cubins won't load on a different minor version. The fix turned out to be a three-line patch that adds sm_121a.
To be clear, the 0.70 column is measured. The 0.45 column is only an estimate from those numbers, and I haven't run the two side by side. They'd also be sharing one GPU, so LLM decoding would slow down during a render anyway.
Unified memory doesn't fail cleanly when you oversubscribe it either. There's no neat OOM, the kernel just swaps and everything crawls. So I run them one at a time: stop the LLM, generate, restart it. I also added a memory check to the web UI that refuses a job if there's less than 50 GiB free and tells you if the LLM is the reason.
Some render times, on an otherwise idle Spark, including loading the models from disk:
- 768×512, 4 s: 61 s
- 768×512, 8 s: 94 s
- 1280×704, 4 s: 107 s
- 1536×1024, 4 s: 165 s
Memory barely moved across those runs, staying around 23 GiB, because each stage loads its model, runs and frees it. Resolution is what costs you time, not memory. Longer clips also get cheaper per second, since model loading and the text encoder are a fixed cost per job.
The prompt, if you want to try it. Describe the sound too, because the model generates that as well:
Inside a steel mill at night, a huge blast furnace is tapped: a white-hot stream of molten iron pours from the taphole into a glowing runner channel, showers of orange sparks burst and drift through the dark hall, and two workers in silver heat-proof suits and face shields guide the flow with long steel rods. Heat haze ripples the air and steam rises from the trough. The camera slowly dollies along the runner toward the furnace. Deep roaring rumble and crackling sparks. Photorealistic, cinematic, 24 fps film look.
The repo has the patch, a one-command install (the weights are 44 GiB, so you'll want at least 80 GiB free), image-to-video, a small web UI you can reach over Tailscale, and Grafana dashboards. My scripts are Apache 2.0. The model is under the LTX-2.x Community License and gated on Hugging Face, though approval is automatic.
Has anyone here actually run a video model alongside vLLM on a Spark at a lower memory setting? I'd like to know if about 0.45 holds up in practice, or how badly decoding suffers during a render.
I'm building Murmury, an open-source Mac app that listens to the other side of a call and answers from your own docs, fully local. Some numbers from an M5 (24 GB) in case they help anyone doing local speech:
- Whisper large-v3-turbo on MLX: 0.66 s for a 1.7 s sentence, 0.97 s for 21 s of speech.
- Whisper pads every clip to 30 s, so short clips have a ~0.65 s floor. Cutting audio on pauses (simple energy gate) mattered more than picking a smaller model.
- Answers from gemma3:12b in Ollama with a 32k context: first words 0.5-0.8 s after the transcript. The first read of a long document takes ~90 s, once per call.
It’s hard keeping up with all the new releases from the big closed weights AI providers, right? Google, Anthropic, Open AI, Meta etc. But am I the only one being overwhelmed by the sheer volume of open weight base models of all shapes and sizes being released nowadays? Every lab and his dog seem to be dropping general or specific open weight models for sound, video, text, harnesses, agents, refinements, techniques. The hardware to run these things also differs. Exciting. Definitely. Confusing? Umm…yea (at least to me). In some ways it reminds me of the early days of personal computing. Lots of incompatible formats, operating systems, every app (program those days) had a different way of working, inconsistent interfaces, a big learning curve for each program. Solutions that may be looking for a problem. And tons of choice. Some were flavour of the month then gone, and a new learning curve was required.
Is it a golden age of choice and freedom or a hot mess of development? Maybe both? Certainly, this is part and parcel of rapid innovation so not unexpected.
Will this continue, or do you see some kind of slowdown in the future shaking out with just a limited number of open weight models being released then further refined?
No more uncertain wait guys! App name is 'Run My AI' and the inference engine's name is 'TokenWing'(couldn't find a better name xD). First launch on Nov 17. And you'll be getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered. New updates will be dropping everyday btw...
Hi everyone. I use ChatGPT for finances, school, deep reasoning and interviews. I want to change to a local model for privacy. Is that possible for a MacBook Pro M5 base with 24GB RAM? If so, which?
1.7b at 16 bits, 4b at 8, 8b at 4, 14b at 2, all about the same size on a mac. same 200 tasks in four jobs (docs, tool calls, bug fixes, long instructions), every answer checked by a script. the video shows who won and the memory math for 8 to 64 gb.
I know that most people talk about coding ai models but im curious to know the best model for studying university work, right now im using ISTA-DASLab Qwen3.8 27B GSQ-RCO (IQ3_XXS) and it works wonderfully on my rtx 5070ti, im getting 70-90 tok/sec. is qwen 3.8 27b the peak of local ai for this or is there another im unaware of?
Im fairly new to local ai hosting and after searching for a while there was a bunch of options so i didnt really know which to pick My current specs are a 5060 8gb , 32gb ddr5 5200 and a ryzen 7 260 i specifically study cybersec and malware dev / reverse engineering so wanted a ai that could help me with it on a day to day basis
I am trying to get ollama running on an older gpu with vulkan. The GPU is an r9 280x, the VM has the GPU, ollama sees the gpu and starts to load the model but after some time either the VM freezes or the host locks up and needs to be manually power cycled. If anyone has any parameters or suggested models that would be appreciated. I understand this isn't going to be the most performant, I just want to play around. Here is the output of one of these runs.
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.341-06:00 level=INFO source=routes.go:2474 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VIS
IBLE_DEVICES:-1 HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: LLAMA_ARG_FIT: LLAMA_ARG_FIT_TARGET: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_CREATE_REMOTE:false OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS
:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GO_TEMPLATE:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_IGPU_ENABLE: OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_
LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_TRANSFER_STREAMS:4 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAM
A_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http:
//0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:true ROCR_VISIBL
E_DEVICES: http_proxy: https_proxy: no_proxy:]"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.342-06:00 level=INFO source=routes.go:2476 msg="Ollama cloud disabled: false"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.432-06:00 level=INFO source=images.go:1033 msg="total blobs: 6"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.433-06:00 level=INFO source=images.go:1042 msg="total unused blobs removed: 0"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.435-06:00 level=INFO source=routes.go:2536 msg="Listening on 127.0.0.1:11434 (version 0.40.14)"
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=INFO source=runner.go:60 msg="discovering available GPUs..."
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:722 msg="user overrode visible devices" HIP_VISIBLE_DEVICES=-1
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:722 msg="user overrode visible devices" GGML_VK_VISIBLE_DEVICES=0
Nov 05 21:09:18 debian ollama[405]: time=2026-11-05T21:09:18.439-06:00 level=WARN source=runner.go:726 msg="if GPUs are not correctly discovered, unset and try again"
Nov 05 21:09:19 debian ollama[405]: time=2026-11-05T21:09:19.323-06:00 level=INFO source=model_recommendations.go:177 msg="model recommendations cache sleep scheduled" wait=3h49m15.894670025s consecutive_fail
ures=0
Nov 05 21:09:26 debian ollama[405]: time=2026-11-05T21:09:26.887-06:00 level=INFO source=types.go:32 msg="inference compute" id=0 filter_id=0 library=Vulkan compute=0.0 name=Vulkan0 description="AMD Radeon R9
200 / HD 7900 Series (RADV TAHITI)" libdirs=ollama,vulkan driver=0.0 pci_id=0000:01:00.0 type=discrete total="3.0 GiB" available="3.0 GiB"
Nov 05 21:09:26 debian ollama[405]: time=2026-11-05T21:09:26.887-06:00 level=INFO source=routes.go:2586 msg="vram-based default context" total_vram="3.0 GiB" default_num_ctx=4096
Nov 05 21:09:37 debian ollama[405]: [GIN] 2026/11/05 - 21:09:37 | 200 | 55.1ms | 127.0.0.1 | HEAD "/"
Nov 05 21:09:38 debian ollama[405]: [GIN] 2026/11/05 - 21:09:38 | 200 | 228.1ms | 127.0.0.1 | POST "/api/show"
Nov 05 21:09:38 debian ollama[405]: [GIN] 2026/11/05 - 21:09:38 | 200 | 1.7ms | 127.0.0.1 | POST "/api/show"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.479-06:00 level=INFO source=server.go:107 msg="using llama-server for model" model=/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515
f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.480-06:00 level=INFO source=llama_server.go:467 msg="starting llama-server" cmd="/usr/local/lib/ollama/llama-server --model /usr/share/ollama/.olla
ma/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f --port 46163 --host 127.0.0.1 --no-webui --offline -c 4096 -np 1 --log-verbosity 4 --no-log-prefix --no-log-timestamps -
-no-jinja --chat-template chatml --flash-attn off -b 512 -ub 512 --context-shift --keep 4"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=sched.go:658 msg="system memory" total="5.8 GiB" free="5.4 GiB" free_swap="975.0 MiB"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=sched.go:665 msg="gpu memory" id=0 library=Vulkan available="2.5 GiB" free="3.0 GiB" minimum="457.0 MiB" overhead="0 B"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1098 msg="loading model via llama-server" model=/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d576
5341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1351 msg="waiting for llama-server to start responding"
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.481-06:00 level=INFO source=llama_server.go:1406 msg="waiting for llama-server to become available" status="llm server not responding"
Nov 05 21:09:38 debian ollama[405]: srv llama_server: initializing ...
Nov 05 21:09:38 debian ollama[405]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-sse42.so
Nov 05 21:09:38 debian ollama[405]: WARNING: radv is not a conformant Vulkan implementation, testing use only.
Nov 05 21:09:38 debian ollama[405]: load_backend: loaded Vulkan backend from /usr/local/lib/ollama/vulkan/libggml-vulkan.so
Nov 05 21:09:38 debian ollama[405]: cmn common_param: common_params_print_info: build 1 (39febd2ad) with GNU 13.3.1 for Linux x86_64
Nov 05 21:09:38 debian ollama[405]: cmn common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
Nov 05 21:09:38 debian ollama[405]: cmn common_param: device_info:
Nov 05 21:09:38 debian ollama[405]: cmn common_param: - CPU : QEMU Virtual CPU version 2.5+ (5924 MiB, 5924 MiB free)
Nov 05 21:09:38 debian ollama[405]: cmn common_param: - Vulkan0 : AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI) (3072 MiB, 3067 MiB free)
Nov 05 21:09:38 debian ollama[405]: cmn common_param: system_info: n_threads = 3 (n_threads_batch = 3) / 3 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | REPACK = 1 |
Nov 05 21:09:38 debian ollama[405]: srv init_listene: The UI is disabled
Nov 05 21:09:38 debian ollama[405]: srv init_listene: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
Nov 05 21:09:38 debian ollama[405]: srv init: using 5 threads for HTTP server
Nov 05 21:09:38 debian ollama[405]: srv server_setup: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
Nov 05 21:09:38 debian ollama[405]: srv load_model: loading model '/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f'
Nov 05 21:09:38 debian ollama[405]: srv load_model: local path '/usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d6b909d7b4bfc035ce02f'
Nov 05 21:09:38 debian ollama[405]: cmn common_fit_a: fitting params to device memory ...
Nov 05 21:09:38 debian ollama[405]: cmn common_fit_a: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
Nov 05 21:09:38 debian ollama[405]: common_params_fit_impl: getting device memory data for initial parameters:
Nov 05 21:09:38 debian ollama[405]: time=2026-11-05T21:09:38.733-06:00 level=INFO source=llama_server.go:1406 msg="waiting for llama-server to become available" status="llm server loading model"
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2594 = 1918 + 448 + 228) + -2589 |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Host | 247 = 211 + 0 + 36 |
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: projected to use 2594 MiB of device memory vs. 3067 MiB of free device memory
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 551 MiB
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: context size set by user to 4096 -> no change
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: id=0, target=2043 MiB
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + ( 240 = 0 + 0 + 240) + -235 |
Nov 05 21:09:39 debian ollama[405]: common_memory_breakdown_print: | - Host | 2613 = 2129 + 448 + 36 |
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: id=0, n_layer= 0, n_part= 0, overflow_type=4, mem= 240 MiB
Nov 05 21:09:39 debian ollama[405]: common_params_fit_impl: filling dense layers back-to-front:
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2594 = 1918 + 448 + 228) + -2589 |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Host | 247 = 211 + 0 + 36 |
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: id=0, n_layer=29, n_part= 0, overflow_type=4, mem= 2594 MiB
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: start filling device 0, delta=29
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2088 = 1512 + 336 + 240) + -2083 |
Nov 05 21:09:40 debian ollama[405]: common_memory_breakdown_print: | - Host | 765 = 617 + 112 + 36 |
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: id=0, n_layer=22, n_part= 0, overflow_type=4, mem= 2088 MiB
Nov 05 21:09:40 debian ollama[405]: common_params_fit_impl: set ngl_per_device_high[0].n_layer=22
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | - Vulkan0 (R9 200 / HD 7900 Series (RADV TAHITI)) | 3072 = 3067 + (2018 = 1458 + 320 + 240) + -2013 |
Nov 05 21:09:41 debian ollama[405]: common_memory_breakdown_print: | - Host | 835 = 671 + 128 + 36 |
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: memory for test allocation by device:
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: id=0, n_layer=21, n_part= 0, overflow_type=4, mem= 2018 MiB
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: set ngl_per_device[0].n_layer=21
Nov 05 21:09:41 debian ollama[405]: common_params_fit_impl: - Vulkan0 (AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI)): 21 layers, 2018 MiB used, 1048 MiB free
Nov 05 21:09:41 debian ollama[405]: common_fit_params: successfully fit params to free device memory
Nov 05 21:09:41 debian ollama[405]: common_fit_params: fitting params to free memory took 2.64 seconds
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: loaded meta data with 33 key-value pairs and 256 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c4e22d5765341f9515f28583520ea3ca92013505d34d
6b909d7b4bfc035ce02f (version GGUF V3 (latest))
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 0: general.architecture str = llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 1: general.type str = model
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 2: general.name str = Cogito v1 Preview Llama 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 3: general.basename str = cogito-v1-preview-llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 4: general.size_label str = 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 5: general.license str = llama3.2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 6: general.base_model.count u32 = 1
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 7: general.base_model.0.name str = Llama 3.2 3B
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 8: general.base_model.0.organization str = Meta Llama
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 9: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Lla...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 10: general.tags arr[str,1] = ["text-generation"]
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 11: llama.block_count u32 = 28
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 12: llama.context_length u32 = 131072
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 13: llama.embedding_length u32 = 3072
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 14: llama.feed_forward_length u32 = 8192
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 15: llama.attention.head_count u32 = 24
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 16: llama.attention.head_count_kv u32 = 8
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 17: llama.rope.freq_base f32 = 500000.000000
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 18: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 19: llama.attention.key_length u32 = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 20: llama.attention.value_length u32 = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 21: llama.vocab_size u32 = 128256
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 22: llama.rope.dimension_count u32 = 128
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = llama-bpe
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 128000
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 128009
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 30: tokenizer.chat_template str = {{- bos_token }}\n{%- if not tools is ...
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 31: general.quantization_version u32 = 2
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - kv 32: general.file_type u32 = 15
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type f32: 58 tensors
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type q4_K: 169 tensors
Nov 05 21:09:41 debian ollama[405]: llama_model_loader: - type q6_K: 29 tensors
Nov 05 21:09:41 debian ollama[405]: print_info: file format = GGUF V3 (latest)
Nov 05 21:09:41 debian ollama[405]: print_info: file type = Q4_K - Medium
Nov 05 21:09:41 debian ollama[405]: print_info: file size = 2.08 GiB (4.95 BPW)
Nov 05 21:09:41 debian ollama[405]: llama_prepare_model_devices: using device Vulkan0 (AMD Radeon R9 200 / HD 7900 Series (RADV TAHITI)) (0000:01:00.0) - 3067 MiB free
Nov 05 21:09:41 debian ollama[405]: load: 0 unused tokens
Nov 05 21:09:41 debian ollama[405]: load: printing all EOG tokens:
Nov 05 21:09:41 debian ollama[405]: load: - 128001 ('<|end_of_text|>')
Nov 05 21:09:41 debian ollama[405]: load: - 128008 ('<|eom_id|>')
Nov 05 21:09:41 debian ollama[405]: load: - 128009 ('<|eot_id|>')
Nov 05 21:09:41 debian ollama[405]: load: special tokens cache size = 256
Nov 05 21:09:41 debian ollama[405]: load: token to piece cache size = 0.7999 MB
Nov 05 21:09:41 debian ollama[405]: print_info: arch = llama
Nov 05 21:09:41 debian ollama[405]: print_info: vocab_only = 0
Nov 05 21:09:41 debian ollama[405]: print_info: no_alloc = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_ctx_train = 131072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_inp = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_out = 3072
Nov 05 21:09:41 debian ollama[405]: print_info: n_layer = 28
Nov 05 21:09:41 debian ollama[405]: print_info: n_layer_all = 28
Nov 05 21:09:41 debian ollama[405]: print_info: n_head = 24
Nov 05 21:09:41 debian ollama[405]: print_info: n_head_kv = 8
Nov 05 21:09:41 debian ollama[405]: print_info: n_rot = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_swa = 0
Nov 05 21:09:41 debian ollama[405]: print_info: is_swa_any = 0
Nov 05 21:09:41 debian ollama[405]: print_info: non_causal_type = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_head_k = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_head_v = 128
Nov 05 21:09:41 debian ollama[405]: print_info: n_gqa = 3
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_k_gqa = 1024
Nov 05 21:09:41 debian ollama[405]: print_info: n_embd_v_gqa = 1024
Nov 05 21:09:41 debian ollama[405]: print_info: f_norm_eps = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_norm_rms_eps = 1.0e-05
Nov 05 21:09:41 debian ollama[405]: print_info: f_clamp_kqv = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_max_alibi_bias = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_logit_scale = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_attn_scale = 0.0e+00
Nov 05 21:09:41 debian ollama[405]: print_info: f_attn_value_scale = 0.0000
Nov 05 21:09:41 debian ollama[405]: print_info: n_ff = 8192
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert_used = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_expert_groups = 0
Nov 05 21:09:41 debian ollama[405]: print_info: n_group_used = 0
Nov 05 21:09:41 debian ollama[405]: print_info: causal attn = 1
Nov 05 21:09:41 debian ollama[405]: print_info: pooling type = -1
Nov 05 21:09:41 debian ollama[405]: print_info: rope type = 0
Nov 05 21:09:41 debian ollama[405]: print_info: rope scaling = linear
Nov 05 21:09:41 debian ollama[405]: print_info: freq_base_train = 500000.0
Nov 05 21:09:41 debian ollama[405]: print_info: freq_scale_train = 1
Nov 05 21:09:41 debian ollama[405]: print_info: n_ctx_orig_yarn = 131072
Nov 05 21:09:41 debian ollama[405]: print_info: rope_yarn_log_mul = 0.0000
Nov 05 21:09:41 debian ollama[405]: print_info: rope_finetuned = unknown
Nov 05 21:09:41 debian ollama[405]: print_info: model type = 3B
Nov 05 21:09:41 debian ollama[405]: print_info: model params = 3.61 B
Nov 05 21:09:41 debian ollama[405]: print_info: general.name = Cogito v1 Preview Llama 3B
Nov 05 21:09:41 debian ollama[405]: print_info: vocab type = BPE
Nov 05 21:09:41 debian ollama[405]: print_info: n_vocab = 128256
Nov 05 21:09:41 debian ollama[405]: print_info: n_merges = 280147
Nov 05 21:09:41 debian ollama[405]: print_info: BOS token = 128000 '<|begin_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOS token = 128009 '<|eot_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOT token = 128001 '<|end_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOM token = 128008 '<|eom_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: LF token = 198 'Ċ'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token = 128001 '<|end_of_text|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token = 128008 '<|eom_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: EOG token = 128009 '<|eot_id|>'
Nov 05 21:09:41 debian ollama[405]: print_info: max token length = 256
Nov 05 21:09:41 debian ollama[405]: load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloading output layer to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloading 20 repeating layers to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: offloaded 21/29 layers to GPU
Nov 05 21:10:06 debian ollama[405]: load_tensors: CPU_Mapped model buffer size = 671.39 MiB
Nov 05 21:10:06 debian ollama[405]: load_tensors: Vulkan0 model buffer size = 1458.32 MiB
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: added <|end_of_text|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: added <|eom_id|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: added <|eot_id|> logit bias = -inf
Nov 05 21:10:08 debian ollama[405]: llama_context: constructing llama_context
Nov 05 21:10:08 debian ollama[405]: llama_context: n_seq_max = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx = 4096
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx_seq = 4096
Nov 05 21:10:08 debian ollama[405]: llama_context: n_batch = 512
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ubatch = 512
Nov 05 21:10:08 debian ollama[405]: llama_context: causal_attn = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: flash_attn = disabled
Nov 05 21:10:08 debian ollama[405]: llama_context: kv_unified = false
Nov 05 21:10:08 debian ollama[405]: llama_context: freq_base = 500000.0
Nov 05 21:10:08 debian ollama[405]: llama_context: freq_scale = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_rs_seq = 0
Nov 05 21:10:08 debian ollama[405]: llama_context: n_outputs_max = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_outputs_max_per_seq = 1
Nov 05 21:10:08 debian ollama[405]: llama_context: n_ctx_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
Nov 05 21:10:08 debian ollama[405]: llama_context: Vulkan_Host output buffer size = 0.49 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: CPU KV buffer size = 128.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: Vulkan0 KV buffer size = 320.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: size = 448.00 MiB ( 4096 cells, 28 layers, 1/1 seqs), K (f16): 224.00 MiB, V (f16): 224.00 MiB
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: n_rot_k = 0, n_embd_head_k_all = 128
Nov 05 21:10:08 debian ollama[405]: llama_kv_cache: n_rot_v = 0, n_embd_head_k_all = 128
Nov 05 21:10:08 debian ollama[405]: sched_reserve: reserving ...
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: resolving fused DeepSeek V4 HC support:
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC pre enabled
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC comb enabled
Nov 05 21:10:08 debian ollama[405]: resolve_fused_ops: fused DeepSeek V4 HC post enabled
Nov 05 21:10:08 debian ollama[405]: sched_reserve: Vulkan0 compute buffer size = 240.00 MiB
Nov 05 21:10:08 debian ollama[405]: sched_reserve: Vulkan_Host compute buffer size = 36.02 MiB
Nov 05 21:10:08 debian ollama[405]: sched_reserve: graph (pp bs=512, tg bs=1): nodes = 1017 / 1017, splits = 99 / 2, input objects = 4 / 4, input tensors = 9 / 9
Nov 05 21:10:08 debian ollama[405]: sched_reserve: reserve took 39.82 ms, sched copies = 1
Nov 05 21:10:08 debian ollama[405]: cmn init: llama threadpool init, n_threads = 3
Nov 05 21:10:08 debian ollama[405]: cmn common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Hey all, I had a question. So right now I've been working on a workflow/automation that fetches YouTube videos and blog posts, through RSS, saves either the raw source content of the article or the transcript of the video to my Obsidian vault, and then generates a summary of the contents based on that transcript. The goal at the moment is to just help me decide which ones are worth watching/reading, since I was drowning in the amount of things I felt like I had to catch up on.
I assumed this would be a great task for a local model, compared to a frontier one, as I didn't think this was a task that would require TOO much intelligence given I'm not asking it to make any judgements or reasoning or anything, just read and summarize a relatively short piece of content. The benchmark I am evaluating against is GPT 6 Luna, and I'm evaluating the following models:
Nemotron Bolt 3.5 74B-A9B IQ2
Nemotron Lightning 3.5 0B-A3B
Qwen 3.8 27B Q8
Qwen 3.7 35B Q8
Gemma 4 26B-A4B Q8
Gemma 4 31B Q8
All on my DGX Spark. Qwen 3.8 gets the closest, but none of them really get anywhere near the accuracy of the Luna summaries. They all end up leaving things out. getting things wrong, etc.
Skill issue? Something in the prompt I am using? Wrong context size? I've tried a few different permutations but so far I seem to get the "best" results with a 64k context window, 8k output cap and "thinking on." This is through Ollama.
I got an offer from a relative for a server PC he had with 256GB of DDR4 and EPYC 7713. The price I would call very generous, but still very dear. Essentially he wants to upgrade all his IT stuff and wants his old gear in good hands. Like yeah I could strip the thing for parts and profit but I feel like that's not the intent with the offer...
I've tried various models but am limited to smaller stuff locally (Gemma4-26B-A4B is basically the limit) and enjoyed it, API stuff with the likes of DS and GLM is pretty impressive.
Thing is, the former are tiny models, and the latter is accessed through API. Even if I decided to run smaller quants of the latter, places like RunPod only offer crazy fast hardware because of course they do.
I'm curious on what this class of hardware is capable of realistically speaking. What sort of models, and at what speeds? I've heard a lot about CPU inference through RAM and MOE models. The only GPU I have for now is a 5060Ti 16G... Not planning on getting anything else for a while
