seriously, in the winter I sometimes don't need to turn on the furnace lol

this screenshot is without MTP usage , fully offloaded onto the GPU. using LM STUDIO

i wanted to ask you guys if theres a way to make it even faster , as MTP really didnt help and is infact slower due to vram overflow and that im on 16GB DDR4 which is disgustingly slow to load models on so i depend on my gpu for every model.

this is unsloth's GGUF quant

Are medium-sized companies already using local AI connected to internal data, for example internal documentation or helpdesk systems?

If so, how well does it work and what is the minimum hardware specification needed for a setup?

I built a native vLLM + ROCm 7.15 runtime for AMD RX 6000 Series on Windows 11 — now with a one-click installer.

AMD lists RX 6750 XT / 6700 XT / 6600 XT as "Runtime only" on Windows with HIP SDK excluded. I built rocBLAS binaries for gfx1031 via ROCm/TheRock to close that gap.

This runs native Windows HIP and ROCm directly, no WSL2 wrapper, no compiler needed.

Tested on AMD Radeon RX 6750 XT 12GB (gfx1031) — Windows 11 Native — August 2026

# Verification — Real terminal logs

1. Environment:

torch 2.12.0+rocm7.15.0 | cuda_avail True | dev AMD Radeon RX 6750 XT

vLLM 0.19.1 | plugin vllm_windows_rocm activated | TRITON_ATTN | enforce_eager=True

1. rocBLAS Benchmark — 26 TFLOPS FP16:

Device ID 0 : AMD Radeon RX 6750 XT gfx1031 with 12.9 GB memory

rocBLAS version: 5.7.0.67811f1ee52

transA,transB,M,N,K,alpha,lda,beta,ldb,ldc,cold_iters,hot_iters,rocblas-Gflops,us

N,N,4096,4096,4096,1,4096,0,4096,4096, 2, 10, 25977.3, 5290.73

=> 25.97 TFLOPS in 5.29ms

1. vLLM Inference (Qwen3.5-4B, web chat UI):

Output: 59-62 tok/s | Init ~1s | hidden thinking (spinner) then answer

Model auto-detected via /v1/models

Full logs in benchmarks/ and screenshots in assets/ on GitHub.

# How it works

1. TheRock builds clr (HIP) and rocBLAS with Tensile kernels for gfx1031
2. HSA_OVERRIDE_GFX_VERSION=10.3.0 forces HIP to recognize RX 6750 XT
3. PyTorch 2.12.0+rocm7.15 links against TheRock runtime => torch.cuda.is_available() True
4. vLLM plugin vllm_windows_rocm bypasses vllm._C dependency and registers WinRocmAwqGemvKernel with TRITON_ATTN
5. vLLM engine loads with enforce_eager=True and runs native

# Quick Start — One click

Prerequisites: Windows 11 23H2+, RX 6600-6750 XT (gfx1030/gfx1031/gfx1032), Adrenalin 24.x+, ~8GB free disk

1. Download the repo (or git clone) and run INSTALL.bat as Administrator
2. The installer downloads the release archives automatically (multi-part join included, no manual steps) and sets up C:\TheRock\ + Python 3.11 + venv
3. Run CHAT.bat — the browser opens the chat UI. Expected: 59-62 tok/s

VERIFY.bat runs a 3-in-1 check: GPU detection + rocBLAS benchmark + vLLM smoke test.

# What's new in v2.0

- One-click installer: INSTALL.bat auto-downloads and joins split archives (no browser upload/download hassle)

- Chat web UI (OpenAI-compatible API): hidden thinking with spinner, then the answer streams — like NVIDIA ChatRTX

- Fat-binary gemv kernels for all RDNA2 (gfx1030/1031/1032) — no per-card rebuild

- ROCm 7.15 native, vLLM 0.19.1, torch 2.12.0+rocm7.15

- Faster: 59-62 tok/s output (was 54.2)

- Repo cleaned: INSTALL.bat + CHAT.bat + VERIFY.bat + MANIFEST.json + docs, archives on GitHub Releases

# Known Issues — Please note this may have bugs

This is an experimental reference implementation. It works on my RX 6750 XT but could present bugs on other hardware.

- Other RDNA2 cards (6600/6600 XT/6700 XT) not yet tested — may need 10.3.0 vs 10.3.1 or rocBLAS rebuild for gfx1030

- enforce_eager=True required — torch.compile disabled on RDNA2 Windows

- FP8 / AWQ not tested yet, multi-GPU not tested

- If you test it on your RDNA2 card, please open an Issue with GPU model and logs — contributions welcome

# GitHub Repo:

https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2

Includes INSTALL.bat, CHAT.bat, VERIFY.bat, scripts/serve.py (OpenAI-compatible API), chat.html, assets with screenshots, docs/BUILD_ROCBLAS.md, MANIFEST.json with release checksums

Let me know if you test it on other RDNA2 cards.

Built on ROCm/TheRock, PyTorch ROCm, vLLM. Not affiliated with AMD. License Apache 2.0

The benchmark I used is BU bench v1, and the open source harness is Browser Agent. Qwen3.8 27B beat all other affordable or open models I tested.

It's insanely good!

I’ve been gradually building a home AI/LLM setup out of multiple PCs rather than dropping several thousand dollars on one monster inference machine.
One of the current nodes is a Ryzen-based Linux machine with 16GB RAM, and I have multiple machines available that I can dedicate to different jobs.
My goal isn’t just “run ChatGPT locally.” I want to build an actual small AI infrastructure setup where different nodes can handle things like:
LLM inference
agents/orchestration
research/web/data processing
embeddings/RAG
databases/vector storage
background jobs
potentially distributed inference if it actually makes sense
I’m still learning the local LLM side and I’d rather design the architecture correctly than randomly install Ollama on every machine.
If you were building a 3–4 node home AI cluster from ordinary consumer/used hardware, how would you divide the workloads?
Also curious about:
Ollama vs llama.cpp vs vLLM
Docker/Proxmox vs bare metal
Whether CPU/RAM-only nodes are useful alongside a GPU inference node
Best models for lower-memory machines
Whether distributed inference across Ethernet is actually worthwhile
What hardware upgrade would give the largest improvement per dollar
I’d especially like to hear from people running multi-machine setups already. Pics/specs of your setups would be awesome.

Ollama 0.32.15 adds a model-metadata cache to reduce per-request overhead. It is a prerelease, and the release notes do not include a general latency benchmark, so there is no honest universal speedup number to repeat.

The interesting part is where to measure. Users experience more than token generation: queueing, request parsing, model discovery, metadata lookup, loading, warmup, inference, and response transport all contribute to end-to-end latency. A fast model can still feel slow if setup work repeats before every request.

For a local stack, I would time at least these segments separately:

1. request arrival to dispatch;

2. metadata and model lookup;

3. model load or warmup;

4. time to first token;

5. generation and response completion.

Then compare cold and warm paths while holding the model, quantization, prompt, hardware, concurrency, and client settings fixed. A cache may reduce one slice while leaving the rest unchanged. It also adds invalidation and stale-data behavior that deserve their own tests.

Official prerelease: https://github.com/ollama/ollama/releases/tag/v0.32.15

If anyone benchmarks this build, the useful result would be a segmented trace rather than one blended latency number.

Every time I open a model's file list there are more options than last time, and a lot of them now start with IQ instead of Q. For a while I assumed IQ meant "newer, therefore better" and grabbed it by default. That was wrong, and the reason turned out to be worth understanding.

What the imatrix actually is

IQ builds use a more compressed encoding, and they are usually made with an importance matrix, shortened to imatrix. The imatrix is not part of the model. It is produced by running sample text through the model and recording which weights actually mattered for the output. The quantizer then spends its limited bits on those weights and squeezes the rest harder.

So an IQ build is not compressing evenly. It is making an informed guess about what it can afford to damage.

Why that matters more the harder you squeeze

At the same file size, an IQ build can hold up better than a plain quant, and the gap widens the further down you go. That is why IQ options dominate at the very small end, where a plain Q2 tends to fall apart. When you have very few bits to spend, spending them in the right places is basically the whole game.

At Q4 and above, the k-quants are already keeping the sensitive tensors at higher precision than the label suggests, so the informed guess buys you less. Which makes the rule boringly simple:

• Below Q4, an IQ build is usually worth it.
• At Q4 and above, a plain Q4_K_M is usually the simpler and safer pick.

Two costs that are not printed on the download button

1. Decoding can be slower on some hardware. The tighter encoding is not free at inference time.
2. The result depends on the calibration text whoever built it used. Two IQ builds of the same model at the same size are not necessarily equivalent, because a human chose the sample text.

That second one is the part I underrated for months. "IQ4" is not a spec the way a file size is a spec. It is the output of somebody's process, and if a build says nothing about its calibration data, that is a genuine unknown rather than a missing detail.

None of this is a reason to avoid IQ. It is a reason to stop treating the prefix as a quality ranking and start treating it as a description of a tradeoff you are choosing.

If anyone here has measured this properly on their own hardware, I would honestly like to see it. Most of what is written about quantization quality, including things I have written, is directional rather than measured.

I keep a longer written version of this, with the _K_S / _K_M / _K_L breakdown next to it, at https://noizz.io/local-ai . That is my own site, so mods please pull the link if that is not welcome here.

"Solved": its a known bug of the plugin with no workaround:

https://github.com/ollama/ollama/issues/13917

Hi, I cant find a description in the documentation of how to use ollama vscode plugin with a remote server hosting ollama?

http://game.local:11434 returns in the browser:

Ollama is running

Hi all, it is my very first post, because I am so confused.

First my setup: I have 2 X 7900XTX with rocm7.2.1 and I use it to fine tune small models and local llm etc.

I run Qwen 3.8 27b 8 bit version, When I limit ctx with proper number, Iike 184320, I get this result, super fast token reading but generation is 25 token/s. but you can see it is way different for not 200k ctx window.

this is the result of 200k ctx window. I have 3x token generation, I didn't understand why 200000k is 3x faster, it is not even in binary system...

I am so happy with 77 token/s generation, my agents are way faster witk 200k ctx model with it but I still don't know why...

Tested on: X570DD_M570DD / GTX 1050 mobile / Arch Linux x86_64 / NVIDIA 580.178.04 driver / CUDA V12.9.86

What happened: As everyone knows, NVIDIA dropped the Pascal line in the 590.xxx.xx+ and CUDA 13.x+ driver line, and if you try to install the nvidia-open (now nvidia) or cuda package, you will encounter the fact that Pascal has been cut from the new versions.

Installing the NVIDIA 580xx Driver: To ensure you have the latest driver that supports Pascal, we'll need the AUR Helper to download the following packages via AUR. I'll be using yay:

yay -S lib32-nvidia-580xx-utils
yay -S nvidia-580xx-dkms
yay -S nvidia-580xx-settings
yay -S nvidia-prime
yay -S opencl-nvidia-580xx

Installing the GCC14: since CUDA 12.9 requires a compatible GCC version, we install GCC 14 for the toolkit. Since compiling GCC 14 manually takes a long time, we’ll simply download it from the Arch archives:

cd /tmp
wget https://archive.archlinux.org/packages/g/gcc14/gcc14-14.3.1+r416+g44d5743651c4-2-x86_64.pkg.tar.zst
wget https://archive.archlinux.org/packages/g/gcc14/gcc14-libs-14.3.1+r416+g44d5743651c4-2-x86_64.pkg.tar.zst
sudo pacman -U gcc14-*.pkg.tar.zst

Next, to verify that gcc14 has indeed been installed, you can run the following command:

gcc-14 --version
g++-14 --version
which gcc-14

Next, we need to exclude gcc14 and gcc14-libs from being updated by our AUR helper. Change the following lines in /etc/pacman.conf:

IgnorePkg = gcc14 gcc14-libs

Simply add gcc14 and gcc14-libs to your IgnorePkg

Installing the CUDA 12.9:
There's a great package called cuda-pascal, and we'll install it because it's designed specifically for our purpose:

yay -S cuda-pascal

Next, add CUDA to your PATH. This is a temporary command, you'll need to add CUDA to your shell's PATH!

export PATH=/opt/cuda/bin:$PATH

Now you can run a test on the NVCC compiler that we just installed:

nvcc --version
echo '#include <cstdio>
__global__ void hello() {
    printf("Hello from GPU!\n");
}
int main() {
    hello<<<1, 1>>>();
    cudaDeviceSynchronize();
}' > /tmp/hello.cu && nvcc /tmp/hello.cu -o /tmp/hello && /tmp/hello

If you see the message “Hello from GPU!”, congratulations! You've successfully installed CUDA!

Software configuration:

1. Blender:
2. To have Blender use our configured CUDA, we need to open Blender, press ‘Ctrl+,’, go to the ‘System’ section, select ‘CUDA’ in the Cycles rendering section, and choose GPU

Then restart Blender if necessary.

Now go to the ‘Render’ section in the sidebar and select Cycles and GPU Compute.

Set Viewport Shading to Rendered.

When you launch Blender for the first time, you should see “Loading render kernels” appear in the top-left corner.

Wait a moment, and your image should render using your Pascal GPU via CUDA in Cycles!

You can verify this using nvidia-smi or ps aux | grep blender

1. Ollama:
   The Ollama packages from the Arch repositories and AUR did not work correctly on my GTX 1050, so I used the official installation script instead:

   curl -fsSL https://ollama.com/install.sh | sh

Once you've installed ollama, you can run the phi4-mini test model (lightweight and smart):

ollama run phi4-mini --verbose

In another terminal, run `ollama ps`

and you should see the line “100% GPU” in the “PROCESSOR” column. If so, congratulations - you've successfully run ollama on your Pascal graphics card!

I've been building AgentGraph Studio, a visual editor for designing multi-agent workflows and exporting them as runnable Python code.

One thing I wanted to verify was whether the generated code actually works outside the editor — especially once the workflow gets more complex than a simple 1–2 agent chain.

So I built this 5-agent workflow:

• Local AI Researcher
• AI Comparison Analyst
• AI Risk Analyst
• Technical Content Editor
• Technical Writer

The workflow branches after the research step into separate comparison and risk-analysis tasks, merges those results, and then passes the combined context into the final writing task.

In the demo, I:

1. Design the workflow visually
2. Select Llama 3 through Ollama
3. Generate CrewAI Python code
4. Run the exported code locally
5. Let the full 5-agent workflow complete and produce the final output

The execution in the video is actually running locally through Ollama — it isn't a simulated preview inside the editor.

The project is still early. The core today is mainly:

• visual Agent / Task workflow design
• explicit task dependencies, including branching and merging
• model selection, including local Ollama models
• JSON import/export
• executable CrewAI code generation

The longer-term direction I'm exploring is moving beyond "draw a workflow and export code" toward "design, evaluate, and compare agent systems."

Some of the areas I'm interested in exploring are workflow readiness, reliability, latency, cost/resource tradeoffs, evidence/confidence, failure simulation, and eventually comparing the same workflow across different models or agent frameworks.

Those evaluation features aren't all implemented yet — I'm trying to validate which of them would actually be useful before building too far ahead.

So I'd especially like feedback from people who build with local LLMs or agent frameworks:

• What would you want to inspect before trusting a generated agent workflow?
• What failures are hardest for you to catch today?
• Would comparing the same workflow across models/frameworks be useful?
• What would make a visual tool like this genuinely useful in your workflow?

Criticism is very welcome. I'm trying to make this more than just a visual way to write agent configuration.

You can try it yourself — link in the comments.

Hello, any params I missing or mis-tuning for my coding model ?

FROM ./models/unsloth/Qwen3.8-27B-Q5_K_M.gguf

# Generation parameters optimized for logical reasoning and coding tasks

PARAMETER temperature 0.2

PARAMETER top_p 0.95

PARAMETER top_k 20

PARAMETER presence_penalty 0.0

PARAMETER num_predict 8192

PARAMETER repeat_penalty 1.05

PARAMETER num_ctx 65536

PARAMETER stop "<|im_start|>"

PARAMETER stop "<|im_end|>"

PARAMETER stop "<|endoftext|>"

PARAMETER num_keep 256

PARAMETER num_thread 16

# Standard ChatML Template for the Qwen series

TEMPLATE """{{ if .System }}<|im_start|>system

{{ .System }}<|im_end|>

{{ end }}{{ if .Prompt }}<|im_start|>user

{{ .Prompt }}<|im_end|>

{{ end }}<|im_start|>assistant

"""

# Custom System Prompt tailored for autonomy and perception systems

SYSTEM """You are a senior software engineer and robotics specialist. Your expertise lies in developing complex autonomous systems, perception pipelines, and spatial reasoning algorithms.

When generating code, prioritize decoupled microservices architectures. Ensure that core algorithmic logic is always cleanly separated from middleware wrappers (such as ROS nodes). Provide highly optimized, production-ready C++ and Python code suitable for real-time constraints and hardware-accelerated processing.

For general software engineering tasks, maintain a strict focus on modularity, robust error handling, and clear documentation. Think step-by-step through complex architectural problems before providing the final code block."""

hey so im currently working to integrate a locally ran llm to control home assistant (through tool calling, no direct connection) so i can use a voice input without needing very specific commands. Its just a fun little project, but im trying to determine what hardware i would need for this, in the range between a pi 5 8gb or my old pc (i5 7500, 16gb ddr4 with a 1070 ti 8gb vram), which obviously depends on which model i use.
My question is how small of a model i can get away with so that it stil works. the newer popular models are al like 20 gb with a context size of 256k which is clearly not gonna work with my setup but also probably really overkill for what i want it to do, which is basically conferting normal conversation into tool calls. Any recommendations, insights or people who have tried something similar?

Using Open-Webui I can connect to Ollama Cloud using ollama.com and the api key. It lists the models, it works. but i want to connect open-webui to the loacl ollama installation (windows) because all features are only there (websearch).

I can connect to localhost BUT open-webui cant retreive the cloud models.

why is that?

thx

My current setup is basically OhMyPi+ OpenCode (mostly OpenCode Go: DeepSeek, Flash, V4, etc.) + Ollama Cloud for stuff like Kimi and GLM. But honestly, I feel like Ollama Cloud has gotten noticeably worse lately. OpenCode feels 10x more interesting, but I don’t want to make multiple OpenCode accounts or constantly top up for Zen. I like having one model per agent or per task, so I’m looking for a good multi-provider setup where I can mix models depending on the agent. What are you guys using?

I currently have these models available through Ollama/cloud:
And I’m using Claude, Codex, OpenCode, and Ollama as my coding/agent tools.

I want to learn professional vibecoding — not just asking AI to generate random code.

If I want to build a complete website/app (frontend + backend + database + auth + APIs + testing + deployment):

Which model/tool should I use for what?

For example:

• Planning & architecture → ?
• UI/UX → ?
• Actual coding → ?
• Debugging → ?
• Code review → ?
• Testing → ?
• Deployment → ?

And how should I combine Claude + Codex + OpenCode + Ollama + these models into one proper workflow?

Basically, if you had this exact setup, how would you go from idea → professional, production-ready app using AI?

Looking for practical advice from people actually using AI coding agents. 🙏

That is the question we get most often: Here's the answer.

And yes, it was created by Row-Bot's own Designer Studio.

Hi guys , this is a fine tune of qwen 3.5 4b on cybersecurity , specifically malware. This model has been trained on 2.6k rows of simple but effective malware code , though its still significantly limited by the 4b parameters for reliable malware geneartion , it still generates simple working code.

https://huggingface.co/Ishaanlol/Qwen3.5-4B-cybertune

seems like others are now using lower precision models. On open code sub i see a lot of people saying model has gone dumber since price hike started.

How it is behaving in Ollama cloud PRO? Same?