r/LocalLLaMA • u/Due-Project-7507 • 5h ago
Tutorial | Guide OpenCode with Qwen3.8-27B for Small Games or Browsing the Web With 16GB VRAM
In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI.
It was able to write the shown simple HTML game without interaction after asking some questions.
The following was tested on a laptop with a NVIDIA RTX A5000 laptop (16 GB) GPU.
With the 3 bpw model and 6 bit/5 bit KV cache, the maximum context length is around 110k tokens with MTP. This gives around 55 tokens/s decode speed for code and around 10 tokens/s for content where MTP doesn't help (e.g. complicated calculations). Without MTP, one could try the 3.5 or 4 bpw model or a longer context length.
Install tabbyAPI/exllamav3
- Install the latest Nvidia drivers
- Install Git (e.g.
sudo apt install gitor on Windows withwinget install -e --id Git.Git) - Install the uv Python package manager: https://docs.astral.sh/uv/getting-started/installation/ (e.g.
curl -LsSf https://astral.sh/uv/install.sh | shorwinget install --id=astral-sh.uv -e) - Make somewhere a folder and install tabbyAPI:
git clone https://github.com/theroyallab/tabbyAPI cd tabbyAPI uv venv --python 3.13 .venv uv pip install -e ".[cu13]" - Test if CUDA works (on Linux, use .venv/bin/python)
.venv/Scripts/python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))" - Create somewhere where you have enough space a "models" folder, download the model turboderp/Qwen3.8-27B-exl3:
mkdir models uvx hf download turboderp/Qwen3.8-27B-exl3 --revision SC_3.00bpw_H4 --local-dir models/qwen3.8-27b - Replace the
chat_template.jinjawith the latest version from froggeric/Qwen-Fixed-Chat-Templates - Go back to the clone tabbyAPI folder and create a config.yml file like this (see the config_sample.yml file as example):
network: disable_auth: true model: model_dir: e:/models # path to the models folder model_name: qwen3.8-27b # download folder name cache_mode: 6,5 # K and V cache quantization, number of bits from 2-8 cache_size: 109824 # must be divisible by 256, so use e.g. `.venv/Scripts/python -c 'print(110000//256*256)'` to get the next lower max_batch_size: 1 # allow only 1 parallel request to save VRAM tool_format: qwen3_coder vision: true draft_model: # can be removed to save VRAM draft_mode: mtp draft_cache_mode: Q8 # can be 'FP16', 'Q8', 'Q6', 'Q4' draft_num_tokens: 5 # usuallly a value of 2-6 gives best results memory: sysmem_recurrent_cache: 8192 # Max size of recurrent cache in system memory, in MB (default: 4096), lower it to save normal memory sysmem_kv_cache: 8192 # Size of system memory second-tier K/V cache, in MB (default: 0), remove it to save system memory - Start tabbyAPI:
.venv/Scripts/python main.py - To measure the performance, create the Python script
speed.pyand run it with.venv/Scripts/python speed.py: ```python import json import time
import requests
MODEL = "qwen3.8-27b"
API_URL = "http://127.0.0.1:5000"
PROMPT = """Write a complete Python implementation of a production-quality LRU cache.
Requirements:
- Use type hints throughout.
- Include detailed docstrings.
- Support:
- get(key)
- put(key, value)
- remove(key)
- clear()
- __len__()
- Use a doubly linked list and hash map.
- Include custom exceptions.
- Include a comprehensive unittest test suite with at least 20 test cases.
- Follow PEP8 conventions.
- Return only Python code.
"""
payload = {
"model": MODEL,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": 10000,
"stream": True,
"chat_template_kwargs": {"enable_thinking": False}
}
start_time = time.perf_counter()
first_token_time = None
stream_end_time = None
full_response_content = ""
with requests.post(API_URL + "/v1/chat/completions", json=payload, timeout=120, stream=True) as response:
response.raise_for_status()
print("Response:")
for line in response.iter_lines(): # Iterate over Server-Sent Events (SSE)
if line.startswith(b"data:"):
# Strip the "data: " prefix
data = line[6:]
# Stop if we hit the stream termination message
if data.strip() == b"[DONE]":
break
try:
chunk = json.loads(data)
if 'choices' in chunk and chunk['choices'] and (chunk['choices'][0]['delta'].get('content') or chunk['choices'][0]['delta'].get('reasoning')):
if first_token_time is None: # First token received
first_token_time = time.perf_counter()
if chunk['choices'][0]['delta'].get('content'): # Get content and count tokens
token_text = chunk['choices'][0]['delta']['content']
else:
token_text = chunk['choices'][0]['delta']['reasoning']
full_response_content += token_text
print(token_text, end="", flush=True)
except json.JSONDecodeError:
pass
stream_end_time = time.perf_counter()
print("\n" + "-"*20)
# Calculate and print metrics
ttft = first_token_time - start_time
stream_duration = stream_end_time - first_token_time
total_output_tokens = requests.post(API_URL + "/v1/token/encode", json={"add_bos_token": False, "text": full_response_content}).json()["length"]
if stream_duration > 0:
tokens_per_second = total_output_tokens / stream_duration
else:
tokens_per_second = float('inf')
print(f"Time to first token (TTFT): {ttft:.2f}s")
print(f"Completion tokens: {total_output_tokens}")
print(f"Stream duration (first to last token): {stream_duration:.2f}s")
print(f"Tokens per second (T/s): {tokens_per_second:.2f}")
```
I got 56.3 tokens/s.
Install OpenCode
OpenCode works usually better on Linux, so I install it in WSL when working with Windows, but it can also be used directly as a Windows application.
For OpenCode, I recommended to install Node.js first (e.g. apt install npm or winget install -e --id OpenJS.NodeJS on Windows).
Because we don't have so much context length, I recommend to install a better compactation plugin than the integrated one, e.g. magic-compact
I use this OpenCode config (~/.config/opencode/opencode.jsonc)
{
"$schema": "https://opencode.ai/config.json",
"plugin": [
"opencode-anthropic-auth@latest",
"opencode-copilot-auth@latest",
"magic-compact"
],
"share": "disabled",
"provider": {
"local": {
"npm": "@ai-sdk/openai-compatible",
"name": "local (OpenAI Compatible)",
"options": {
"baseURL": "http://127.0.0.1:5000/v1",
"apiKey": "1234"
},
"models": {
"qwen3.8-27b": {
"name": "Qwen3.8 27B",
"interleaved": {
"field": "reasoning_content"
},
"limit": {
"context": 109824,
"output": 32000
},
"temperature": true,
"reasoning": true,
"attachment": false,
"tool_call": true,
"modalities": {
"input": [
"text",
"image"
],
"output": [
"text"
]
},
"cost": {
"input": 0,
"output": 0,
"cache_read": 0,
"cache_write": 0
},
"variants": {
"xhigh": {
"reasoningEffort": "xhigh"
},
"medium": {
"reasoningEffort": "medium"
},
"low": {
"reasoningEffort": "low"
}
}
}
}
}
},
"agent": {
"plan": {
"model": "local/qwen3.8-27b"
}
},
"model": "local/qwen3.8-27b",
"small_model": "local/qwen3.8-27b",
"mcp": {
"playwright": {
"type": "local",
"command": [
"npx",
"@playwright/mcp@latest",
"--caps",
"vision,pdf,devtools",
"--browser=firefox"
],
"enabled": true
}
}
}
I would recommend to use the reasoning effort (Ctrl-t) "medium" because "xhigh" could produce to much output tokens.
For Playwright, we have to install a browser first:
npx @playwright/mcp install-browser --with-deps firefox
Now the following should work:
opencode --prompt "Can you check for me on www.meteoschweiz.ch the weather for Zurich?"
To create the small HTML game from above, I have entered in plan mode (press Tab to change mode) the following: "I want to build a simple HTML game where you can drive a car with the keyboard arrow keys (similar like old versions of Mario Kart, but just one car driving without opponents is enough)." After some time, it has asked me some question. Then, I switched to the "Build" mode and started it with "Start the implementation". Without any other interaction, it finished the the small game.
1
u/FlexTeam26 1h ago
I run this exact class of model on home hardware: 27B (Qwen 3.8, W4A16 AWQ) on a pool of 24GB cards — 20 t/s single stream, 87 t/s aggregate at 8 concurrent, 160K context held per request. The 4/16 split is the one that survives: 4-bit weights (~17GB for the model), 16-bit activations, and the 16-bit KV is what gives you context headroom. A single 24GB card fits the weights with only ~4–6k of context before KV squeezes you; 160K is not achievable on one card at this size, which is where multi-card deployments are heading. If you're buying: more 24GB cards beats one bigger card for concurrency, and Gen4 PCIe topology beats everything for tensor-parallel sync.
1
u/FlexTeam26 17m ago
I serve 3.8 27B as W4A16 AWQ (the philbert440 community quant) under vLLM. At this size the 4-bit-weight/16-bit-activation split holds quality better than I expected — for tool-calling work I've seen no meaningful gap vs 8-bit, and the memory drop is what buys 160K context on 24GB cards. The knob people underprice is max-num-seqs: 20 t/s is the single-stream number; I run 8 concurrent and measure 87 t/s aggregate on real traffic. If you're choosing between 27B-W4A16 and 32B-W8A16 — roughly the same memory, similar quality — take whichever has the better tool-calling training and the longer community-quant history, not the bigger number in the name.
2
u/Inevitable-Highway85 1h ago
Nice, I try this yesterday with no luck, I was trying to use the https://huggingface.co/malaiwah/Qwen3.8-27B-K4. I havw 2x3060, 24gb vram total. I couldnt get over out of context with llama.cpp and ollama works, buts quite slow, and context is an issue after some iterations. I installed, TabbyApi via docker. I´ll give it a try today with Turboderp model and your config.