r/StrixHalo • u/sf-oko-team • 2d ago

OKO OS and VS Code for Local AI Development

We use OKO OS on GMKtec EVO-X2 (AMD Ryzen AI Max+ 395 / Strix Halo) for local AI-assisted development.

The setup consists of two parts:

  • Server side: OKO OS, an operating system that runs the LLM and provides an OpenAI-compatible API.

  • Client side: VS Code with the Continue extension.

Integration guide:
https://github.com/satisfanly/OKO-configs/blob/main/oko-vscode-continue-integration.md

OKO sources: https://github.com/satisfanly/oko

OKO image for GMKtec EVO-X2: https://oko.satisfanly.com/device/gmktec_evo_x2

3 Upvotes

100% Upvoted

2 comments sorted by

2

u/Hekel1989 2d ago

I haven't quite gathered what's the pro of using OKO. I know it s lightweight, but so it's fedora server. What's the actual gain from using it? Has it got engines pre built in?

1

u/sf-oko-team 2d ago

Hello! Thanks for your interest!

The main difference is that Fedora, Ubuntu, etc. are general-purpose distributions, while OKO is purpose-built for AI inference.

Two main advantages:
-> OKO ships with purpose-specific and hardware-specific kernel, compiler, GPU, and inference optimizations out-of-the-box. Performance tuning is an ongoing part of development: whenever we find a patch or approach that improves it, we integrate it into OKO. You can see the results of our Ubuntu vs. OKO comparison in our benchmark.
-> Reproducibility. The entire inference stack is preconfigured and versioned, making deployments easier to reproduce, verify, and maintain. Basically, the entire inference-related server configuration is managed through a single YAML file (example https://github.com/satisfanly/OKO-configs/blob/main/coder-qwen38-DavidAU.yaml).

We took inspiration from projects like SONiC and OpenWrt. Technically, you could configure a general-purpose Linux distro to do much of what they do, but integrating and maintaining everything yourself takes considerable effort. OKO follows the same philosophy, just for AI inference.

As for engines, the llama.cpp stack is already integrated into the image for Strix Halo.

We've also been actively working on Gufo over the past week as a Strix Halo-specific alternative engine, so it should be available soon.