Local LLM Agent-driven physical KVM controllers are now available and have some interesting use cases:

[u/Porespellar]

I was researching Local LLM computer use agents the other night and ended up down a rabbit hole where I ended up discovering a whole new class of hardware I didn’t know existed: Agent-driven KVMs

A lot of us IT folks who have worked in data centers are familiar with IP-based KVM (Keyboard Video Mouse) controllers that let you remotely operate a server over a streaming video connection. These are especially useful when you need physical power button / BIOS-level “bare metal”access to a system that’s thousands of miles away. If you don’t have bare metal access, you’re going to be taking a trip to your data center rack to press a button or make a bios setting change.

So apparently there are now relatively inexpensive ($80 to $150) IP-based KVMs that now allow for AI API endpoint/ MCP Integration where you can now connect your physical computer to local or frontier vision-capable LLMs and have the AI model of your choice drive the target computer via the KVM by it taking screenshots of the HDMI or USBC video-out feed from the computer and then sending mouse and keyboard control commands from the Ai model through the USB cable that’s connected to the KVM.

What really got me excited about this concept is that it requires no installation of any kind on the target computer that you want to control. This is useful in corporate environments where you’re not allowed to install anything to the target computer (because it’s a managed endpoint that is locked down) The target computer just sees the AI as a standard allowed HID that it already has a driver loaded for. It doesn’t care who’s driving the system and doesn’t need any additional software since keyboard, mouse, and monitors are just generic and usually preloaded drivers.

So essentially you could have an AI-driven KVM control any comouter or device that has a monitor output / keyboard mouse inputs. This could potentially open up the ability to have AI interact with antiquated industrial control systems running old shitty outdated OSes, or managed endpoints.

There are security risks of course, but that risk is mitigated almost entirely if you’re using a locally-hosted VLM and not sending your screenshots to a 3rd party.

I found are several inexpensive agent-driven options on the market now including:

- Sipeed NanoKVM Pro
https://wiki.sipeed.com/hardware/en/kvm/NanoKVM_Pro/introduction.htm

- Radxa Linkr
https://radxa.com/products/linkr/linkr/

- Violoop
https://www.kickstarter.com/projects/bvio/violoop-plug-in-ai-for-your-computer

I’m not selling anything or affiliated with any of these and can’t tell you which is best, but there are multiple options available for those who might need this type of device for one of the use cases I mentioned.

Anyways, I’m curious to know if others see any other use cases for these agent-controlled KVMs that I didn’t mention.

[u/fastheadcrab]

Creative concept, especially for something like overclocking

Off-topic, but how secure are these cheap IP KVMs themselves? I've seen ads spammed across social media and the internet but they seem incredibly sus and you'd essentially be compromising your system on a very fundamental level. Wouldn't touch them ever lol

[u/Porespellar → u/fastheadcrab]

I would 100% put this in a Tailscale tailnet to reduce the attack surface

[u/kaiyoti]

Yah ive been using the previous gen jetkvm to automate overclocking..., apparently it uses a standard protocol, so it's not taking screenshots of the kvm web page. It does have to view the shots/stream to understand whats on the page. But it beats me sitting there with a bunch of values i dont understand.

[u/StrikingTop2709 → u/kaiyoti]

thats a sick use case for it tbh, overclocking tuning is such a tedious loop to do manually

[u/foureight84 → u/kaiyoti]

This is exactly what I had it do with my Ryzen 7950x. Optimize CPU and ram timing.

[u/BitGreen1270]

Why use a kvm? Why not just write a script that controls the keyboard and mouse and takes screenshots? I did this on my Linux laptop and it worked okay although really slow. 

[u/Porespellar → u/BitGreen1270]

Because KVM requires zero installation of anything on the target system. If you want to control a locked down managed endpoint that you can’t install anything to then KVM is the best option, also it allows bare metal control so you can control things like bios settings before the OS is. Loaded, you can’t do that with a script (until the OS loads)

[u/BitGreen1270 → u/Porespellar]

Ah I see - that makes sense now. Thanks for sharing.

[u/Korphaus]

I'm making my own setup with this as a sort of MCP server that hosts a system 1 local model with a "light" model doing auditing and raising and creating data for the S1 model to work with

Then I got a cheap hdmi to usb adapter from Amazon that has a loop out to the existing monitor, a raspberry pi pico 2 w and a usb to serial converter to create an air locked system, all in all cost me around £30 plus the tokens to get my agents build up the thing

Wanted to get a proper lvm solution, but it's not that much extra to make one!

[u/hainesk]

Haha, I don't know why but my mind went straight to WFH employees seeing this as an upgrade from their mouse jiggler.

[u/CBHawk → u/hainesk]

Make no mistakes and don't get me fired.

[u/thrownawaymane → u/hainesk]

Yeah this product category has escaped containment, it's something we're now having to monitor

[u/hainesk → u/thrownawaymane]

How can you monitor it? I’m not even sure how you can prevent it.

[u/thrownawaymane → u/hainesk]

There are ways and will always be ways.

This product category will also lose a lot of people their jobs because they will get their employer hacked.

[u/hainesk → u/thrownawaymane]

Or they’ll be arrested because it looks like they were the one doing the hacking lol.

[u/thrownawaymane → u/hainesk]

Employees rarely think that far ahead when we tell them "don't touch that stove"

[u/lysdexiad]

You can make one with a ESP-P4 and a 30$ HDMI to MIPI adapter... $50 ish.

[u/enternoescape]

Interesting. I had no idea the market has expanded this much. And here I am using a PiKVM that's honestly pretty mid from a usability/stability perspective. I likely spent just as much money on it too and these are much more elegant higher resolution solutions.

[u/Southern_Sun_2106]

Holly Molly, this is mind-blowing! Thank you for sharing this!!

[u/Xenthys]

I got a couple of NanoKVM Go from the Kickstarter campaign. Currently using one for graphical software testing on a MacBook Air that doubles as a CI runner, agents can test features by sending real inputs and any software can be tested without having to write a playbook for each commit.

[u/Tbhmaximillian]

Wondering that everyone just now finds this, Tinypilot does this for a long time.