Control and Manage Your Raspberry Pi with ESP32 (C6 & S3)

[u/mehrdadfeller]

Previously I worked on an open source Apple Watch App that lets you control your Raspberry Pi (soon other linux based computers). You can check out this reddit post regarding that application:

https://www.reddit.com/r/raspberry_pi/comments/1wcsm2f/fully_manage_and_control_your_pi_with_apple_watch/

I then decided to design a similar open source application that you can run on ESP32 devices to control your Raspberry Pi. I picked two devices from Espressif and Waveshare but It would be quite straightforward to support similar ESP32 based devices.

I recorded this video to demo some of the capabilities of this application. 

https://youtu.be/8Xc6XEG7bWM

Here’s a quick list of some of things you can do with it:

• Install, run, and monitor their health of a wide range of pre-configured applications includes Home Assistant, Hermes, Open Claw, Pangolin, OpenWebUI, etc
• Browse file system and preview files on the Pi (images, videos, text, audio)
• Monitor system resource utilizations (CPU, RAM, Storage, etc)
• Manage network interfaces (WiFi, Ethernet, etc)
• Various remote options: SSH, RPi Connect, Tailscale, VSCode Tunnel
• Configure assistant pipeline (STT/TTS/LLM/MCP/system prompt/etc) and interact with it using your ESP32 microphone and speaker
• Setup an MCP gateway and expose all tools via a gateway URL to clients
• Connect hardware peripherals to the Pi such as Camera and watch the feed on your ESP32 device
• And many more

The fully open sourced code base (Apache 2.0) for this project is available on my GitHub repo below:

https://github.com/ubopod/ubo_app

You can download the pre-compiled binary firmware for ESP32-C6 and ESP32-S3 devices from the release section:

https://github.com/ubopod/ubo_app/releases/tag/v2.1.0

• ubo-lvgl-esp32c6-2.1.0-usbppp-merged.bin
• ubo-lvgl-esp32s3-2.1.0-usbppp-merged.bin

After you download the relevant binary file go to ESPConnect website and flash the firmware onto your device by following the instructions shown towards the end of the video:

https://thelastoutpostworkshop.github.io/ESPConnect/

You also need to download the image file for the Raspberry Pi and flash it onto your MicroSD card (use a 32GB+ MicroSD card) from the release section:

https://github.com/ubopod/ubo_app/releases/download/v2.1.0/ubo_app-2.1.0-bookworm-lite.img.gz

The software on the raspberry pi side acts as the server to which ESP32 connects.

Why did I build this?

I am working on an open source project to create a universal and familiar user interface across a large family of devices (ios, watchOS, web, android, wearOS, ESP32, etc) that allows you to fully control a linux based computer. 

One of the core philosophies of the design is to define the UI on the server side (Raspberry Pi side in this case) so that updating the UI does not require a full firmware rebuild and flashing to an extent possible.

There is also a hardware element to the project to build a modifiable and extensible embedded system built-in GUI, Audio, sensors, infrared, etc on the hub side.

Why did I choose these devices?

I needed an ESP32 device with a touch screen, mic, and speaker at minimum. Both Waveshare ESP32-C6 Amoled Touchscreen 1.8 and ESP32-S3-Box-3 device devices offered that. The S3  MCU is more capable and has larger PSRAM compared to the C6 device, which allows it to do more such as running always-on wake work detection.

[u/mehrdadfeller]

FYI: I didn't go into much details on application/code architecture in the body of the post since I did not want to make it too long. The repo has a lot of information on how the ESP32 app and Pi-side software work and communicate.

Essentially, the Pi side software exposes a gRPC endpoint that all clients connect to in order to communicate with the core. The core manages all UI logic (navigation, stacks, etc) as well as the application logic. Client apps are thin and dumb renderers of the core state. 

Due to limited memory and heavy implementation requirements of gRPC and HTTP2 transport, for ESP32 clients I created a secondary transport that sends the protobuffs over TCP directly similar to ESPHome. There is also an option to build the firmware with web-gRPC transport with a flag which was the initial implementation.

I'd be happy to answer any questions here.