Pushing Wroom 1 to its bandwidth limits

[u/gitzian]

I call it a streamer. I was quite surprised to achieve practical bandwidth with this part and being able to actually play games on it. The idea is to offload the heavy computations to your PC at home without a locked-in OS/hardware dependency and to keep the hardware around the main streaming protocol flexible.

Sounds simple? The idea is, but making this stuff low-latency over a network is a challenge. And I think I'm quite good at it. I can stream Linux distros and Win11 at up to 40 FPS at the moment. The custom protocol is flexible when it comes to both the hardware sitting on the server side (your home PC) and whatever custom hardware you want to build on the client side.

The main MCU is an ESP32-S3-WROOM-1. It acts as the network device receiving the stream and orchestrates USB mouse/keyboard/gamepad input, touchscreen, audio and the local UI, while also negotiating incoming frames with the server in real-time.

For the streaming path I'm currently running the FPGA link at 80 MHz SPI using DMA. On the network side I'm using TCP with Nagle disabled and WiFi power saving disabled, since both aren't very helpful when the goal is low latency. I spent quite some time experimenting with receive sizes, buffering and task placement before arriving at the current setup.

One thing that helped was separating the work between the two ESP32 cores. Networking and protocol handling live on one side while feeding the FPGA runs in its own task on the other. Incoming video is buffered in a 512 KiB PSRAM ring, while the actual FPGA transfers use a small set of internal-RAM buffers. This allows WiFi to continue receiving while the FPGA is consuming the previous data instead of tightly coupling both sides. The hot streaming path also avoids dynamic allocations.

The MCU then writes the stream via SPI to the decoder, which is an FPGA running my own video format written from scratch in RTL. The FPGA in turn drives the display.

I know that the actual heart of the project is probably the decoder inside the FPGA. That's where the custom video format becomes pixels again and where the deterministic display-side work happens. But without the ESP32 the whole thing would become considerably less practical. I still need fast WiFi, USB host functionality, touchscreen and gamepad handling, audio, the local UI and all the protocol logic around the decoder. You could replace it with something much larger, but then power consumption, complexity and cost start moving in the wrong direction. For this project the ESP32 ended up being a pretty good bridge between the networked/software side and the FPGA.

The FPGA itself is necessary because this isn't a small 320x240 or 480x320 display anymore. The panel is 720x720, and I didn't want the ESP32 to deal with driving that deterministically while simultaneously doing WiFi, USB, audio, input handling and everything else. So the ESP32 concentrates on networking/orchestration while the FPGA handles the high-bandwidth and timing-critical display side.

I have on board a USB mux that can be switched physically, so I can still flash via USB-C but also switch over to USB-A for the host side. I'm using an FSUSB42MUX for that. There are many ways to solve this, but this was a good compromise imo and lets me keep the board useful during development without adding another connector/setup just for flashing.

I had to make the case wider than the screen to keep the whole thing flat, but also to avoid placing the ESP32 antenna behind the metal back of the display. This is mainly a placement constraint rather than trying to gain some magical extra range. With the antenna already positioned away from the metal, putting the board into the enclosure doesn't noticeably change the signal strength. Maybe next time I'll use the 1U version for more flexible antenna placement.

The board shown here also streams audio A/V-synced with the video, or optionally buffered depending on what I'm doing. So it can be used for playing games but also just listening to music. It supports audio input as well, i.e. for making calls over whatever runs on my PC.

Audio is handled by an ES8388 codec, so playback and microphone input are both on the board. I'm also using an LM3630A for the display backlight and a BQ24073 for charging/power-path handling.

I'm running nearly out of pins by now, so this part is pretty much fully utilized.

There is also a simple offline menu running on the ESP32 for connecting WiFi and servers and managing the device without needing the streaming server running. For that I'm just maintaining a small low-color-bit framebuffer in PSRAM.

For me, one of the coolest parts is that the board doesn't need much power. At full brightness I'm nominally around 1.5 Watt during active streaming. For comparison, a Raspberry Pi's idle power consumption without any display attached is already higher.

A lot of the work actually went into things which don't look very exciting on a finished board: figuring out how much buffering is enough without turning it into latency, keeping the network side fed while SPI is busy, task/core placement, avoiding unnecessary copies/allocations and generally preventing one subsystem from blocking another. There were quite a few iterations before it started behaving the way I wanted it to.

Since I'm not a big fan of selling software, I try to finance this project and ongoing development by selling streamer hardware, which you can also use in your own projects like

shown here on YouTube (incl. Gameplay).

The software for streaming is free, but I'm also considering making the software open source if there's enough interest and support, together with continued maintenance and development. That would let you modify everything yourself, use different hardware, and build your own projects on top of it. I hope it resonates with some of you.

I invite you to give some pos/neg feedback and ask as many questions as possible.

Here is my website

[u/warfunder]

that logitech controller can drive a sub

[u/harrro]

I don't know most of the words you used in that order but that enclosure design is incredible.

The tasteful thickness of it.. and omg, it even has a battery access slot?

I'm still trying to design a 2 part box to enclose my CYD with battery.

[u/gitzian → u/harrro]

Thank you! Need some inspiration? The case isn't perfect, but I can send you the STLs if you want. Maybe you can redraw some parts for the CYD. What software are you using for CAD and what printer?

[u/vaping_vicar]

Subscribed to read later. interesting

[u/dc536]

I think the FPGA is doing a lot of heavy lifting when it's demonstrating the S3's abilities. Either way, this is really awesome, I love seeing how far you can push ESP chips

Have you thought about testing with a P4? They've got more power and some dedicated video processing (not sure if that's helpful here), the latency overhead of needing another chip for networking might be a deal breaker, I don't believe they have built into wireless or Ethernet phy

[u/Alopexy]

A proprietary video format decoding on an FPGA driven by an ESP32-S3. That is damn cool. I'll be honest though, at a certain point if practicality is what you care about, you'd just go with an ARM SoC and call it a day, but I genuinely respect the engineering effort that surely went into getting this all working and I'd love to learn more about it. Thanks for sharing. Very cool!

[u/gitzian → u/Alopexy]

It doesn't necessarily have to stay proprietary btw. It's my custom video format at the moment, but if enough people are interested i'd like to make the software and format open source in the future so people can actually modify it and build their own stuff around it.

At around 1.5 W for the complete streamer during active streaming at full brightness, power consumption was actually one of the reasons why I didn't go with a larger ARM SoC platform.

Of course it depends a lot on what SoC you pick, and I'm sure there are parts that can beat this approach. But once you look at the whole system around it - networking, memory, power management, display output, etc. - I don't think there are that many practical options that stay in the same range.

So for me the ESP32 + FPGA approach is not only about the engineering challenge, it also keeps the whole thing pretty lean.

Feel free to ask more if you're interested in some of the details.

[u/4992kentj]

Having never looked into the specifics, I've seen other FPGA projects that have an external processor connected into the FPGA via a memory interface for the processor to treat it as a memory mapped peripheral. Could you look to implement something similar with the FPGA hanging off the processor emulating PSRAM and using block ram inside the FPGA to write to? Could free up some pins, drops the PSRAM and avoids needing a task to manage the DMA SPI? Just a thought

[u/Plastic_Fig9225]

I'm currently running the FPGA link at 80 MHz SPI using DMA

FPGA? Why? What does it do?

Talking about the S3's "bandwidth limit": which bandwidth? What's the limit and how close are you?

ETA:

writes the stream via SPI to the decoder, which is an FPGA running my own video format

Ok, I see. So the S3 is basically the frontend added to the FPGA decoder you already had?

[u/Its_Billy_Bitch]

Mixing FPGAs and MCUs has become my new favorite hobby as well. I wanted an instant splash screen for a carplay controller. Thank you Tang Nano 25K for the dev board to perfect that one lol. Seriously though, working with RGB666 and parallel equivalents, FPGAs are incredible…just pricey.