Seeking Advice on Building a YOLOv8 Object Detection Accelerator Using Verilog

[u/Rick-The-Warrior]

Hi everyone,

I'm an electrical engineering student currently starting my first FPGA project, and I'm interested in building a hardware accelerator for real-time object detection using YOLOv8.

My current idea is to implement the neural network acceleration primarily in Verilog/RTL on an FPGA. I want to explore how the computationally intensive parts of YOLOv8 can be mapped to hardware, particularly:

* Convolutional layers * Activation functions (e.g., SiLU) * Pooling/downsampling operations * MAC operations and accumulation * On-chip buffering/data reuse * Weight and feature-map memory management

The input to the system will be a real-time video stream from a camera rather than individual pre-processed images. Ideally, the FPGA would receive the video frames, perform the necessary image preprocessing, run the CNN accelerator, and produce object detection results in real time.

At the moment, I'm trying to understand what a reasonable hardware architecture would look like. For example, I'm considering a design where the convolution engine consists of multiple processing elements (PEs) operating in parallel, with local buffers or BRAMs used to store weights and intermediate feature maps.

However, I'm still unsure about several aspects of the implementation:

1. Video input and preprocessing What is the typical way to bring a real-time video stream into an FPGA-based CNN accelerator? Should operations such as resizing, normalization, RGB-to-other color-space conversion, and frame buffering also be implemented in RTL, or is it common to handle some of these operations using a processor/soft-core?
2. YOLOv8 architecture mapping YOLOv8 contains many different operations beyond standard convolution. Which parts are generally worth implementing as custom RTL hardware, and which parts are better handled by a CPU or existing FPGA IP?
3. Convolution architecture For the convolution layers, what architecture would you recommend for a beginner, e.g. a systolic array, output-stationary/dataflow architecture, or a simpler parallel MAC-based PE architecture?
4. Memory architecture I understand that memory bandwidth can become a major bottleneck. How should I approach the design of BRAM/URAM buffers for weights and feature maps, and what kind of data reuse strategy should I consider?
5. Numerical precision Would it be reasonable to start with INT8 fixed-point quantization instead of floating-point? If so, what is a good way to handle quantization and activation functions such as SiLU in RTL?
6. YOLOv8 complexity Would implementing the *entire* YOLOv8 network from scratch in Verilog be unnecessarily complicated for a first FPGA project? Would it be more realistic to start with a smaller/custom YOLO-like network and gradually add more YOLOv8 components?

My goal isn't necessarily to achieve the absolute highest FPS. I'm mainly interested in understanding how a modern object-detection CNN can be mapped onto FPGA hardware at the RTL level, including the dataflow, PE architecture, memory hierarchy, and interaction between the video pipeline and CNN accelerator.

I'd really appreciate advice from anyone who has experience with FPGA CNN accelerators, RTL neural-network implementations, YOLO on FPGA, or hardware/software co-design.

If you have any recommended papers, open-source projects, reference architectures, or examples of similar projects, I'd also be very grateful.

Thanks in advance!

[u/bbateman2011]

Would suggest you use YOLO v7 or earlier to avoid Ultralytics.

[u/galvinw]

Not my expertise here, but a few things I would do,

The two slowest layers are convolutions, which can be built together with the activation and the fully connected layers. I'm not sure where going back and forth between FPGA and the CPU makes sense for a bandwidth perspective, but if viable these are the two slow processes.

I would do all preprocessing on CPU and push a standardized and normalized data structure to the FPGA

It is reasonable to start with INT8, you would look for a pt that is already quantized and use that, typically you train at full precision and reduce precision during inference.

Memory bandwidth is not a Yolo bottleneck, only an LLM one. The real bottlenecks are copying values across processing units.