PYNQ Framework Based Object Recognition Implementation Using Convolution Neural Network (CNN)

Object recognition is a research hotspot in the field of computer vision, aiming to use some technique or method to obtain the location of the target in the image sequence. And determine the category to which the target belongs. Object recognition has high engineering application value in the fields of intelligent transportation, medical devices, security surveillance, etc. With the rapid development of target detection technology, many object recognition algorithms have been proposed by scholars, among which the target detection algorithm implemented using Convolution Neural Network (CNN) in deep learning has better detection performance compared with the traditional target detection algorithm. However, the computational process of CNN generally requires hundreds of millions of multiplication and accumulation operations, and the extremely large amount of computation demands high resources, bandwidth and computational speed of the hardware platform for CNN implementation.

To implement CNN-based object recognition algorithms, many embedded systems and supporting software development tools have emerged. And these systems are designed for faster computation, smaller device size, and lower operating power. CPUs, which use instruction sets for computing, are suitable for running software programs, but are not good at multiplying and accumulating operations on a large scale. GPUs are good at multiplying and accumulating operations and image processing, and are suitable for training and inference calculations of CNN. But GPUs’ high power consumption makes them impossible to implement in low-power embedded platforms. Compared with CPU and GPU, ASIC has the advantages of both high computing rate and low power consumption, and has a small size, but the customization cost is too high and the development flexibility is poor. FPGA is a high-speed parallel general-purpose device with high flexibility, low cost, high integration, low power consumption, moderate development cycle and wide application. FPGA uses data parallel connection to perform various parallel calculations, and is therefore well suited for implementing accelerated computation of CNN.

In this paper, we aim to use the PYNQ framework of Xilinx FPGA to accomplish object recognition implementation. We take full advantage of the reconfigurable and low power consumption of FPGA to quickly implement complex data parallelism and large-scale multiplication and accumulation operations in CNN based the PYNQ framework.

The contributions of this paper are:

(1) Hardware-software co-design scheme to implement object recognition on Xilinx FPGAs via ARM and FPGAs.

(2) Fast implementation of CNN acceleration based on the PYNQ framework.

This paper is organized as follows: Chapter 2 provides related work, Chapter 3 represents a PYNQ framework based CNN Accelerator. Testing and analysis is presented out in Chapter 4. Finally, gives a conclusion.

3.1 CNN

Due to the improvement of deep learning theory, CNN have been developed rapidly, and a number of new algorithms have emerged in speech recognition [10], face recognition [11], reinforcement learning [12], and natural language processing [13]. In CNN algorithms, neurons in the same convolutional layer are not associated with each other, while there are computational relationships between neurons in neighboring convolutional layers. Five main parts-data input layer, convolutional layer, pooling layer, fully connected layer and classification output layer, constitute the CNN. The data input layer represents the input of the original image data, and the input image features are collected through the convolutional layer and the pooling layer, and the collected image features are input to the fully connected layer and the classification output layer to complete the classification and recognition of the input image. The structure of the CNN is shown in Figure 1.

(1) Convolutional layer

The core idea of convolutional layers is to simulate neurons in the human brain, but unlike information transfer in the brain, in CNN, information is passed from layer to layer to each other, from the convolutional layer to the pooling layer, then from the pooling layer to the convolutional layer, and finally arrives in the classifier, which outputs the information of recognition result.

The main role of the convolutional layer is to extract features from the input feature map, and the parameters of the optimized convolutional kernel and BIAS can be obtained by BP (back propagation algorithm). Generally, a single convolutional layer extracts simple feature parameters, while a multi-layer convolutional layer can obtain more complex feature information by iterative methods.

The convolution is mathematically divided into the convolution of continuous and discrete functions, defined as follows.

$\begin{aligned} & \left(\mathrm{f}^* g\right)(n)=\int_{-\infty}^{+\infty} f(\tau) g(n-\tau) d \tau \\ & (\mathrm{f} * g)(n)=\sum_{\tau=-\infty}^{\infty} f(\tau) g(n-\tau)\end{aligned}$                     (1)

The convolution formula in CNN is defined as follows.

$\mathrm{f}(n, i, j)=\sum_{m=0}^{x n-1}\left(\sum_{v=0}^{k h-1}\left(\sum_{u=0}^{k w-1} w(n, m, u, v) * x\left(m, u+i^* s w, v+j * s h\right)\right)\right)$                      (2)

where, kw and kh stands for width and height, sw and sh represent the step lengths in the horizontal and vertical directions. f(n, i, j) represents the n output of the first image at the point (i, j), w(n, m, u, v) represent w weights.

(2) Pooling layer

The features collected through the convolutional layer can, in principle, be directly passed into the fully connected layer for network computation. However, the computational overhead of this approach is high, and the training and testing time of the CNN can be seriously increased, thus affecting the real-time performance of the algorithm. The goal of designing the pooling layer is to reduce the computational effort of the convolutional network by reducing the parameters of the feature map output from the convolutional layer. Therefore, the pooling layer is usually added after the convolutional layer to downsample the feature maps. Since a picture contains a lot of redundant information, the pooling layer can remove the redundant information to extract the important features. Pooling layer can also prevent overfitting to a certain extent. Overall, pooling layer effectively reduces the amount of data in the feature map after convolution calculation and improves the computation speed. As shown in Figure 2, the maximum pooling method and the average pooling method are the more common pooling methods.

(3) Fully connected layer

In a CNN structure, after multiple convolutional and pooling layers, one or more fully connected layers are usually connected. Each neuron in a fully connected layer is connected to all the neurons in its previous layer. Fusion of the local feature information collected in the convolutional or pooling layers is the main function of the fully connected layers. In order to reduce the network computation time of the CNN, the activation function of each neuron in the fully connected layer is usually a Rectified linear unit (ReLU) function. The output feature information of the last fully connected layer is fed to the classifier for data classification.

(4) Activation function

The activation function used Sigmoid function is a function used for binary classification which maps the input real numbers between (0, 1). The formula of Sigmoid function is defined as follows.

$f(x)=\frac{1}{1+e^{-x}}$                      (3)

11.png

Figure 1. Convolutional neural network structure for object recognition

2.png

Figure 2. Maximum pooling & average pooling

3.2 PYNQ framework

3.png

Figure 3. PYNQ framework

The PYNQ architecture is shown in Figure 3. From this figure, we can see that the PYNQ system architecture is divided into three layers: a hardware layer based on FPGA design, a software layer based on ARM-based Linux kernel plus Python, and an application layer based on Jupyter Notebook.

The hardware layer is designed mainly to realize the cooperative interaction between PS and PL, the whole FPGA part of the design is called Overlay also called hardware library, which is ZYNQ's PL (FPGA) design. overlay is mainly used to accelerate software applications, also for multi-user and multi-application to generate different bitstream files, and can be called through software APIs to the logic functions on the FPGA can be dynamically switched. The whole process is based on PYNQ, the C driver software is encapsulated in the PYNQ framework and provides the user with a Python call interface. The user can change the access IP of the overlay by changing the parameters of the Python interface, thus completing the underlying control of the programmable hardware.

3.3 Hardware-Software co-design framework

The CNN hardware-software co-design framework based on the PYNQ framework is shown in Figure 4. The test images are taken through the hardware, software and application layers of the PYNQ architecture and thus the test results. The hardware layer is supported by the PYNQ development framework, which also encapsulates the driver program, regulates the underlying hardware of the DMA, burns bitstream files, and solves tcl files. The software layer mainly completes the configuration process of encapsulating the computational tasks of each layer and provides a parameterized class call interface for designers. The application layer can automatically deploy the already trained Tensorflow models and designed to run on configurable CNN accelerator. During the development of the hardware architecture-driven control program, the program writer needs to complete the configuration of the entire DMA channel transfer using a high-level programming language based hardware driver. As showed as Figure 4, a large amount of computation is embedded in the computation of the network layer based on the PL part. Therefore, the basic model of CNN needs to be built by PC and passed to the PYNQ framework for more deep learning as well as practical operation, in order to meet the design requirements of hardware architecture and configuration. The ultimate goal is to make it easier and faster for users to design CNN accelerator based on the PYNQ framework for object recognition implementation.

4.png

Figure 4. Hardware-Software co-design framework

In this paper, we present PYNQ framework Based object recognition implementation using CNN. A hardware-software co-design framework is designed for the hardware and software of the CNN accelerator based on the PYNQ framework. The experimental results demonstrates that the computational speed of CNN accelerator designed and implemented in this paper reached 35.39 fps when testing the CIFAR-10 dataset. The computational speed is 43.2 times faster than the only ARM processor on the PYNQ-Z1 with CNN program. The recognition ratio of the hardware implementation and software implementation remains basically the same, both achieving more than 98%. The average power consumption of the CNN accelerator based on PYNQ framework in this paper for object recognition implementation is only 1.921w, which is very suitable for the rapid deployment and use of low-power embedded devices.