Image-Based Machine Learning Framework for LiDAR Data Analysis Accelerated on FPGA Platforms

The use of LiDAR data has become increasingly crucial for many applications, such as smart cities, autonomous vehicles, forest management, precision agriculture, and robotics. LiDAR technology is used to get a detailed picture of the shape and structure of an area of interest in either 2D or 3D point-cloud data. This data can then be utilized for object detection and classification of the designated area. However, the complex and large amount of data captured by LiDAR poses significant challenges to efficient processing in real-time decision-making applications [1, 2].

Techniques for analyzing LiDAR data have shifted from feature extraction methods that were hand-crafted to the utilization of machine learning (ML) methods such as deep learning. This has completely changed how LiDAR data is processed. The use of ML techniques cleared the way for LiDAR’s solutions, which are more robust and adaptive. Training the ML algorithms on diverse and large LiDAR datasets can be beneficial in extracting meaningful patterns and features from them, which can achieve efficient classification, object detection, and semantic segmentation with high accuracy [2, 3].

While machine learning models can be useful, they pose significant challenges as they need a great deal of processing power, especially when they are used in real-time systems. FPGAs present a viable solution for LiDAR dataset analysis because the FPGA technology offers a unique blend of flexibility, energy efficiency, and parallelism. FPGAs are a well-suited processing element for the implementation of ML-based LiDAR datasets, especially for robotic or embedded systems applications. They can customize hardware architectures that align with the ML-based LiDAR datasets algorithm requirements. This allows for a real-time LiDAR implementation that uses little energy and low latency [4-6].

For easier understanding, LiDAR data can be shown as pictures that make the data easier to understand [7, 8]. Also, for machine learning algorithms like CNNs, images are a standard format that can make integrating the ML model into an FPGA easier [9, 10]. The images’ hierarchical features can be extracted automatically, which benefits the classification performance greatly without the need for extensive feature engineering [11]. An image represents each processed version of the LiDAR dataset, illustrating intensity, height information, or other attributes. Therefore, different colours can visually represent the image’s various features, aiding in data interpretation. In addition, images make the analysis and understanding of its object characteristics or the underlying terrain more accessible, as the pixel values correspond to various measurements (e.g., elevation).

The research in this paper goes into excellent detail about how to use machine learning methods, especially image-based deep learning methods, to look at LiDAR data on FPGA platforms. The suggested framework looks at the difficulties and chances of creating effective and reliable LiDAR processing solutions. It shows how real-time decision-making and edge computing could be used in various robotic and embedded systems. Although numerous FPGA accelerators for LiDAR exist, most operate on raw point clouds, voxels, or hand-crafted features, which complicate hardware mapping and increase on-chip memory use. Therefore, there is a need for an alternative that can map the LiDAR frame into a compact 2D image that captures intensity/elevation patterns. This image representation enables efficient use of standard 2D CNN layers, simplifies the operator set needed for hardware acceleration, and reduces the memory usage. Therefore, there is a need for an alternative that can map the LiDAR frame into a compact 2D image that captures intensity/elevation patterns. This image representation enables efficient use of standard 2D CNN layers, simplifies the operator set needed for hardware acceleration, and reduces the memory usage. Therefore, the research objectives of this paper aim to design a compact and reproducible pipeline to convert LiDAR scan rows into 2D image representations that are compatible with lightweight CNNs, develop and train a lightweight CNN that can be used for FPGA acceleration, and deploy and evaluate the CNN model on a PYNQ-Z2 board using the Tensil AI toolchain.

This paper proposes a novel method that utilizes FPGAs and CNN algorithms to address challenges in processing LiDAR technology datasets. The proposed method proposes a unique method of converting the LiDAR dataset point cloud to an image format that is mostly compatible with FPGA hardware features. This approach shows an effective technique for achieving a reduction in processing time with low power consumption and increasing LiDAR accuracy.

Our contribution unfolds into three points as follows:

• Convert LiDAR readings to compact image representations compatible with CNNs and FPGA constraints.
• Design and train a CNN optimized for small-footprint hardware acceleration.
• Deploy and evaluate the model on a PYNQ-Z2 FPGA board using the Tensil AI toolchain, reporting accuracy, latency, and power.

The rest of the paper is structured as follows: Section 2 discusses the related work on LiDAR-FPGA-based solutions. Section 3 presents the methodology implemented. Section 4 delves into the experimental results with details on both the hardware and software findings. Section 5 discusses the results, implications, and findings. Section 6 concludes the work by suggesting potential avenues for future research.

This section presents the methodology followed and details the tools, LiDAR datasets, ML models, and hardware integration that are utilized in this work. In addition, the process of converting the LiDAR dataset to images and the deployment of machine learning models on the FPGA PYNQ-Z2 platform is detailed.

To conduct this study, the publicly available LiDAR dataset on Kaggle [25] is used. The data is generated by an RPLIDAR-A1 laser range scanner that is equipped in a mobile robot. It contains 360 different LiDAR signals for four different environments: corridors, doorways, halls, and rooms. Each of the four different environments (corridors, doorways, halls, and rooms) corresponds to one of the 411 instances in the data. The dataset is a near-balanced class distribution where each environment type has a similar number of instances.

The Xilinx PYNQ-Z2 [26] is chosen as the FPGA platform, which is an evaluation board that has the Zynq-7000 All Programmable SoC (System on Chip) devices. PYNQ-Z2 is part of the PYNQ (Python + Zynq) ecosystem, which offers the programming of the FPGA using Python to simplify the development process of hardware programming. Besides the Zynq-7000 SoC, the board contains an ARM Cortex-A9 processor, which makes it a flexible platform for applications such as robotics, signal processing, and embedded systems.

Figure 1 shows the framework methodology followed, which consists of five phases. The framework starts with the model development, which consists of several steps. Then in the second phase, the model is converted. The third phase involves preparing the FPGA platform. Phases two and three serve the fourth phase, which relates to the development of the model. And the final phase is where the inferencing takes place. The following subsections present these different phases.

Figure 1. Overall methodology framework of the proposed system

3.1 Model development phase

This phase has three steps, which start with the framework selection to specify the environment that is best suited for the model building.

Data Preparation: For the purpose of this work, the LiDAR dataset [25] is converted from its original CSV format into images for every class in order to prepare it for CNN machine learning tasks. Firstly, the data is obtained using the Pandas library, concentrating on only the relevant measurement data. The core of the conversion process involves reshaping the data into a three-dimensional array suitable for image representation. Specifically, the measurements are restructured into a defined height and width of 20 by 18 pixels in this case—resulting in a format of (num_samples, height, width). Normalization is then applied to the pixel values, scaling them to a range between 0 and 1, which is essential for improving the performance of machine learning models. In addition, the dataset is split into two subsets using the 80-20 ratio, one subset for training and the other for validation. It is a vital step to preventing overfitting and performance assessments. The dataset used in this work is provided in the CSV format and contains a total of 411 examples. Each row contains an ID column, 360 flattened feature values, and a final label column. The LiDAR dataset is stored as CSV by removing non-signal columns (ID and label) and extracting the 1D scan vector of length L for each sample. Each vector was reshaped into a compact 2D image of size height × width, such that height × width = L (used 20 × 18 = 360). Missing values were imputed using column means, and values outside the 0.5th–99.5th percentiles were clipped. Dataset-wise min–max normalization is applied to scale values to 0,1, expanded the tensors to single-channel format (Height, Width, 1). Finally, the images were saved, and arrays were also saved as .npy files for reproducibility. This compactness is essential for the FPGA implementation on the PYNQ-Z2, where on-chip BRAM and DSP resources are limited. Table 3 summarizes the dataset and its class naming and distribution.

Model Training: To facilitate the organization of the images for the model training and validation tasks, directories are established for each class label within the two subsets. And the final step is generating and saving the images. The corresponding LIDAR data is visualized in both the training and validation sets using Matplotlib. Then the visualized images are saved as greyscale images within their respective class directories. This process helps with the image classification tasks. These steps are essential for transforming the LiDAR dataset [25] into the desired format that is suitable for image classification tasks in this work. Furthermore, these steps enhance the deep learning techniques for analysis and prediction. The model was compiled using the Adam optimizer with the sparse categorical cross-entropy loss and trained to monitor accuracy. Training was performed with a batch size (B) of 32 and up to 300 epochs (E), learning rate (a) of 0.0001 and no momentum/decay parameter (b). A Model Checkpoint callback was used to save the best model according to validation loss. The training call used an augmented data generator and after training the best model for final evaluation is reloaded.

Table 3. Dataset summary

During training on-the-fly augmentation is applied using the Keras ImageDataGenerator with the following parameters: rotation_range = 20, width_shift_range = 0.2, height_shift_range = 0.2, shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True, and fill_mode = 'nearest'. The augmentation generator was fit on the training set and then used to provide augmented batches to the network during training.

For model selection, the checkpoint with the minimum validation loss is saved and is used for evaluation. Final performance on the held-out validation set was obtained using model.evaluate(X_val, y_val), which reports validation loss and accuracy. To assess per-class behavior, predictions are computed to convert probabilities into discrete labels. The confusion matrix and a full classification report (precision, recall, F1-score, and support) are generated. The confusion matrix was visualized, displaying integer counts with class labels 0–3 on the axes.

3.2 Model conversion phase

Prior to the model training, the TensorFlow platform [27] output in HD5 format is utilized for the model conversion phase. The TensorFlow output is exported using a Python script as a .pb file. It is then compiled and converted using Tensil AI framework that is designed for the ML model to the FPGA integration process. Upon successful conversion, Tensil AI produces an RTL (Register Transfer Level) written using Verilog HW language, which is used for the preparation of the board’s internal logic, and three model files: .tdata, .tmodel and .tprog. The .tdata files typically contain information related to the dataset structure and input shape. The .tmodel file represents the architecture and parameters of the trained model. The .tprog contains programming or configuration data related to deploying the model on the PYNQ-Z2 board. The Verilog file is used to prepare the HW design of the board using the Xilinx Vivado tool, while the other three files are used later to deploy the model into the PYNQ-Z2 board using Jupyter Notebook. Figure 2 illustrates the Tensil AI framework and the files it is outputting. It is worthwhile to mention that, unlike other frameworks, Tensil AI only requires the HD5 format file for the model to generate the HW files of the CNN model, which does not require a high level of experience in HW design flow.

Tensil AI facilitates the compilation and generation of the HW files of the CNN model.

Figure 2. Tensil AI framework and its role

3.3 Prepare the PYNQ-Z2 environment phase

The PYNQ-Z2 platform is selected for deployment because its Zynq-7000 SoC integrates an ARM Cortex-A9 processing system with programmable logic (PL), allowing the Tensil runtime and Jupyter notebooks to run directly on-board while the accelerator overlay executes in the PL. The PYNQ framework provides overlay support and Python APIs that significantly accelerate development, debugging, and reproducibility. This enables flexible integration between the host code and the Hardware files loading.

In order to deploy the CNN model in the PYNQ-Z2 FPGA board, the board must be prepared in order to receive the HW and CNN model files and perform the inference. To prepare the FPGA platform for the implementation of our model, three steps are necessary. First is the setting up of the PYNQ-Z2 board. Setting up the board involves loading the PYNQ-Z1 image, officially available on the PYNQ IO website, into an SD card and then inserting the SD card image to the board to boot the board from it. The second step is the installation of necessary packages, in which software tools such as Xilinx Vivado, the PYNQ framework, and Python with its libraries are installed. Xilinx Vivado is installed inside the host computer, while the PYNQ framework and Python are installed inside the PYNQ-Z2 board. The last step involves preparation of the HW design of the PYNQ-Z2 fabric. The HW design is an RTL file written using Verilog language and is produced by Tensil AI. It describes the internal logic that should be inside the FPGA fabric and is compiled, synthesized, and installed using the Xilinx Vivado tool.

After preparing the PYNQ-Z2 board, it can be booted as a network component with a specific IP address, and ML models can be downloaded to it using an Ethernet cable and Jupyter Notebook. It is worthwhile to mention that this preparation can be done only once, and every time the board is needed, it can be booted normally without any further preparation.

3.4 The model deployment phase

The CNN model was chosen to process and sort the LiDAR image dataset that was made by using multiple convolutional and pooling layers to pull out features. The CNN model architecture starts with an input layer that processes the raw data; see Figure 3. The next layer is a group of convolutional layers (Conv2D), each with a set of filters for capturing spatial hierarchies in the data. As you go deeper, the number of parameters increases. The MaxPooling2D layers disperse throughout the model, aiding in the reduction of the spatial dimensions of the map’s features. In addition, they maintain essential information, which ensures computational efficiency and the avoidance of overfitting; see Figure 3. To avoid overfitting and encourage generalization, the model includes a dropout layer that randomly disables a portion of neurons during training; see Figure 3. The final layer of the model architecture flattens the output into a one-dimensional vector, preparing the data for the dense layers. The dense layer has 128 neurons and combines the information learned in the earlier layers. It ends with an output layer that has four neurons that handle the classification tasks.

Figure 3. Model architecture and layers

3.5 Inference phase

The final phase in our framework is the inference phase, which is where the model is run to make predictions on new data. This step includes refining the output using post-processing and ensuring that it achieves the application requirements. Following that, as a last step, a comprehensive testing procedure was conducted for the model’s performance validation.

This framework gives a planned way to create and use machine learning models on FPGA platforms, making sure that each step is done carefully to get the best performance and dependability.

The results indicated that the proposed image-based machine learning framework for analyzing the LiDAR dataset is effective, particularly its deployment on an FPGA platform. Transforming the raw LiDAR data into images has facilitated the use of the convolutional neural network (CNN) method, which helps in classification accuracy and feature extraction tasks. In addition, further performance optimization resulted from integrating FPGA-based acceleration, which provided real-time inference along with efficient energy consumption.

The main achievement of this work is improving the accuracy and inference speed of the LiDAR dataset classification. The FPGA-based implementation achieved an accuracy of 98.80%, which is slightly higher than the software-based implementation (98%). It also took 2 times less time for the FPGA version to draw conclusions than the software version (0.6207 seconds for 83 test images, or 0.0077 seconds per image), which shows that it can be used in real-time situations. In real-time applications, the inference time is a vital element, particularly in autonomous navigation, robotics, and edge computing. The performance boost comes from the PYNQ-Z2 FPGA’s efficient hardware and the ability to do computing in parallel. This work’s FPGA implementation consumed ultra-low power, just 1.616 W, which is about 24 times more efficient than the GPU-based processing used in the study. These enhancement results show that our proposal is an efficient real-time candidate solution for applications such as autonomous navigation, robotics, and edge computing.

In comparison with the related studies listed in the related work section, our work provides a different approach that facilitates the integration of deep learning models on FPGA platforms. It leverages CNN-based classification on LiDAR data, represented as images. Unlike our approach, which is highly compatible with CNN architectures, the other approaches [1, 12, 28, 29] are focused mostly on traditional feature extraction techniques or direct point cloud computations. These approaches did not explore image transformation techniques that enable seamless deployment of the ML models to FPGAs. Studies [20, 22] utilized a CNN-based method for processing the LiDAR dataset; however, it utilized high FPGA resources, while our method achieved a balanced solution with high accuracy and speed and low power consumption (1.616W).

Furthermore, the LiDAR data-to-image transformation facilitates the data interpretation and allows for the reuse of well-established image processing and deep learning methodologies. The CNN architecture works well for classifying images, and the proposed framework takes advantage of this to find intensity-related and spatial patterns in the dataset that might be hard to handle with traditional point-cloud-based processing methods. The results provided a more intuitive interpretation of the LiDAR dataset features and improved the classification capabilities.

Although the proposed framework showed many advantages, some drawbacks should be acknowledged. One of them is that the dataset used does not represent the complexity of real-world environments and might not be a candidate that can be generalized. Therefore, the incorporation of diverse datasets that have different outdoor scenes, complex obstacles, varying lighting conditions, and different environmental settings can address this limitation. Also, the CNN model can be made even more efficient on FPGA platforms by using techniques like quantization, hardware-aware neural architecture search (NAS), and pruning, along with other optimizations.

The preprocessing overhead associated with the initial conversion of the LiDAR dataset to the image format presents another limitation. Therefore, exploring more efficient direct feature extraction techniques or efficient encoding strategies to convert LiDAR raw data into images is recommended. In addition, the performance can be further enhanced by integrating extra neural networks, like lightweight deep learning frameworks or transformer-based models, which could be useful to explore in future work.

The findings in this work proved the viability of combining FPGA acceleration with the CNN method to analyze and classify LiDAR data. The result paves the way for integrating the energy-efficient hardware elements and real-time-capable LiDAR classification systems, particularly in applications such as autonomous navigation, edge computing, and robotics. In addition, this work shows a promising solution for elevating the real-time LiDAR processing.

This work is supported by Taibah University.