Technological Trends in DL-Based 3D Digitization of Infrastructures by Using Point Clouds

To achieve our objective, a systematic literature review was conducted using the guidelines and principles defined by Kitchenham [10], ensuring the rigor and reproducibility of the process. Figure 1 shows the Kitchenham approach, which comprises three steps: planning, execution, and reporting.

1.jpg

Figure 1. Kitchenham methodology

2.1 Planning stage

The objective of the systematic review is to evaluate the impact of technological advancement on 3D digitalization with AI according to the type of infrastructure. For this, the review of works published between the years 2020 and 2024 was considered. The reason we chose the year 2000 as the starting point for our review is the rise in the use of AI.

This research focuses on the analysis of identification machine learning techniques used for the segmentation and classification of point clouds according to the type of infrastructure, and to achieve this, the questions presented in Table 1 must be answered.

Table 1. Research questions

2.2 The implementation stage

Table 2 details the organization of the keywords into four blocks. In the first block, words related to technologies such as "3D scanning," "laser scanning," and "point cloud" were established. In the second block, the keywords related to AI and the geometric recognition of objects in point clouds were defined as: "object recognition", "object detection", "classification", "segmentation", "instance segmentation", "semantic segmentation", "part segmentation", "segmented point cloud". In the third block, keywords related to AI techniques were selected using keywords such as: "Artificial intelligence", "machine learning", "deep learning", "convolutional neural network". Finally, in the fourth block, articles related to the field of application were selected using keywords such as: "construction", "structural components", "engineering", "infrastructure", "structural elements", "industrial plants", "processing plants", "structural parts", "infrastructure elements", "infrastructure components", "industrial process plant", "pipework components", "industrial facilities". This search resulted in the identification of 552 articles, which underwent a selection process considering the inclusion and exclusion criteria, see Table 2 and Section 2.3.

Table 2. Inclusion and exclusion criteria

In this systematic review, models and datasets are classified according to the type of infrastructure (transportation, construction, industrial, and public and social services). It aims to identify trends in the use of AI and 3D digitization for the design, construction, and maintenance of industrial infrastructures.

2.3 The reporting stage

For the report, a bibliometric analysis was conducted [11], for which version 1.16.19 of the VOSviewer Software [12] was used; this tool allows for the creation, visualization, and analysis of bibliometric networks. The extraction of information and its synthesis ensured the updating and inclusion of all bibliographic sources and allowed for the reduction of complex and abundant information. Particularly, in the present article, topics related to 3D digitization, semantic segmentation, point clouds, and AI in the context of structural components in the field of engineering were addressed.

From the initial set of 522 articles, the selection process was carried out considering the inclusion and exclusion criteria, resulting in 236 articles. After conducting a complete reading of the articles and excluding those that were not relevant and/or unavailable for download, the final set consisted of 92 scientific articles indexed in Scopus. Figure 2 shows the described process.

2.jpg

Figure 2. The selection process of Kitchenham’s approach

Figure 3 shows the most frequent keywords obtained from the literature review, highlighting "object detection", "semantics", "deep learning", and "point cloud", derived from processing 92 articles in VOSviewer.

3.jpg

Figure 3. Keywords of the review article

Despite the classification of the scientific contribution of 3D digitization with AI according to the type of infrastructure, as shown in Figure 4 and described in the responses to research questions RQ1a-RQ1d, this article extends its development in industrial infrastructure, as it is linked to the mining industry.

RQ1a: What are the most relevant applications of integrating 3D digitization through point clouds and AI in industrial infrastructures?

With the aim of answering the question RQ1a, a comprehensive literature review was conducted. Table 3 summarizes the main findings: a) the datasets covering industrial infrastructures, among which Pipework, CLOI, and PSNet5 stand out; b) the DL models used for training, and c) the accuracy of these models.

Table 3. Precision of IA-based 3D digitization models of industrial infrastructures using point clouds

The advances in the classification and segmentation of industrial infrastructures using advanced 3D digitization techniques and DL models stand out for their high accuracy in identifying structural components. DL models such as PointNet and its variants, like PointNet++ and SE-PseudoGrid, have proven to be very effective, achieving accuracies ranging from 71.5% to 99.7%, depending on the complexity of the dataset and the number of classes involved.

For the Pipework dataset (2020), which includes 17 classes achieved an accuracy of 73.3% using the PointNet model. In the case of the Semantic 3D dataset (2020) and the PointNet-RDNet model, which was applied in mining terrains, classifying 8 classes achieved an accuracy of 94.0%. Another significant example is the CLOI dataset (2020), which classifies 10 classes of industrial components, achieving 82% accuracy with the CLOI-NET model. The PSNet5 dataset (2021) excels in the classification of 5 classes of components, using the ResPointNet++ model, achieving an accuracy of 95%. This demonstrates the model's ability to handle the identification of industrial elements with high precision.

On the other hand, the model that combines CNN and RNN achieved an accuracy of 86.1%, performance reached in the recognition and classification of 6 classes of components. Similarly, the DNN model achieved an accuracy of 99.7% in the recognition and classification of pipes, standing out over the other models. Figure 6 shows the models with an accuracy above 95% are shown, a notable example is the SE-PseudoGrid model, which achieved an outstanding accuracy of 97.5%.

6.jpg

Figure 6. IA-based 3D digitization through point cloud processing for industrial infrastructure with over 95% accuracy

PointNet-based models stand out for their ability to directly process cluttered point clouds without the need for preprocessing. This approach has proven effective in industrial contexts, achieving high accuracy in specific tasks such as pipe classification (97.5% with SE-PseudoGrid in [9]). Furthermore, its lightweight architecture makes it suitable for real-time applications. However, it has limitations when semantically segmenting complex geometries and lacks contextual awareness in large-scale environments, which is reflected in lower performances, such as in the studies [13, 18], where accuracies were 73.3% and 71.5%, respectively.

These results show the potential of AI models, such as DNN, SE-PseudoGrid, and ResPointNet++ to enhance automation in the identification of critical components in industrial infrastructures. Clearly, Industrial component identification is being increasingly investigated as can be seen in Table 3. Pipes and valves are the main structures researched because they are key components in many industrial sectors, such as energy, petrochemicals, water management and industrial processes, so their research is justified by the critical need to monitor, maintain and optimize these systems. The high frequency of research on pipelines reflects the importance of automating their inspection and management in order to prevent failures, leaks or accidents, reducing downtime and operating costs.

RQ1b: What are the most relevant applications of integrating 3D digitization through point clouds and AI in transportation infrastructure?

In the analysis of transportation infrastructures, the results of the models in various studies demonstrate remarkable effectiveness in the classification and recognition of critical components. Figure 7 shows the models that exceed 95% accuracy. The SP-Network model achieved an impressive accuracy of 99.7%, using the Road infrastructure dataset which covers a variety of classes including road surfaces, buildings, and traffic signs. Another relevant model is EffNet, which achieved an accuracy of 97.7% using the Railway dataset that includes: poles, catenaries, and cables.

In 2024, the KPConv model achieved an accuracy of 99% using the Rail3D dataset, demonstrating the ability of current models to classify a wide range of elements, from vegetation to signals and support structures. This is confirmed by looking at the Figure 7 that reports the BrIM model, which achieves an accuracy of 97.26% using the Highway bridges dataset, excels in identifying essential components such as beams and slabs. These results demonstrate notable progress in automating and enhancing transportation infrastructure management, which optimizes safety and operational efficiency.

7.jpg

Figure 7. IA-based 3D digitization through point cloud processing for transportation infrastructure with over 95% accuracy

Automation through the use of IA models can significantly improve the efficiency and accuracy of inspections. Table 4 shows that the high frequency of research into bridges, railways, roads and tunnels reflects their critical importance in the transportation system and the urgent need to keep these infrastructures in good condition. The practical applications of this research in predictive maintenance, autonomous monitoring, asset management and road safety are marking a step towards smarter, safer and more efficient transportation systems. Moreover, advancements in technologies for the automatic identification of components within transportation infrastructure are expected to continue, thereby contributing to enhanced quality of life and promoting greater sustainability within both public and private transportation sectors.

Table 4. Precision of IA-based 3D digitization models of transportation infrastructures using point clouds

RQ1c: What are the most relevant applications of integrating 3D digitization through point clouds and AI in construction infrastructure?

In the analysis of construction infrastructures, the results from datasets of various studies demonstrate remarkable effectiveness in the classification and recognition of critical components. Figure 8 shows the models that exceed 90% accuracy. For example, a notable result comes from the BIM models, where the Hybrid model achieved 90% accuracy in classifying essential elements such as beams, columns, walls, and doors. This approach is fundamental in the implementation of BIM, which facilitates the planning and management of construction projects.

8.jpg

Figure 8. IA-based 3D digitization through point cloud processing for construction infrastructure with over 90% accuracy

In the case of the U-Net model that used the CrackSeg dataset, used for crack detection in pavements, it achieved an accuracy of 97.8%, highlighting the importance of computer vision techniques in infrastructure maintenance.

The NVE-DGCNN model showed an accuracy of 96.9% with the Bridges dataset, highlighting the effectiveness of models in identifying critical structures such as bridges. Moreover, in the analysis of metro stations in Beijing, the use of PointNet++ achieved an accuracy of 60%, which demonstrates that classification in complex environments remains an area with significant opportunities for improvement.

Finally, the DGCNN model reported an accuracy of 94.2% on the S3DIS dataset, demonstrating a robust ability to identify elements within buildings, which is essential for interior space management. These results reflect the continuous development and application of advanced models in the classification of components in the construction sector, improving the efficiency in the design and maintenance of infrastructures.

It is well known for the use of BIM models for the digitalisation of infrastructures, facilitating project planning, resource optimisation and management during construction. Then, by integrating machine learning models, it enables the detection of possible errors in the design phase and improves the accuracy in the execution of works. Table 5 shows that the most investigated classes are: pillars, columns and beams, which are fundamental components in almost all construction infrastructures, since they support structural loads. Therefore, automatic inspections of these elements are essential to identify cracks, corrosion and other defects that could compromise the stability of a building or infrastructure.

Table 5. Precision of IA-based 3D digitization models of construction infrastructures using point clouds

RQ1d: What are the most relevant applications of integrating 3D digitization through point clouds and AI in ISPS?

The information presented in Table 6 reflects the performance of various models in the classification of components within the ISPS category. Over the years, remarkable results in terms of accuracy have been observed. In 2022, the GFSAE model achieved an accuracy of 95.4%, operating on the Semantic 3D dataset which includes 8 classes such as terrain, vehicles, among others. This highlights the effectiveness of DL techniques in segmenting complex data in urban environments.

Likewise, in 2022, the U-Net 3D model achieved an impressive accuracy of 96.9% with the dataset City center of Stuttgart, Germany, demonstrating its ability to distinguish between various urban elements such as buildings and vegetation. In 2023, the D-Net model achieved an accuracy of 97.6% with the US Highway dataset, correctly identifying various objects, including buildings and electrical cables.

Figure 9 shows the IA-based 3D digitization through point cloud processing for ISPS with over 90% accuracy, as it can be seen the model MIF-PointNet++ offer a 98% of mAcc. These results underscore the progress in the use of advanced models for the identification and classification of components in ISPS, which could have a significant impact on the planning and management of these vital resources. The continuous improvement in the accuracy of these models suggests progress towards automation and optimization in urban infrastructure.

Evidently, urban buildings are the basic element in any city, and therefore 3D modelling of buildings is essential for urban planning, asset management and land use determination. The sector classes urban buildings, vegetation and natural elements are the most frequent in Table 6. The use of 3D models in public infrastructure allows urban planners to create accurate digital maps of urban areas, facilitating decision making on land use, construction areas, green areas and road infrastructure.

9.jpg

Figure 9. IA-based 3D digitization through point cloud processing for ISPS with over 90% accuracy

Table 6. Precision of IA-based 3D digitization models of ISPS using point clouds

In the case of digitalisation of vegetation and natural spaces in cities, it can be used to manage urban biodiversity, plan the renewal of green areas and ensure that cities are more sustainable. It also allows us to assess the environmental impact of infrastructure and develop mitigation strategies for the changing effects of climate.

RQ2: What available datasets are related to structural components of industrial infrastructure?

Currently, the availability of public datasets, which encompass structural components in industrial infrastructure, is significantly limited due to the high costs associated with the acquisition and labeling of these data [15]. The datasets presented in Table 7 are the most relevant resources available for industrial infrastructure.

Table 7. Most used dataset for 3D digitization with AI for industrial infrastructure

Pipework [13]: It is a repository specifically designed for semantic segmentation tasks of 3D point clouds in the context of industrial components, particularly focused on the segmentation and classification of pipes and their associated elements within industrial facilities, such as factories or processing plants. For the creation of this dataset, a terrestrial laser scanning (TLS) was conducted at a petroleum refinery plant. Pipework comprises a total of 4647 shapes grouped into 17 classes, which together encompass 110 million points addressing various industrial components.

CLOI [15]: It is a dataset designed for semantic segmentation and large-scale object classification, with a specific focus on industrial components and large structures in manufacturing environments or industrial plants. It is one of the most extensive TLS collections of industrial plants, this dataset was carried out in five industrial facilities, three were warehouses, one was a petrochemical plant, and the fifth was an oil refinery. CLOI consists of 12125 shapes grouped into 10 classes, covering 140 million points that address industrial, indoor, and outdoor situations. These classes include electrical conduits, straight pipes, circular hollow sections, elbows, channels, solid bars, beams, angles, flanges, and valves. This dataset provides a detailed and comprehensive representation of industrial infrastructure, making it a valuable tool for applications that require precise analysis of industrial environments.

PSNet5 [8]: It is a well-known public dataset that presents a collection of 3D point clouds of industrial components. PSNet5 was built using a TLS, selecting four industrial scenarios from water treatment plants in Hong Kong, which include Chiller House (CH), On-Site Chlorine Generation (OSCG), Sludge Press House (SPH), and Wash-water Recovery Tank. (WRT2). The four scenes used are mainly composed of elements such as pipes, pumps, beams, and other industrial categories. This dataset consists of 5 classes with a total of 80 million points. The large amount of training data is sufficient to train DL models and fully leverage their capability in perceiving three-dimensional data of industrial structural components.

The most representative datasets for this case study are CLOI, Pipework, and PSNet5. These datasets were generated using TLS, which includes RGB features. To classify the classes, a 3D semantic segmentation was performed, with classes related to structural components.

RQ3: What AI models use point clouds for the recognition of structural components in industrial infrastructures?

To address this research question, a thorough review of DL-based models intended for semantic segmentation of point clouds was conducted, considering publications from recent years. Attention has been given to the approaches developed and applied in the context of structural components in infrastructures.

Figure 10 shows the taxonomy of DL models with an emphasis on applications with 3D point clouds. In this figure, three main categories can be observed: 3D shape classification, 3D point cloud segmentation, and 3D object detection-tracking.

10.jpg

Figure 10. Taxonomy of DL methods for 3D point cloud segmentation [102]

3D point cloud segmentation requires comprehension of global geometric structures and the specific details of individual points. Based on the level of granularity, 3D point cloud segmentation methods can be categorized into three subtypes: semantic segmentation (scene level), instance segmentation (object level), and part segmentation (part level) [102]. The approach to semantic segmentation of point clouds is to separate them into several subsets based on the semantic meaning of the points, where four specific methods stand out: the projection-based method, the discretization-based method, the point-based method, and the hybrid-based method.

This research focuses on the point-based method, which is subdivided into several approaches. One of the main ones is the so-called Pointwise MLP; these point-based methods perform learning directly on the irregular point cloud. These clouds lack a defined order and structure, which makes difficult the direct application of standard Convolutional Neural Networks (CNNs), originally designed for data organized in regular grids. This limitation was overcome in 2017 with the introduction of the PointNet model [6], which was designed to process unstructured 3D point clouds without requiring conversion to a structured grid or voxel representation, then this contribution has served as a reference for various segmentation methods.

In general, most existing DL-based point cloud segmentation applications, especially those implemented in the field of industrial structural components, have opted for point-based methods. In what follows, a brief description of the most commonly used and most accurate models, according to the literature review is described.

PointNet [6]: Pointnet is a novel type of neural network that directly consumes unordered point clouds, which also takes care of the permutation invariance of points in the point cloud. Pointnet can do object classification, part segmentation, and semantic parsing. The main feature of Pointnet is the network is robust with respect to input perturbation and corruption. Also, the network can learn to summarize a shape by a sparse set of key points.

PointNet++ [7]: is a hierarchical network that applies Pointnet recursively on a nested portioning of the input point cloud. It proposes novel set learning layers to adaptively combine features from multiple scales from varying densities. Similar to CNNs, Pointnet++ extracts local features from a small neighborhood and further grouping into larger units and processes to produce higher level features. This process is recursive until we obtain the feature of the whole point set.

ResPointNet++ [8]: ResPointNet++ is a model for industrial point clouds, which significantly improves the performance of DL-based methods for industrial point cloud segmentation. Through the implementation of the ResPointNet++, it is feasible to effectively segment large-scale industrial LiDAR point clouds by using a neural architecture with deep residual settings.

SE-PseudoGrid [9]: The SE-PseudoGrid architecture enhances the network's ability to differentiate and classify 3D objects, excelling in industrial and robotics applications. SE-PseudoGrid model, called SE-LAO, extracts descriptive features from the point cloud during the process of learning the representation of these clouds. The integration of the SE mechanism and the PseudoGrid model contributed to a more effective approach for classification. The model architecture is composed of two components, the encoder and the header. The encoder uses a shared fully connected (FC) layer, generating feature embeddings. Through SE layers and residual blocks (RB), the encoder learns significant local structures. Then, a series of SE-LAO and SE-RB layers are applied to extract local geometric structures from each point.

The precision of a model is also measured by metrics such as Mean Overall Accuracy (mOA) and Mean Intersection over Union (mIoU). These metrics are commonly expressed in percentage terms and are widely used to evaluate the performance of models in point cloud processing with AI in a 3D digitization process, see Table 8.

Table 8 presents the Pipework, CLOI, and PSNet5 datasets. These datasets are related to the case study and are available to the academic community. The Pipework dataset was first used in 2020. The study [14] proposed a DL-based method to retrieve pipe component catalogs to support the reconstruction of a 3D CAD model of a process plant with point clouds. The data retrieval system identifies the type of pipe components, searches for and extracts data from the catalog, and performs post-processing of the catalog data. The authors applied the PointNet model, obtaining the following metrics: mOA of 79.9%, mAcc of 73.6%, and mIoU of 60.8%, with their respective mIoU per class presented in Table 5.

In the study [15], the CLOI dataset was used to evaluate DL classification models along with their new model CLOI-NET. They achieved superior results compared to other models, reaching the following metrics: mOA of 83%, mAcc of 59%, and mIoU of 45.1%, with the respective mIoU per class presented in Table 8.

In the study [19], the CLOI dataset was used to evaluate DL classification models along with their new model SFR-PointNet++. The authors achieved superior results compared to other models, reaching the following metrics: mOA of 72%, mAcc of 50%, and mIoU of 38.1%, with the respective mIoU per class presented in Table 3.

The study [8] used the PSNet5 dataset to evaluate DL classification models along with their new model ResPointNet++. They obtained superior results compared to other models, achieving the following metrics: mOA of 94% and mIoU of 87.3%, with the respective mIoU per class presented in Table 8.

The pipework dataset was used to evaluate DL classification models along with their new SE-PseudoGrid model [9], and it achieved superior results compared to other models, reaching the following metrics: mOA of 96.2%, mAcc of 97.5%, and mIoU of 96.5%, with the respective mIoU per class presented in Table 8.

Table 8. Most used models with corresponding precision for 3D digitization with AI for the industrial infrastructure

Industrial structural components play a critical role in industrial plants, and the 3D digitization of these components from point clouds has gained significant attention. A major challenge has been the ineffectiveness of semantic segmentation. Accurate classification of structural components typically requires professional intervention, leading to inefficiencies. However, recent advancements in DL methods for classification, semantic segmentation, and object detection in point clouds have shown promise for industrial structural components, particularly with the development of pointwise MLP methods for segmentation, as illustrated in Figure 10.

After conducting a thorough analysis of the datasets, the datasets related to industrial structural components are as follows: Pipework, CLOI, and PSNet5, these datasets have played a fundamental role in the evaluation of DL models designed to address the interpretation of three-dimensional industrial scenes.

In particular, it is essential to highlight that 3D digitalization with AI is a growing trend in the mining industry, as it allows for better understanding and planning of infrastructures. The aforementioned DL models, such as PointNet, PointNet++, ResPointNet++, and SE-PseudoGrid, have proven to be highly effective in analyzing and processing large datasets of point clouds in environments other than industrial mining infrastructures. However, they have been successfully used to identify and classify different types of rocks and minerals in three-dimensional images using the PointNet model, in a type of infrastructure that could be considered similar to transportation infrastructure or construction infrastructure, according to the classification described in this paper.

Therefore, by combining 3D digitization with AI and using appropriate models and datasets, greater efficiency and accuracy can be achieved in the characterization of mining infrastructure. This can lead to better planning and management of mining projects, as well as a reduction in costs and risks associated with resource exploration and extraction.

A global analysis of the industries related to the 92 articles investigated in this article confirms that: a) the industry where this technology is most used is the oil or natural gas industry, where it is employed for the inspection and maintenance of metal structures in difficult and hazardous environments, b) there are no articles of the industrial infrastructure type that are related to the mining industry. There are some articles that could be related to rocks or mining areas, but not those related to industrial processes. Consequently, it can be stated that 3D digitization with AI in mining infrastructure is in full development and requires deep attention.

Paradoxically, the main industrial activity in Peru is mining; however, this study demonstrates that there has been no advancement in 3D digitization with AI in journals indexed in Scopus. Therefore, it is recommended to consider the present research topic in the strategic funding areas of the competent entities, such as Prociencia in Peru.

As a final result of this article, to accelerate the adoption of 3D digitalization tools with AI in industrial mining infrastructures, it is recommended to consider articles [8, 9], 13-31] as a starting point. Additionally, it is suggested to use the models (PointNet++, ResPointNet++, and SE-PseudoGrid) and the dataset (Pipework, CLOI, and PSNet5) for the digitalization of industrial mining infrastructures with AI.

In consideration of future research activities, a number of potential pathways are suggested subsequent to the completion of this comprehensive review. Several of these are detailed in what follows:

1) The generation of synthetic data with enhanced geometric accuracy and a broader variety of materials, including concrete, wood, steel, and glass, aims to augment the capabilities of DL models.

2) Exploration of datasets pertinent to particular industrial sectors, incorporating data from infrastructure situated in diverse geographical regions while taking into account distinct cultural and construction elements.

3) Given that the findings associated with ISPS exhibit the lowest degree of accuracy, subsequent research endeavors may aim to enhance the precision of the following categories: Soil, Tall Vegetation, Buildings, Walls, Parking, Circulation Paths, Street Furniture, Automobiles, and Path. This improvement is intended to strengthen the DL models.