Pixel to 3D Voxel Reconstruction Using Octree-Based Network and Deep Learning

Many researchers have proposed methods for 3D reconstruction from single-view images. Shu et al. [4] introduced Pix2Vox, a framework capable of handling both single-view and multi-view 3D reconstruction. This approach incorporates an encoder-decoder architecture, a context-aware fusion module, and a refinement stage, achieving higher accuracy and consistency than many existing methods. Additionally, it offers faster inference times and exhibits strong generalization capabilities. The task of recovering 3D representations of objects from single-view or multi-view RGB images using deep neural networks has gained significant attention in recent research. Traditional approaches, such as 3D-R2N2, rely on recurrent neural networks (RNNs) to sequentially integrate feature maps extracted from input images. However, these methods often face challenges such as inconsistent results and memory limitations. Renat and Imangali [9] proposed an end-to-end network for efficient 3D model generation from a single image. This network consists of an encoder, a 2D–3D fusion module, and a decoder, which together produce detailed point clouds from single-object images and retrieve the most similar shapes from the ShapeNetCore dataset. The method demonstrates state-of-the-art performance when compared to volumetric and point-set generation techniques, particularly excelling in capturing intricate details. Additionally, it performs well in environments with complex backgrounds and across diverse viewpoints.

Bae et al. [10] proposed a GAN-based approach for predicting voxel models from a single view. Their method utilizes the alignment of 2D silhouettes and slices within a camera frustum to reconstruct voxel representations of scenes containing multiple object instances. This approach demonstrates excellent performance in reconstructing complex scenes with non-rigid and multi-object configurations.

Reconstructing 3D objects from multiple 2D images has been a common focus in computer vision research. However, Huang et al. [11] addressed a more challenging problem: estimating 3D locations and shapes of multiple objects from just a single 2D image. Unlike prior approaches that either predict a single 3D property or focus exclusively on individual objects, their method employs a comprehensive framework. This includes learning a 3D voxel grid from the input image, utilizing CenterNet-3D for keypoint detection, and applying a coarse-to-fine reconstruction module to achieve efficient and detailed 3D reconstructions. Their approach proved effective for both single- and multi-object scenarios. In another advancement, Yuan et al. [12] presented the Voxel Transformer (VoTr), a novel voxel-based Transformer backbone for 3D object detection using point clouds. By incorporating self-attention mechanisms, VoTr overcomes the limitations of conventional 3D convolutional backbones, such as constrained receptive fields, thereby enabling the modeling of long-range voxel relationships.

Some research studies are based on images captured with sensors such as Light Detection and Ranging (LiDAR) and depth cameras. Kuang et al. [13] proposed a Voxel-CRF model for 3D scene understanding by integrating a voxel-based representation with a conditional random field model to infer semantic labels and object instances in indoor scenes. Shi et al. [14] employed an automated approach for large-scale 3D scene reconstruction of urban areas using LiDAR sensors. They created a meshed representation of a 3.7 km-long area with high detail and no user intervention, and investigated the effects of sensor models on reconstruction quality.

Visualization of 3D objects can be represented in various forms, including volumetric representations using voxels (Liu et al. [7]), mesh representations (Tahir et al. [15]), and point clouds (Ji et al. [16]). Yasir and Ahn [17] investigated different approaches to 3D object shape representation, focusing on surface-based and volumetric methods, as well as viewer-centered versus object-centered reference frames in single-view 3D shape prediction. Their analysis revealed that surface-based techniques perform better than voxel-based representations for novel objects, whereas voxel representations are more effective for familiar objects.

Gbadago et al. [18] introduced a framework known as Hierarchical Surface Prediction (HSP), which utilizes convolutional neural networks (CNNs) to generate high-resolution voxel grids. Their findings indicated that HSP produces more accurate results compared to low-resolution predictions, regardless of the input format.

Traditional methods often use convolutional neural networks (CNNs) such as ResNet, VGG, or other backbone architectures that might not be as parameter-efficient as EfficientNet. Many state-of-the-art approaches rely on regular voxel grids or point-cloud representations. Voxel grids suffer from substantial memory consumption at higher resolutions, while point-cloud methods might lack the explicit structure required for detailed 3D reconstruction. Methods such as 3D-R2N2 or AtlasNet employ RNNs or mesh-based approaches that focus on sequential data processing or direct mesh generation but might not be as efficient in capturing high-resolution details as octree-based approaches.

This section shows the experimental setup, dataset details, testing and validation performed on our proposed architecture. For Experimentation 3 Benchmark datasets have been used. A description of these datasets is given below and then Different types of evaluation have been performed.

4.1 Datasets description

4.1.1 ShapeNet dataset

The experimentation for 3D voxel reconstruction has been done using the ShapeNetCore dataset [7]. ShapeNetCore is a large repository of 3D CAD models that are annotated with semantic information like consistent alignments, parts, and sizes. It is organized by the WordNet taxonomy and contains over 3 million models with more than 220,000 models classified into 3,135 categories. ShapeNetCore provides a web-based interface for data visualization and serves as a benchmark for computer graphics and vision research. Our proposed system achieves high accuracy and detail in reconstruction due to the octree's ability to focus on relevant areas, effectively handling the variety of object geometries. The SOTA methods Perform well in capturing general object shapes but often struggle with fine details due to voxel resolution limits or inefficient use of mesh structures.

4.1.2 Pix3D sun dataset

The Pix3D sun dataset consists of 395 3D models spanning nine item categories, with each model paired with real-world photographs taken in diverse settings. This dataset includes 10,069 image-shape pairs, all of which are annotated with precise 3D data, enabling accurate pixel-level alignment between object shapes and their silhouettes in the images. Our proposed system utilizes the octree structure to enhance reconstruction quality, excelling in capturing fine details, particularly in complex indoor scenes and furniture models. While state-of-the-art methods based on mesh and point-based approaches achieve good results, they often struggle to preserve intricate details or require post-processing steps to mitigate noise in the data.4.1.3 PASCAL3D+Silberman Dataset.

PASCAL3D+Silberman, is a novel and difficult dataset for 3D object detection and pose estimation. PASCAL3D+adds 3D annotations to the PASCAL VOC 2012's 12 rigid categories. Additionally, new photographs from ImageNet are added to each category. PASCAL3D+images are substantially more variable than previous 3D datasets, with more than 3,000 object occurrences per category on average. Our proposed system excels in reconstructing real-world objects with occlusions and complex surfaces thanks to the efficient feature extraction of EfficientNet and the octree’s adaptability. The other methods typically have issues with occluded or partially visible objects, leading to lower reconstruction accuracy when compared to the octree-based approach.

In this article, the ShapeNetCore dataset has been used for the training purposes of the proposed system. For testing and evaluation purposes Pix3D and PasCAL3D+datasets have been used.

4.2 Results

Experiment I: Loss functions (CD) and earth movers distance (EMD)

To evaluate the reconstructed 3D models, two loss functions were applied for point-to-point comparisons with the ground truth data. The first metric, CD, was computed using three benchmark datasets, with the corresponding results presented in Table 1. The second metric, Earth Mover’s Distance (EMD), was employed to measure the dissimilarity between the reconstructed models and the ground truth. EMD was calculated using the Sinkhorn and Wasserstein distances, with the results illustrated in Figure 8. The mathematical equations used for calculating CD and EMD are provided.

$\begin{gathered}C D(X, Y)=\frac{1}{|X|} *\left(\sum \min \left(| | x-y| |^2\right)\right)+\frac{1}{|Y|} *\left(\sum \min \left(||x-y||^2\right)\right)\end{gathered}$              (2)

In this equation, $X$ irepresents the predicted point set of the reconstructed 3D model, while denotes the ground truth point set. The cardinalities $|X|$ and $|Y|$ correspond to the number of points in $X$ and $Y$ respectively. The term $\|x-y\|^2$ refers to the squared Euclidean distance between a point $x$ in $X$ and a point $y$ in $Y$. This equation calculates the minimum distance between the points in the predicted set and those in the ground truth set.

$E M D(X, Y)=\min _{s_{g t} \rightarrow s_{p r}} \sum\|x-\emptyset(x)\|$          (3)

where, $x \in S_{p r}$ and $\emptyset(x) \in S_{p r} . \emptyset(x)$ is the closest point with the ground truth as shown in Table 1.

Table 1. CD on ShapeNetCore, Pascal 3D and Pix3D

(a)

(b)

(c)

Figure 8. Graphical representation of EMD values on (a) ShapeNetCore, (b) Pascal3D and (c) Pix3D

Experiment II: Evaluation matrix: accuracy, precision, recall and F1-score

When using machine learning methods and artificial intelligence methods precision, recall, F1-Score and accuracy are commonly used matrices for evaluation purposes. Precision is the fraction of the 3D reconstructions that are correct among all the reconstructed 3D models. In other words, it measures how many of the reconstructed models are relevant to the ground truth. In 3D reconstruction, precision is typically used to evaluate the accuracy of shape and pose estimation. Recall is the fraction of the ground truth models that are correctly reconstructed among all the ground truth models. It measures how many of the relevant models were correctly identified by the algorithm. In 3D reconstruction, recall is often used to evaluate the completeness of the reconstructed models [24]. The F1-score is a performance metric that integrates precision and recall into a single value by calculating their harmonic mean. Precision reflects the proportion of accurately detected objects out of all detected objects, whereas recall represents the proportion of correctly identified objects relative to the total number of ground truth objects. The F1-score ranges from 0 to 1, with higher values signifying improved performance as shown in Tables 2 and 3. Accuracy is the overall correctness of the reconstructed models. It measures how many of the reconstructed models are both relevant and correctly identified. In 3D reconstruction, accuracy is a composite metric that considers both precision and recall. The following equations are used to calculate precision and recall using max distance.

$D(a, b)=\sqrt{\sum_i^{G t_n}\left(a_i-b_i\right)^2}$               (4)

$Max$_$Distance$ $={Max}\left(D_1(g t, p t), D_2(p t, g t)\right)$             (5)

$Precision$ $=D\left(g t, D_1\right) /$ $Max$_$Distance$             (6)

$Recall$ $=D\left(p t, D_2\right) /$ $Max$_$Distance$                 (7)

where, $D(a, b)$ is distance between two different point cloud models. Max Distance is the maximum distance computed between the $g t$ ground truth and the pt predicted point cloud. Precision and Recall are calculated using distances and Max Distance results (See Table 4). To calculate the F1-Score the harmonic mean of precision and recall is used in the following equation:

$F 1$ $Score$ $=2 *$ $Recall$ $* \frac{ { Precision }}{ { Recall }+ { Precision }}$               (8)

where, F1Score is the precision and recall combined matric to measure the accuracy of the model? To calculate accuracy, we used different loss functions to calculate overall loss and then this overall loss has been used to compute accuracy.

$L_{{Edge }}\left(P_C\right.$, $Target$$\left._{{Length }}\right)=\sum(i, j)\left\|P_i-P_j\right\|$                 (9)

$L_{{Normal }\left(P_C, { Target }_{{Length }}\right)}=\sum(i)\left(| | P_i-P_j| |\right)^2$             (10)

$L_{ {Laplacian }}\left(P_C\right.$, $Target$$\left._{{Length }}\right)=\sum(i)\left(\left\|\Delta^2 P_i\right\|\right)^2$                  (11)

where, $P_C$ is predicted point cloud and $Target$${ }_{ {Length }}$. The number of points in the point cloud in this article is set to 1024 . $P_i$ is the point in the predicted point set and $P_j$ is the closest point in the ground truth point set. $\Delta^2$ is the Laplacian operator that calculates the 2nd derivative of the position of the point (Table 5).

$\begin{aligned} \text { Overall loss }= & \lambda_E * L_{{Edge }}+\lambda_N * L_{ {Normal }}+\lambda_L * L_{ {Laplacian }}\end{aligned}$             (12)

where, Overall loss is the loss function calculated using the combined effect of different weighted loss functions. $\lambda_E, \lambda_N$ and $\lambda_L$ is the weighted value used at the end to control the importance of edge, normal and Laplacian loss functions respectively in the overall loss function.

$\operatorname{Accuracy}(X, Y)=1-$ overall loss                   (13)

where, Accuracy is the overall evaluation of the proposed system. Overall loss has been computed using a combination of different types of loss functions with specific weights.

Table 2. Precision, Recall, F1-Score and Accuracy on 3D using ShapeNetCore

Table 3. Precision, Recall, F1-Score and Accuracy on 3D using Pascal3D

Table 4. Precision, Recall, F1-Score and Accuracy on 3D using Pix3D

Table 5. IoU on ShapeNetCore, Pascal3D and Pix3D

Experiment III: Evaluation object overlappling: Intersection over Union (IoU).

This system has been also evaluated using a comparison of predicted and ground truth overlapping using Kato, Intersection over Union (IoU) calculation. This method is best for the evaluation of a volumetric 3D reconstruction system (See Table 6). The higher the value of IoU the better the reconstruction will be.

$I o U=\frac{O b j_{g t} \cap O b j_{p r}}{O b j_{g t} \cup O b j_{p r}}=\frac{\sum\left\{I\left(O b j_{p r}>\epsilon\right) * I\left(O b j_{g t}\right)\right\}}{\sum\left\{I\left(I\left(O b j_{p r}>\epsilon\right)+I\left(O b j_{g t}\right)\right)\right\}}$             (14)

where, $I$ is the indicator function that shows the ith voxel in the volumetric 3D shape. $\epsilon$ is the threshold where the value has been computed.

Experiment IV: Comparison with the State-of-the-art systems

Comparing the performance of 3D reconstruction methods with the state of the art (SOTA) is an important task in evaluating the effectiveness of these methods. The state of the art refers to the best-known method or model that achieves the highest performance on a given task. In 3D reconstruction, the state of the art can be determined by comparing the performance of different methods on standard benchmark datasets ShapeNetCore, Pascal3D and Pix3D for object reconstruction. To compare the performance various metrics have been compared like CD, EMD and IoU.

Table 6. Comparison of IoU with different state-of-the-art methods