Deep Learning for Marine Vehicles Parking Availability: A ResNet50-Based Deep Feature Engineering Model

Table 1 section showcases existing ship detection models, which mainly focus on general ship classification using CNNs and object detection approaches like Faster R-CNN and YOLO. Although these models perform well in identifying various ship types, they do not specifically address the challenge of parking availability detection. In addition, IoT-based solutions, though effective, are costly and complex, making them less practical for widespread use.

The introduced ResNet50-based DFE model overcomes these limitations by specifically targeting parking availability detection. By using the semi-overlapping patch splitting technique, it captures both local and global features that are important for identifying parking spaces.

The introduced approach provides a novel and effective solution to the specific challenge of parking availability detection.

Table 1 has summarized a wide range of methods that combine marine vessels with ML. Table 1 has presented the characteristics of the datasets utilized by these methods, as well as the classification/detection performances achieved. Table 1 demonstrates the diversity of current research and the performance of various approaches in the field of ship detection and classification. Most of the studies listed achieved high accuracy rates by utilizing various DL architectures. For example, the study by Kim et al. [15] employed a faster region-based CNN to attain a mean average precision of 93.92%. Another study by Escorcia-Gutierrez et al. [16], which used a mask regional CNN and Weighted extreme learning machine, achieved an accuracy of 98.75%. These studies illustrate the effectiveness of DL techniques in applications within the maritime domain. However, a research by Zhang et al. [22], which utilized YOLOv5, listed a relatively lower performance with an accuracy of 71.60%. This indicates significant performance disparities across research due to the variety of methods and datasets used. This table also shows that some studies applied data augmentation, suggesting that enhancing the model's generalization capability can be achieved through this technique. Overall, Table 1 demonstrates the successful application of ML and DL techniques in ship detection and classification. The results of these studies highlight the potential of these technologies for significant applications in the maritime industry, including determining the availability of parking spaces. These contributions provide insights that could serve as a foundation for future research.

1.2 Literature gaps

The following gaps have been identified according to the literature review:

· There is a notable scarcity of research papers specifically focused on ship parking detection. Most existing studies rely on IoT-based systems, which are complex and expensive to implement, limiting their practicality in large-scale marine environments.

· Current methods emphasize IoT systems, but there is a pressing need for more mobile and cost-effective solutions to detect parking availability for ships. Such systems could provide greater flexibility and scalability in diverse marine environments.

· The lack of open-access datasets for ship parking detection presents a significant barrier to developing and benchmarking new models, particularly those using advanced computer vision and DL techniques. This hinders the progress of next-generation solutions in the field.

1.3 Motivation and study outline

Ship parking detection is a critical task in marine environments such as ports, marinas, and fishing ports to ensure smooth operations and minimize congestion. However, existing solutions in this area are mostly based on IoT systems, which are often complex and costly to implement. These systems are typically expensive. Moreover, there is a significant lack of research specifically focused on ship parking detection, which further limits the development of innovative, automated solutions. Additionally, the scarcity of open-access datasets poses a major obstacle to the advancement of computer vision and DL models that could address these challenges more efficiently. Given the urgent need for cost-effective alternatives, this research aims to fill the gaps in the literature by proposing a novel DL-based approach for ship parking detection.

The literature review highlights a scarcity of automated systems for detecting ship parking availability despite the potential of DL models to address various computer vision challenges. Our primary goal is to address the parking availability issue in marines, ports, and fisher shelters, among other areas. To this end, we have compiled a new dataset using an unmanned aerial vehicle (UAV) divided into two categories: 'full' and ' parking available.'

The proposed ResNet50-based DFE model is inspired by a popular image transformers (ViT) [29], which demonstrate the effectiveness of fixed-size patch division for extracting features from local areas. However, the patch-based feature extraction approach is a complex approach. For example, ViT uses patch sizes such as 14 × 14, 16 × 16 and 32 × 32, and using these sizes, 256, 196 and 49 fixed-size patches are obtained from a 224 × 224 image, respectively. To obtain a more linear method and use the effectiveness of fixed-size patch division, we proposed a new patch division approach, the semi-overlapped patch division technique. In this work, we used 112 × 112 patches with a stride of 56 and the created 9 patches have been employed to extract local features. Additionally, we used the entire image to extract global features. At this stage, we used trained ResNet50 as feature extractor. We generated the ultimate feature vector by combining the obtained local and global features. Then, we employed iterative neighborhood component analysis (INCA) [30] to choose the most meaningful/valuable features to solve the problem in the ultimate feature vector.

Classification results have been obtained using a shallow classifier, specifically a support vector machine (SVM) [31]. Additionally, a Bayesian optimizer [32] has been employed to tune the optimal hyperparameters for the utilized SVM classifier to attain higher classification performance.

1.4 Novelties and contributions

We have presented a parking availability detection model for marine vehicles and as our surveyed literature, we are the first team is to present a solving for parking availability detection for marine vessels using computer vision.

Novelties:

· We have gathered a novel image dataset using an UAV to create a testbed for parking availability detection.

· The semi-overlapped patch division technique has been presented to extract local features from the local areas using less complexity in this research.

Contributions:

· Addressing the parking availability issue for ships is a prevalent and economically significant problem, especially considering its implications for port congestion. By compiling a new dataset from fisher shelters and marines specifically for parking availability detection, we contribute to refining the methodology for parking availability analysis.

· We propose a novel DFE model inspired by ViT. This model employs semi-overlapped blocks to extract local and global deep features effectively. The DFE model has demonstrated a test classification accuracy of 97.27% on the marine image dataset, underscoring its capability to detect parking availability for ships accurately. This result shows that DL can be useful for real-world problems in marine logistics. It also supports the use of ML for detecting ship parking availability.

2.png

Figure 2. The general block diagram of the recommended approach

The dataset was collected using a UAV to capture images of various ship parking areas, and labeling was performed after the dataset was collected. ResNet50, a widely used CNN, was selected as the base feature extractor for the approach. Leveraging the pre-trained version of ResNet50 allowed us to benefit from transfer learning, reduce the need for large amounts of training data, and increase model efficiency. The ResNet50 architecture includes residual connections to avoid the vanishing gradient problem, which enables the approach to perform well even with deeper networks.

A novel semi-overlapping patch splitting strategy was used to improve feature extraction. The images were split into 112 × 112 × 3 patches with a stride of 56 steps. After patch splitting, both local features from the patches and global features from the entire image were extracted using the GAP (Global Average Pooling) layer of the pre-trained ResNet50 approach. This step reduced the complexity of the feature maps while preserving important information. In total, nine feature vectors were created from the patches and one global feature vector from the entire image, and these were combined to form a comprehensive feature representation of the image.

INCA was then implemented to the combined feature vector to choose the most informative features. The chosen features were passed through the SVM. Bayesian optimization was used to fine-tune the hyperparameters of the SVM, improving the classification accuracy by minimizing the misclassification rate.

In this research, the used CNN is ResNet50. Therefore, the core architecture of ResNet50, along with the mathematical formulation of the recommended approach, is detailed in Figure 3.

We present a schematic representation of the main block of ResNet50 in Figure 3. The mathematical framework underpinning ResNet50 is detailed in the following sections.

$F:(256,512,1024,2048), R:(3,4,6,3)$       (1)

Herein, F: number of filters and R: number of repetitions.

3.png

Figure 3. The block design of the ResNet50

Utilizing ResNet50, we trained the collected image dataset and created a pretrained ResNet50 CNN alongside a DFE model. This DFE model comprises three main phases:

Feature extraction: In this phase, semi-overlapped patch division and the GAP layer of the pretrained ResNet50 were used to generate local and global features.

Feature selection: The INCA method was employed to obtain the most informative results.

Classification: The SVM classifier was implemented in this phase. Furthermore, a Bayesian optimizer was applied to the SVM classifier to determine the optimal hyperparameters.

Through these phases, we implemented the introduced ResNet50-based DFE approach. A graphical overview of the presented ResNet50-based approach is outlined in Figure 4.

The details of the introduced ResNet50-based DFE approach are given as below step by step.

Step 1: Train the ResNet50 by implementing the training image dataset.

Step 2: Employ semi-overlapped patch division to each test image.

$\begin{gathered}P_k=\operatorname{Im}(i: i+111, j: j+11,:), \\ i \in\{1,57, \ldots, 113\}, \\ k \in\{1,2, \ldots, 9\}, j \in\{1,57, \ldots, 113\}\end{gathered}$                  (2)

Herein, P: fixed-size patch, Im: image. Herein, the size of the used patch is equal to 112 × 112 × 3 and stride is 56. In this step, we have generated nine fixed-size patches.

The semi-overlapping patch splitting technique was used to overcome the limitations of traditional fixed-size patch splitting methods, such as those used in Vision Transformers (ViT). Traditional patch splitting usually relies on non-overlapping patches, which can result in the loss of important contextual information between adjacent patches. Additionally, to minimize this issue, the complexity of the model is often increased by using too many patches. In contrast, the semi-overlapping approach allows partial overlap between patches, preserving more local information, ensuring continuity between regions, and enabling more meaningful information to be extracted with larger patch sizes (fewer patches). This approach is especially important in marine parking spot detection, where small details such as ship edges or parking boundaries need to be captured accurately.

Step 3: Generate global features from all images by implementing the GAP layer of the pretrained ResNet50.

$f_1=\operatorname{ResNet} 50(\operatorname{Im}, G A P)$        (3)

Herein, $f_1$ defines the created feature vector as having a length of 2048, and this feature vector represents the global feature vector.

Step 4: Extract local features from the generated patches implementing the GAP layer of the pretrained ResNet50.

$f_{k+1}=\operatorname{ResNet} 50\left(P_k, G A P\right)$       (4)

where, $f_{(2-10)}$: the local features and the length of each of them is 2048.

Step 5: Concatenate the derived feature vectors.

$F=\operatorname{merge}\left(f_1,, f_2, \ldots, f_{10}\right)$         (5)

where, F: the derived feature vector with a length of 20,480 (= 2048 × 10).

4.png

Figure 4. The introduced DFE approach based on the ResNet50

Step 6: Implement the INCA to the merged feature vector.

$i d x=N C A(F, y)$       (6)

$\begin{gathered}S^{r-99}(d, i)=F(d, i d x(i)), i \in\{1,2, \ldots, r\}, \\ r \in\{100,101, \ldots, 1000\}\end{gathered}$        (7)

where, F: the derived feature vector with a length of 20,480 (= 2048 × 10), idx: the qualified indices, y: actual output, S: selected feature vector, and r: range of the iteration. Above (see Eqs. (6) and (7)), the iterative feature selection process has mathematically been explained, and INCA chooses 901 (=1000-100+1) feature vectors. Among these 901 feature vectors, the best one has been selected using a greedy algorithm according to the loss values. We have used the SVM classifier to compute loss values. The best feature vector selection process is specified below.

$loss(r-99)=SVM\left(S^{r-99}, y\right)$      (8)

$[mini, i n x]=\min\,\,(loss)$      (9)

$SelFeat=S^{inx}$     (10)

Herein, loss: misclassification rate, mini: minimum loss rate, inx: index of the minimum loss rate, and SelFeat: the ultimate chosen feature vector by INCA. Herein, the length of the optimal selected feature vector is 111.

Step 7: Optimize the hyperparameters of the SVM classifier by implementing a Bayesian optimizer.

Step 8: Classify the choose features by deploying the fine-tuned SVM classifier.

$Pred= SVM\,\,(SelFeat, y)$       (11)

where, Pred: predicted vectors and we have used this predicted vector to compute the classification results.

In this study, a novel DFE model based on ResNet50 was introduced, and when implemented to the collected image dataset, it achieved a test accuracy of over 97% (97.27% and see Table 4 for more details) utilizing this ResNet50-based DFE model.

To design the proposed model, we tested the well-known CNNs and these are (1) SqueezeNet [33], (2) EfficientNetB0 [34], (3) DenseNet201 [35], (4) DarkNet53 [36], (5) AlexNet [37], (6) VGG19 [38], (7) GoogLeNet [39], (8) ConvNeXt [40] (a CNN designed by us), (9) MobileNetV2 [41], and (10) ResNet50 [42]. These CNNs were trained using our collected image dataset to obtain validation accuracies. Herein, we selected the training and validation separation ration as 70:30. Herein, we aimed to use the best CNN to extract features. Therefore, we used a greedy-based the most suitable CNN selection strategy and the computed validation results of these 10 CNN architectures have been illustrated in Figure 8.

8.png

Figure 8. Comparison of the CNNs

Figure 8 has demonstrated that the best-performing CNN among these 10 CNNs is the 10^th CNN, which is ResNet50, since this CNN attained a 94.23% validation accuracy. The second CNN, according to validation accuracy, is the 3^rd CNN (DenseNet201), which reached a 93.22% validation accuracy. Therefore, we selected ResNet50 (ResNet50 has been employed as a feature extractor in our DFE) to create our presented DFE model.

By using ResNet50 and a semi-overlapped patch division technique, we designed the feature extraction method for the suggested DFE model.

To choose the best features from the generated feature vector, INCA (it is an effective iterative feature selector) feature selector has been used.

The last phase of the presented approach is the classification. To choose the most effective shallow classifier for the recommended DFE model, we tested eight shallow classifiers and these are: Decision Tree (DT) [43], Linear Discriminant Analysis (LDA) [44], Quadratic Discriminant Analysis (QDA) [45], Naïve Bayes (NB) [46], SVM [31], k-Nearest Neighbors (kNN) [47], Bagged Tree (BT) [48], and Multilayer Perceptron (MLP) [49]. We selected the best accurate classifier for our proposed DFE model and the computed classification accuracies by using these classifiers have been depicted in Figure 9.

9.png

Figure 9. The classification accuracies of the employed classifiers

According to Figure 9, the SVM (accuracy: 96.83%) is the most accurate classifier for the parking space availability detection problem. To set the parameters of the SVM, we applied Bayesian optimization. With the fine-tuning obtained through the Bayesian optimizer, the test classification accuracy increased from 96.83% to 97.27%.

Above, we have summarized the story of the presented DFE model and highlighted that a cognitive approximation was utilized to create the proposed model. In the second step of the discussion, we explained the explainable results of our model. To present XAI results, we used feature vector analysis and activations.

The presented DFE model has used a semi-overlapped patch division model, and nine patches have been generated by deploying the presented patch division technique. By deploying patches, the local features have been extracted, and the whole image has been used to extract global features. To evaluate the effectiveness of each created patch and the local and global features, we have employed the findings of the INCA feature selector. The INCA has selected the best 111 features. Here, we have analyzed the source of these features by using a sample image, and this analysis has been illustrated in Figure 10.

10.png

Figure 10. The number of the chosen features according to the employed input

Figure 10 has illustrated that 50 out of the selected 111 features were generated from the whole image. Therefore, these 50 features are termed global features, and the remaining 61 features were generated from the patches. In this aspect, these features are considered local features. The most valuable patches are the 4^th and 5^th, since 20 features were generated using the 4^th patch, and 20 features were selected from those generated by the 5^th patch.

Furthermore, Gradient-weighted Class Activation Mapping (Grad-CAM) has been employed to provide more explainable results about the proposed model, and we have used sample images to give visually explainable results. These visualizations are shown in Figure 11. In this figure, we have demonstrated heatmaps of the sample images and selected sample images from both classes. The used Grad-CAM model shows how ResNet50 focuses on the region of interest (ROI).

11a.png

(a) Parking available

11b.png

(b) Full

Figure 11. Heatmap images by generating Grad-CAM

Figure 11 illustrates that the proposed ResNet50-based model can easily focus on the ROI.

The findings obtained using Grad-CAM allow the model to highlight specific areas of interest, such as parking areas or ship locations, during decision-making. This capability is critical for practical deployment in the maritime industry, as it helps port managers trust the model and easily integrate it into their workflows.

Grad-CAM also ensures the reliability of the model by verifying that the model focuses on relevant features, avoiding noise or irrelevant parts of the image. It helps troubleshoot and improve the model, identify areas for improvement, and reduce misclassifications. The explainability offered by Grad-CAM increases user confidence, making it more feasible to deploy in real-world applications such as port congestion management and ship tracking. Grad-CAM findings improve the transparency and robustness of the model, facilitating deployment in dynamic environments and enabling data-driven, interpretable decisions.

These results (see Figure 9 and Figure 10) clearly highlight the explainability of the presented ResNet50-based model.

By using these results and findings, the findings, advantages, limitations and future directions of the presented research are discussed below.

Findings:

1. The best CNN among the tested CNNs is ResNet50 (see Figure 7). Therefore, we have used it as a feature extractor in the proposed model).
2. SVM is the best classifier among the tested classifiers. Thus, we tuned the parameters of this classifier.
3. The length of the chosen feature vector is 111, and this feature vector contains both global and local features. According to the feature analysis, 50 of them are global, while 61 are local features.
4. Figure 10 demonstrates that the presented model can focus on the ROI.

Advantages:

1. We have introduced a novel solution for detecting ship parking space availability, employing computer vision as our proposed solution.
2. This study unveils a new patch division model termed semi-overlapped patch division.
3. A cognitive model has been introduced to attain high classification performance.
4. The proposed DFE approach attained over 97% test accuracy on the collected image dataset for parking availability detection.
5. In this research, we have provided both classification results and explainable outcomes.

Limitations:

1. The dataset includes images from over 200 fisher shelters in Turkey and focuses on parking space availability. However, the dataset may have regional or seasonal biases that could affect generalizability.
2. The dataset may also contain class imbalances, potentially skewing model performance in underrepresented scenarios.
3. UAV-collected data could be influenced by external factors such as weather or lighting conditions, impacting model accuracy.

Future directions:

1. We will collect a larger, more diverse dataset using UAVs and internet images. This will address biases related to region, season, and lighting for better generalization.
2. We plan to extend the model to detect ship congestion, dock occupancy, and vessel types, beyond parking availability.
3. Grad-CAM will be used to improve model explainability, providing better visual insights into predictions.
4. Future work will explore EfficientNet and lightweight custom CNN models to boost performance and reduce computational costs.
5. We will partner with maritime authorities and stakeholders to test the model in real-world environments like ports and fisher shelters.
6. The model will be adapted for parking and occupancy detection in other areas like urban car parks, airports, and logistics hubs.
7. A real-time monitoring system using UAV feeds and AI-based alerts will be developed for ongoing parking management.
8. We plan to release part of our dataset publicly to promote research and collaboration in maritime computer vision.
9. Future studies will combine radar, thermal imaging, and visual data to enhance detection accuracy in challenging weather or lighting conditions.
10. We plan to introduce an explainable DFE model employing Grad-CAM.