Diagnosis of Purtscher Retinopathy from Spectral-Domain OCT Images Using Panoptic Segmentation Technique

Purtscher's and Purtscher-like retinopathy have an estimated yearly occurrence of 0.24 cases/million, and its identification is mostly symptomatic in the absence of a defined etiology [9]. Although the pathogenesis of Purtscher-like retinopathy is unknown, numerous ideas have been offered based on the fundamental systemic condition, although it is assumed to be induced by embolization that causes arteriolar blockage and ischemia [10]. The lesions indicate the existence of the disease, but they appear in only half of the patients. Individuals must be recognized from cotton-wool patches, which partially cover vessels and lack appropriate boundaries. PR patients frequently describe sudden, painless vision loss 2 to 48 hours after the underlying illness. Findings are frequently apparent bilaterally also be seen unilaterally [11-13].

Xiao et al. [14] developed multimodal imaging in PR, which revealed capillary nonperfusion, quantified the foveal avascular zone, and found an ellipsoid zone abnormality in acute PR. Optic disc enlargement, retinal edema, and pseudo-cherry spots may also appear in the early stages, yet they are less prevalent. In the majority of circumstances, the illness is bilateral [15, 16]. Four young women experienced severe bilateral sight loss due to numerous retinal arteriolar occlusions within 24 hours of delivery [17]. Preeclampsia complicated labor in two cases, necessitating a cesarean surgery. Rothbächer et al. [18] provided insight into PR after ENT surgery. The study lacks advanced imaging techniques like OCT angiography or fluorescein angiography that could have offered more detailed visualization of the retinal vasculature. These factors reduce the strength of the conclusions and highlight the need for further investigation.

Ophthalmoscopy and fluorescein angiography revealed signs of Purtscher's Retinopathy-like ischemia retinal whitening in several superficial peripapillary and macular regions. The white spots in the eyes of all four patients had disappeared after eight weeks, and three of them had much better visual acuity. The pathogenesis of this illness is unknown. It may include arteriolar blockage caused by complement-induced leukoemboli generated after parturition [19, 20].

PR is a painless visual activity problem and causes visual disturbance. To optimize the Panoptic FPN for detecting PR from SD-OCT images, several key modifications has been made to suit the specific characteristics and challenges of this medical imaging task. These changes primarily focused on the network’s architecture, preprocessing techniques, and fine-tuning of hyperparameters.

Figure 2 shows the structure of the proposed technique. Instance and semantic segmentation outputs are concurrently predicted by the proposed CNN. ResNet-50 serves as the foundation of the proposed network and was selected due to its capacity to extract reliable features from intricate medical images, including SD-OCT scans. By addressing the vanishing gradient issue, ResNet-50's residual connections make deep network training more effective. ResNet-50 balances model performance and computational efficiency in comparison to other possible backbones like ResNet-101 or EfficientNet. ResNet-101 has a deeper architecture that may be able to capture finer-grained information, but because of its greater complexity, training durations, and processing costs may rise. However, EfficientNet needs a lot of fine-tuning for medical image applications, even though it could provide better accuracy. ResNet-50 was selected due to its shown efficacy in medical imaging applications and its ability to handle large image datasets with manageable processing requirements.

2.png

Figure 2. Architecture of the proposed system

A ResNet-50 feature extractor constitutes the foundation for the two separate semantic and instance segmentation sections of the network. First, a Region Proposal Network (RPN) is used to generate region proposals for potential objects in the images. Following their retrieval from the feature map, the final layers of ResNet-50 are applied to the features that correspond to these predictions. The produced feature map is first enlarged by the semantic segmentation branch using a Pyramid Pooling Module, and then the source image's dimensions are adjusted using hybrid upsampling. Once a deconvolution operation has been completed, the predictions are bilinearly resized by this hybrid upsampling technique to suit the dimensions of the input image. The result of this branch is a pixel map, where each element denotes the anticipated class name for that specific pixel in the input image.

Lastly, Panoptic output image is input using these attributes. Extractor of features (ResNet-50) CNN's Semantic Segmentation CNN Instance Segmentation. This branch produces a set of pixel clusters with class labels that should match the locations of different items in the image following non-maximum suppression. Post-processing adjustments are used to these pixel clusters to produce per-object normalized instance masks with the dimensions of the input.

The Panoptic FPN was effectively customized for PR detection by:

• Choosing ResNet-50 as the backbone for efficient feature extraction.
• Applying mean filtering and CLAHE for preprocessing to reduce noise and enhance image contrast.
• Designing a dual-branch network for semantic and instance segmentation.
• Using hybrid upsampling to ensure accurate pixel-level labeling.
• Fine-tuning hyperparameters and loss functions to optimize segmentation performance.

This customized method addresses the unique difficulties in identifying and classifying lesions linked to PR in SD-OCT images by utilizing the advantages of instance and semantic segmentation as well as the multi-scale feature extraction capabilities of the FPN architecture. Furthermore, a comparison of the segmentation findings with ground truth annotations from experienced ophthalmologists is necessary to confirm the results. The model undergoes iterative refinement and fine-tuning based on feedback from the validation process.

4.1 Preprocessing

To ensure the network’s robustness in handling noise and variations in SD-OCT images, two preprocessing techniques were employed: Mean Filtering and Contrast Limited Adaptive Histogram Equalization (CLAHE).

Mean Filtering reduces noise in the SD-OCT images. The mean filter smoothes out noise while maintaining edges by averaging pixel values within a nearby region. This is crucial in medical imaging, where fine details are often vital for diagnosis but can be overshadowed by noise.

Consider, an input OCT image I and a mean filter of size $k \times k$, the output image I′ is obtained by convolving I with the mean filter to reduce the noise. For a pixel located at $(i, j)$, the mean-filtered pixel value $I^{\prime}(i, j)$, is expressed as follows.

$I^{\prime}(i, j)=\frac{1}{k^2} \sum_{m=-\frac{k-1}{2}}^{\frac{k-1}{2}} \sum_{n=-\frac{k-1}{2}}^{\frac{k-1}{2}} I(i+m, j+n)$          (1)

where, the input image is represented by I. I' is the output (mean-filtered) image; The size of the kernel (filter) is denoted by k. The indices that run across the kernel are m and n, whereas (i,j) denotes the orientation of the pixel in the image.

The mean filter kernel K of size $k \times k$ is defined as follows. 

$K=\frac{1}{k^2}\left[\begin{array}{cccc}1 & 1 & \ldots & 1 \\ 1 & 1 & \ldots & 1 \\ . . & . . & \ldots & . . \\ 1 & 1 & \ldots & 1\end{array}\right]$           (2)

This kernel has each element equal to $\frac{1}{k^2}$.

The mathematical model states that the new value of pixel $I^\prime(i,j)$ is the average of the values of the pixels over time $k\times k$ neighborhood cantered around (i,j).

Normalizing the intensity values of the mean-filtered image involves rescaling the pixel intensities to a standard range, such as [0, 1] or [0, 255] through the following equation. 

$I^{\prime}{ }_{ {norm }}(i, j)=\frac{I^{\prime}(i, j)-\min \left(I^{\prime}\right)}{\max \left(I^{\prime}\right)-\min \left(\left(I^{\prime}\right)\right.}$           (3)

where, $\min \left(I^{\prime}\right)$ and $\max \left(I^{\prime}\right)$ are the lowest and highest pixel values in the mean-filtered image.

CLAHE highlights fine-grained details of the lesions that are critical for accurate segmentation. CLAHE adapts the contrast enhancement locally, ensuring that regions with varying intensities are adjusted without over-amplifying noise. This phase was essential for increasing the visibility of lesions that are important markers of PR, such as hemorrhages, exudates, and cotton wool patches. CLAHE is used to increase the contrast of the image.

$I_{ {enhanced }}^{\prime}=C L A H E\left(I_{ {norm }}^{\prime}\right)$           (4)

4.2 Architecture of panoptic FPN with semantic and instance segmentation

The network was designed with two main branches: semantic segmentation and instance segmentation, to handle lesion types (semantic) and the precise localization of each lesion (instance). This dual approach allows the model to effectively differentiate between multiple lesions of the same type and improve the granularity of the segmentation.

Semantic Segmentation was tackled using a Pyramid Pooling Module within the FPN architecture. This module collects contextual information from different scales, which is critical for segmenting various types of lesions at different sizes and levels of detail. The up-sampling process involved bilinear interpolation, Group Normalization, ReLU activations, and 3×3 convolutions to ensure the feature maps were appropriately scaled before pixel-wise classification was performed.

Using the Region Proposal Network (RPN), which creates possible areas of interest (ROIs) for every lesion, instance segmentation was accomplished. Following the identification of ROIs, the object localization was improved and redundant predictions were eliminated using bounding box regression and Non-Maximum Suppression (NMS). This process was essential for accurately counting the number of lesions and localizing them with high precision.

4.2.1 Panoptic segmentation

Semantic and instance segmentation are the two components of panoptic segmentation. The instance segmentation consists of four sub-parts class, label, bounding box, and binary mask. The object detection accurately predicts hemorrhages and the corresponding bounding boxes. The prediction processes do the predictions based on the probability score. This score is the probability of all classes. The binary mask is labeled with 1 or 0 as per the predicted instance. The coordinate system produces the object bounding box for each prediction. The x and y coordinate system has a bounding box to extract the object within the anchor points with its length and height.

The classification results obtained from the probability output scores are combined with the regression results of bounding box outputs to predict the final output. Finding instances and classifying the contents of an image are the goals of semantic segmentation. Panoptic segmentation classifies all the pixels in the image which belong to a class label. Also, it identifies the instance of the class. The panoptic segmentation technique consists of the following phases.

1. Prepare an image by resizing, normalizing, and removing noise.
2. Extract features from the image that can be used to classify pixels into different semantic classes and instance IDs.
3. Assign a distinct instance ID to every object in the image and categorize every pixel into a useful class.
4. To produce a panoptic segmentation, integrate the outcomes of the instance and semantic segmentation processes.

4.2.2 Semantic segmentation

The first step is to perform semantic segmentation to identify different types of lesions present in retinal images. This entails assigning a class to every pixel in the image that corresponds to a particular kind of lesion, such as microaneurysms, hemorrhages, exudates, or cotton wool patches. The semantic segmentation model should be trained on annotated retinal images with pixel-level labels for each type of lesion.

Let S To do this, each pixel in the image must be given a class that represents a certain type of lesion, such as microaneurysms, hemorrhages, exudates, or cotton wool patches. It is represented as follows.

$S=f_{ {semantic }}\left(I ; \Theta_{ {semantic }}\right)$           (5)

where, the function denoting the semantic segmentation model is called $f_{semantic}$, the input image is called I, and the semantic segmentation model's parameters are called $\Theta_{semantic}$. Three phases of upsampling are performed, beginning at the deepest FPN level (at the 1/32 size). Figure 3 shows how to make a 1/4 scale feature map. Each upsampling stage comprises of two bilinear upsampling, Group Norm, ReLU, and 3×3 convolution. For the 1/16, 1/8, and 1/4 FPN scales, the same method is repeated.

3.png

Figure 3. Semantic segmentation process

A collection of feature maps with the same scale of 1/4 constitutes the result, and these feature maps are then added one by one. Softmax, a final 1×1 convolution, and four bilinear upsampling are used to construct the per-pixel class labels at the original image resolution.

4.2.3 Instance segmentation

Once the types of lesions are identified, instance segmentation can be applied to distinguish individual instances of each lesion type. Instance segmentation assigns a unique label to each instance of a lesion, enabling precise localization and counting of lesions within the retinal image. The instance segmentation model should be trained on annotated retinal images with instance-level labels for each lesion type.

Let $I$ denote the instance segmentation output, which is a set of instance masks indicating the pixel-wise segmentation of each object instance. It is represented as follows:

$I=f_{ {instance }}\left(I ; \Theta_{{instance }}\right)$           (6)  

where, $\Theta_ {instance }$ are the parameters of the instance segmentation model, and $f_ {instance }$ is the function representing the instance segmentation model.

4.2.4 Merging using heuristics

Panoptic segmentation is a technique that combines the output of instance and semantic segmentation into a single segmentation map that includes both "stuff" and "things." This is represented as follows:

$P=\{(S, I)\}$           (7)  

The panoptic segmentation output is denoted as P, and each tuple $(S, I)$ represents the semantic and instance segmentation results of the image.

To create a thorough segmentation map for retinopathy, the results of instance and semantic segmentation are integrated. To do this, the instance segmentation masks which offer comprehensive details on specific lesion instances are combined with the semantic segmentation map, which offers a high-level summary of lesion varieties. The integration process may involve techniques such as overlaying instance segmentation masks onto the semantic segmentation map or using conditional probability distributions to refine segmentation boundaries.

4.3 Panoptic FPN

The model uses an encoder-decoder architecture with FPN and a novel merging algorithm. The following steps are performed as follows:

1. The input image is first passed through an encoder-decoder architecture. The encoder extracts feature from the image at different scales, while the decoder reconstructs the image.
2. The encoder extracts features, which are subsequently transmitted via the FPN. Semantic and instance segmentation benefit from the model's ability to extract features at different sizes.
3. The features extracted by FPN are then passed through the RPN. It identifies areas of the image that are likely to have things in them.
4. The regions proposed by RPN are then passed through a mask head. Mask head predicts masks for the objects that are proposed.
5. The mask head predicts masks, which are used with semantic segmentation predictions to generate panoptical segmentation.

Through lateral connections and a top-down methodology, it combines low-resolution, semantically strong features with high-resolution, semantically weak data, as shown in Figure 4. This feature pyramid has extensive semantics at all levels and can be quickly built from a single input image scale without reducing textual strength, performance, or space.

4.png

Figure 4. FPN

The Panoptic FPN's bottom-up route is in charge of constructing a feature pyramid out of the feature maps that the backbone network has retrieved. This pyramid allows the network to leverage multi-scale information, capturing both detailed spatial information and high-level semantic features.

4.3.1 Backbone feature extraction

The backbone network will be represented by $\mathcal{B}$. The backbone network retrieves a series of feature maps at various network levels given an input image I.

 $\left\{C_1, C_2, C_3, C_4\right\}=B(I)$          (8)  

where, $C_i$ represents the feature map at level $i$ of the backbone.

4.3.2 Bottom-up pathway

The bottom-up pathway involves applying convolutional transformations to each feature map $C_i$ to create a set of feature maps $P_i$ for the feature pyramid.

Initial Feature Maps from Backbone Given the feature maps $C_1, C_2, C_3, C_4$, the bottom-up pathway creates feature maps $\left\{P_1, P_2, P_3, P_4\right\}$.

$P_i={Conv}_{ {bottom-up }}^i\left(C_i\right)$          (9) 

where,  $Conv_{ {bottom-up }}^i$ represents a convolutional operation applied to the feature map $C_i$ at level i.

Fusion in bottom-up pathway Each feature map $P_i$ at a given level i can be expressed as:

$P_i=W_i * C_i$        (10) 

where, * denotes the convolution operation and $W_i$ are the learnable convolutional weights at level i. This operation can be formalized as:

$P_i=\sigma\left(W_i * C_i+b_i\right)$        (11) 

where, $\sigma$ is an activation function and $b_i$ is the bias term.

Combining Feature Maps To produce the ultimate feature maps for the bottom-up approach, each feature map $P_i$ is concatenated or added based on upsampling or pooling operations:

$P_i={Concat}\left(P_{i+1}^{\uparrow}, {Conv}\left(C_i\right)\right)$        (12) 

where, $P_{i+1}^{\uparrow}$ denotes the upsampled feature map from the higher level i+1 and ${Conv}\left(C_i\right)$ is the convolution applied to the feature map at level i.

Full feature pyramid Finally, the set of feature maps for the entire feature pyramid is: $\left\{P_1, P_2, P_3, P_4\right\}$. The bottom-up pathway in Panoptic FPN is essential for constructing a multi-scale feature pyramid from backbone feature maps. Accurate panoptic segmentation depends on the integration of multi-scale information, which is achieved by the use of these feature maps in the top-down route and lateral linkages.

4.3.3 Top-down pathway

Using a top-down pathway, fused feature maps are produced by upsampling higher-level feature maps and adding them to lower-level feature maps via lateral connections. Let's denote the top-down feature maps as $\left\{T_1, T_2, T_3, T_4\right\}$. The top-down pathway can be represented as follows.

$T_i={Upsample}\left(T_{i+1}\right)+{Conv}\left(P_i ; \Theta_{t o p-d o w n}\right)$        (13) 

where, Upsample denotes upsampling operation, and $\Theta_{t o p-d o w n}$ represents the parameters of the convolutional layers.

4.3.4 Semantic segmentation head

The semantic segmentation head takes the top-down feature maps $T_i$ and produces semantic segmentation predictions $S_i$ at each level. The semantic segmentation head can be represented as follows.

$S_i={Conv}\left(T_i ; \Theta_{ {semantic }}\right)$        (14)

where, $\Theta_{ {semantic }}$ represents the parameters of the convolutional layers.

4.3.5 Instance segmentation head

The segmentation head takes the top-down feature maps $T_i$ and produces instance segmentation predictions $I_i$ at each level. The instance segmentation head can be represented as follows.

$I_i={Conv}\left(T_i ; \Theta_{ {instance }}\right)$        (15)

where, $\Theta_{ {instance }}$ represents the parameters of the convolutional layers.

4.3.6 Hybrid upsampling for accurate pixel labeling

To ensure that the segmentation outputs match the input image’s resolution, the proposed method utilizes hybrid upsampling. After feature extraction from the backbone network, the upsampling process enlarges the feature map through multiple stages of bilinear upsampling, followed by convolution and normalization layers, to generate per-pixel class labels. This hybrid technique effectively combines the detailed local features from the shallow layers of ResNet-50 with the global context captured at deeper layers, resulting in fine-grained pixel-level segmentation.

4.3.7 Panoptic segmentation integration

Finally, the semantic and instance segmentation predictions $S_i$ and $I_i$ are integrated to generate the panoptic segmentation output. The integration process can involve merging $S_i$ and $I_i$ based on pixel-wise classification scores and instance bounding boxes.

4.3.8 Lateral convolution

Prior to being integrated, lateral connections guarantee that feature maps from various levels have an equal number of channels.

$L_i={Conv}_{\text {lateral }}^i\left(P_i\right)=\sigma\left(W_i^L * P_i+b_i^L\right)$        (16)

where, $L_i$ is the output of the lateral convolution at level I; $W_i^L$ are the learnable weights for the lateral convolution; $b_i^L$ lateral convolution's bias term is denoted by L, while its activation function is represented by σ.

4.4 Pseudocode of panoptic FPN

4.5 Loss function and hyperparameter tuning

The network was optimized by combining binary cross-entropy loss for instance segmentation with cross-entropy loss for semantic segmentation. The instance segmentation branch also utilized IoU loss to ensure precise localization of lesions within the SD-OCT images. Standard optimization techniques like Adam were used to optimize these loss functions, and learning rates were modified in response to the model's performance during training. Through iterative refinement and feedback from validation, hyperparameters such as learning rate, batch size, and filter sizes were adjusted.

4.6 Classification

The Deep Residual Learning network (Res-Net-50) is used to classify the images into object categories. The network learns a variety of features from the images. Res-Net-50 created feature maps, which were supplied to the panoptic field head. The bounding box predicts the class ID of the corresponding pixel. Semantic segmentation employs the same border box properties to reduce the number of parameters and network interference time. The outputs of instance and semantic segmentation are combined to get the final segmentation results. The panoptic segmented image is sent into ResNet-50's residual blocks and convolutional layers. The network structure is mathematically represented as a series of transformations as follows.

$x_{l+1}=F\left(x_l,\left\{w_l\right\}\right)+x_l$        (17)

where,

1. $x_l$ is the input to the l^th layer.
2. $F\left(x_l,\left\{w_l\right\}\right)$ represents the residual function, typically composed of convolutional, batch normalization, and ReLU layers.
3. $\left\{w_l\right\}$ are the weights of the l^th layer.

The following represents each residual block in ResNet-50. 

$y={ReLU}\left(B N\left({Conv}\left(x, W_1, b_1\right)\right)\right)+x$        (18)

where,

1. $x$ is the input feature map.
2. ${Conv}\left(x, W_1, b_1\right)$ is the convolution operation.
3. BN is batch normalization.
4. y is the output of the residual block.

After passing through multiple layers and residual blocks, the high-dimensional feature maps are reduced using global average pooling.

$z=\frac{1}{H \times W} \sum_{i=1}^H \sum_{j=1}^W y_{i j}$        (19)

The feature map's height and breadth are denoted by H and W. A Fully Connected (FC) layer is then applied to the pooled feature vector $z$ in order to transfer the features to the categorization space.

$f=W_{f c} z+b_{f c}$        (20)

where,

1. $W_{f c}$ are the weights of the FC layer.
2. $b_{f c}$ is the bias term.
3. f is the output vector from the FC layer.

To generate a distribution of probabilities over each of the three groups (Normal, Abnormal, and Moderate), the output vector $f$ is subjected to a softmax function.

$\widehat{y_l}={Softmax}\left(f_i\right)=\frac{e^{f_i}}{\sum_j e^{f_i}}$        (21)

where,

1. $\widehat{y}_l$ is the predicted probability for the i^th category.
2. $f_i$ is the i^th element of the output vector f.

The final classification decision is based on the highest probability from the softmax output:

${Class}(I)=\arg \max_i \widehat{y}_i$     (22)

where, $\arg \max _i \widehat{y}_l$ yields the greatest value's index i in the probability distribution $\widehat{y_l}$.

Purtscher's Retinopathy is a rare but devastating eye condition that mostly affects young or middle-aged men and is brought on by trauma. A PR clinical diagnosis has been proposed. Three of the five conditions listed below must be met.

1. Purtscher Flecken
2. Low-to-moderate retinal hemorrhages of 1 - 10.
3. Cotton-wool stains are limited to the posterior pole.
4. Explanatory etiology that is probable or reasonable.
5. Investigations that are coherent to the diagnosis

The level of retinal degeneration in PR is determined, as shown in Figure 5. To analyze the disease, the retina is divided into three zones such as A, B, and C respectively. Zone A is with a radius of four-disc diameters and 2/3 of cases arise here. Zone B extends outside A. Zone C is anterior to Zone B and it is affected rarely. Panoptic segmentation assigns a class label to each pixel through its semantic process and the instance segmentation segments the object instance.

1. Zone A - Needs to be extended horizontally with four-disc diameters along both edges.
2. Zone B's outside edge horizon.
3. Zone C is the furthest, extending to the Ora Serrata.

PR, occlusion occurs at the level of pre-capillary arterioles, which have a diameter of roughly 45 m. This causes whiteness in the areas of the distal capillary beds and typically perfused retina nearer to the arterioles. Occlusions close to the pre-capillary arterioles cause confluent whitening, similar to a branch retinal artery occlusion, but occlusions further away cause cotton wool patches that conceal the underlying arteries.

As shown in Figure 6, there are several etiologies to take into consideration when diagnosing peripapillary cotton wool patches with associated macular retinal whitening, including vascular, metabolic, inflammatory, autoimmune, traumatic, viral, and hematologic. Every occurrence was symptomatic, exhibiting either a decrease in vision field or acuity, or both. Retinal pigment epithelium atrophy and optic disc pallor were the most common chronic symptoms. The diagnosis of PR was validated by the patient's manifestation of Purtscher Flecken in the context of C5a hypercomplementemia.

5.png

Figure 5. Retinal Zones involvement in PR

6.png

Figure 6. The retinal nerve fiber layer becomes thinner as a result of cotton wool patches [6]

7.png

Figure 7. Results reflect the best, median, and worst performances

Figures 7a to 7c show the best case, Figures 7d to 7f show the median case, and Figures 7g to 7i show the worst case of the obtained results through the proposed approach. The optimum best performance occurs when the OCT image displays the layers of the retina. Its Dice Coefficient is > 0.90 and IoU > 0.85. The retinal layers are precisely segmented by the model, which also detects anomalies such as retinal hemorrhages and whitening. The segmentation delineates the impacted areas, closely matching the ground reality.

The OCT image with mild retinal abnormalities and considerable noise is in the median performance. Metrics: Dice coefficient between 0.70 and 0.80, IoU between 0.60 and 0.75. With certain inconsistencies, the model offers a comparatively accurate segmentation. While it accurately detects the main characteristics of Purtscher's retinopathy, noise or poorly defined boundaries may cause it to ignore minute details or add a little segmentation error.

8a.png

(a) sub-retinal detachment

8b.png

(b) Macular edema

8c.png

(c) Purtscher’s flecken

8d.png

(d) Infected Purtscher’s retina analysis

Figure 8. Results of Purtscher’s Retinopathy analysis

When the OCT image exhibits extreme retinal distortion, artifacts, or noise, it is said to have the lowest performance. Dice coefficient < 0.60 and IoU < 0.50. The segmentation of anomalies and retinal layers by the model is not very accurate. Noise and artifacts can have a significant impact on predictions, resulting in a low overlap with the ground truth. In certain situations, the model is unable to offer meaningful diagnostic data. The spectral-domain optical coherence is shown in Figure 8. Left eye tomography shows a subfoveal neurosensory separation with rupture of the ellipsoid zone, along with significantly affected inner retinal areas and characteristic Purtscher Flecken (arrow).

The FPN results on OCT segmentation involve evaluating different performance metrics across several test images. This includes metrics like Intersection over Union (IoU), Dice coefficient, pixel accuracy, precision, recall, and F1 score.

1. IoU (Intersection over Union): It determines the degree to which the ground truth and the expected segmentation overlap.

$I o U=\frac{T P}{T P+F P+F N}$          (23)

2. Dice Coefficient: A similar measure to IoU, but it tends to be more sensitive to small objects. It is defined as follows.

 $Dice \ Coefficient=\frac{2 \times T P}{2 \times T P+F P+F N}$          (24)

3. Pixel Accuracy: The percentage of all pixels that were correctly predicted.

 $Pixel\ Accuracy=\frac{\ { No.\ of\ Correctly\ Predicted\ Pixels }}{ { Total\ no.of\ Pixels }}$          (25)

4. Precision: The ratio of correctly predicted positive outcomes to all predicted positive outcomes.

$Precision =\frac{T P}{T P+F P}$          (26)

5. Recall: The ratio of true positive forecasts to all real positives.

$Recall =\frac{T P}{T P+F N}$          (27)

Interpretation of Table 2 is as follows.

• Best Case (Image 4): Excellent segmentation performance is indicated by high IoU, Dice Coefficient, Pixel Accuracy, Precision, and Recall.
• Median Case (Image 7): Moderate values across all metrics indicate average performance.
• Worst Case (Image 5): Poor segmentation is indicated by low IoU, Dice Coefficient, Pixel Accuracy, Precision, and Recall; this might be because of noise or intricate structures in the OCT images.

To improve the statistical validity and generalizability of the results, the sample size of PR patients must be increased for further research. A larger dataset would enable more thorough research and a more accurate depiction of the features of the illness. Furthermore, to prevent bias and guarantee that the results apply to both male and female populations, equal gender representation is vital. A more thorough knowledge of the disease's course and its effects on various phases would also be possible with the inclusion of a wider variety of disease severity levels. In addition to increasing the prediction models' accuracy, these actions would help develop more individualized and efficient methods for diagnosing and treating PR.

The PR segmentation findings are shown in Figure 9, along with an evaluation of the IOUs. It is found that the model performs better when segmented photos are used as an extra source of training data.

9.png

Figure 9. IOU (Intersection over Union) of PR segmentation over epochs

Table 2. Comparison of FPN results on SD-OCT segmentation

Table 3. Comparison of OCT segmentation with competitive methods

In particular, the model converges somewhat quicker when merging segmented images than it does with training segmentation data, especially in the initial few epochs. To be more precise, the model's initial epoch performs nearly identically to the 10^th epoch when taking the validation set's performance into account. This phenomenon indicates that there is a substantial association between the segmentation task and the characteristics used for object detection. Table 2 aids in evaluating the FPN model's performance on OCT segmentation tasks and pointing out its advantages and disadvantages in various situations.

Table 3 provides a detailed comparison of the performance of different segmentation methods (Panoptic FPN, U-Net, SegNet, DeepLabV3, and PSPNet) on OCT images across several performance metrics.

• Panoptic FPN shows the highest Mean IoU values, with Panoptic FPN at 0.88. U-Net and DeepLabV3 have moderate IoU values, with U-Net at 0.71 and DeepLabV3 at 0.70. They are slightly less effective than Panoptic FPN and PSPNet but still perform well. SegNet has the lowest Mean IoU at 0.68, suggesting it has the least effective overlap in segmentation compared to the other methods, and shown in Figure 10.

• With a Mean Dice Coefficient of 0.92, Panoptic FPN is once again in the lead, followed by PSPNet at 0.81. A significant degree of similarity between the genuine and anticipated segmentation is indicated by high dice coefficients. The mean dice coefficients for U-Net and DeepLabV3 are 0.80 and 0.79, respectively, indicating comparable performance. With a Mean Dice Coefficient of 0.77, SegNet has the lowest score and is less comparable, as shown in Figure 11.
• With a mean pixel accuracy of 0.97, Panoptic FPN has the greatest percentage of properly predicted pixels. The robustness of PSPNet (0.86), U-Net (0.85), and DeepLabV3 (0.84) is demonstrated by their close follow-up. As seen in Figure 12, SegNet has the lowest Pixel precision (0.83), indicating that it is marginally less dependable in terms of pixel-wise precision.
• With Mean Precision values of 0.99 and 0.78, respectively, Panoptic FPN and PSPNet exhibit the highest precision. This indicates that a large percentage of their expected positives are really positive. With accuracy values of 0.77 and 0.76, respectively, U-Net and DeepLabV3 are comparable. As seen in Figure 13, SegNet once more has the lowest score, with a Mean Precision of 0.74.
• With Mean Recall values of 0.98 and 0.75, respectively, Panoptic FPN is the most successful at detecting true positives. Following with Mean Recall scores of 0.74 and 0.73 are U-Net and DeepLabV3. Figure 14 shows SegNet, which has the lowest recall at 0.71.
• With the highest Mean F1 Score of 0.93, Panoptic FPN exhibits an excellent balance between recall and accuracy. The performance of PSPNet (0.76), U-Net (0.75), and DeepLabV3 (0.74) is good but marginally worse. With a Mean F1 Score of 0.72, SegNet has the lowest score and is displayed in Figure 15.

The U-Net design, which was proposed by Karn and Abdull [21], is a common method for segmenting OCT images, particularly in situations where pixel-level classification is essential. Both fine-grained geographical features and high-level contextual information may be captured by its encoder-decoder structure with skip connections. When it comes to complicated, overlapping lesions like those found in PR, U-Net's capacity to differentiate distinct diseased characteristics is limited by its emphasis on semantic segmentation. This presents a major obstacle to the identification of retinal diseases because precise lesion-level segmentation is essential. To provide a more thorough knowledge of retinal defects, Panoptic FPN, on the other hand, combines instance and semantic segmentation, enabling it to recognize and distinguish particular lesions such as hemorrhages and cotton-wool patches. Additionally, the proposed Panoptic FPN's multi-scale feature extraction, which makes use of the FPN, improves its capacity to identify dispersed and tiny lesions, which is a drawback of U-Net in intricate retinal structures.

To improve segmentation performance in scene parsing tasks, PSPNet uses a global pooling layer in conjunction with a Pyramid Scene Parsing (PSP) module to gather context from various spatial scales [22]. The local, specific qualities that are essential for precise lesion diagnosis in retinal images are not sufficiently addressed, even though this is helpful for comprehending global scene contexts. In contrast, the proposed Panoptic FPN employs both instance and semantic segmentation, enabling the differentiation of individual lesions and enhancing the precision of detection in conditions such as PR, where accurate localization of lesions is crucial.

Panoptic FPN consistently performs the best across all metrics, demonstrating its effectiveness and robustness in OCT image segmentation. PSPNet shows competitive performance, slightly trailing behind Panoptic FPN, making it a strong contender. U-Net and DeepLabV3 both methods show moderate performance, proving their reliability but indicating room for improvement compared to Panoptic FPN and PSPNet. SegNet shows the least performance across all metrics, suggesting it is less suitable for SD-OCT segmentation tasks compared to the other methods to diagnose PR. Additionally, the architecture of the proposed Panoptic FPN is better suited to complicated retinal structures, whilst PSPNet's global scene parsing technique may easily overlook minor and overlapping lesions. As a result, Panoptic FPN provides a better model for tracking and detecting retinal problems early.

DeepLab V3 employs Atrous Spatial Pyramid Pooling (ASPP) for multi-scale feature extraction in order to better capture context at different resolutions [23]. DeepLab V3 has limits in retinal image segmentation, where minute and subtle lesions (typical of PR) need to be properly located, despite being successful for large-scale scene parsing. Furthermore, DeepLab V3 cannot independently detect distinct problematic components, instead concentrating on semantic segmentation. However, the proposed Panoptic FPN is more capable since it offers both instance and semantic segmentation. Panoptic FPN's multi-scale feature extraction provides better diagnostic capabilities for retinal disorders like PR by more successfully catching both major retinal abnormalities and fine-grained lesions.

To diagnose PR using SD-OCT images, the proposed Panoptic FPN outperforms U-Net [21], PSPNet [22], and DeepLab V3 [23] because it integrates instance and semantic segmentation, allowing it to recognize and differentiate certain lesions. PSPNet's emphasis on global image parsing renders it less successful at recognizing tiny or dispersed lesions, like those present in PR, whereas U-Net performs well for segmentation tasks but has trouble with complex overlapping lesions. Similar to this, the ASPP method for multi-scale detection in DeepLab V3 is effective for comprehending scenes but lacks the granularity needed to differentiate minute retinal lesions that are typical of PR. The proposed Panoptic FPN is especially well-suited for identifying the delicate, intricate, and dispersed lesions in PR because to its dual segmentation technique and multi-scale feature extraction, which yields more precise, localized, and clinically useful results.

10.png

Figure 10. Comparison of mean IOU

11.png

Figure 11. Comparison of mean dice coefficient

12.png

Figure 12. Comparison of mean pixel accuracy

13.png

Figure 13. Comparison of mean precision

14.png

Figure 14. Comparison of mean recall

15.png

Figure 15. Comparison of F1 score

The proposed method for diagnosing PR using panoptic segmentation offers several practical implications in a clinical setting. In terms of computational efficiency, the system is designed to operate with reasonable speed, benefiting from the use of ResNet-50 as a backbone, which strikes a balance between performance and computational cost. While the system may not yet operate in real-time in resource-constrained environments, with optimized hardware and further model refinement, real-time processing could be achievable. The system’s outputs, particularly the panoptic segmentation maps, can be presented in a clinically interpretable format through color-coded heatmaps, where specific lesions are highlighted for the ophthalmologist’s review. This enables visual confirmation of the machine's findings while also providing the segmentation of individual lesions, which aids in diagnosis. To further enhance interpretability, the system could include confidence scores or probability maps to reflect the certainty of the predictions made.

However, there are potential challenges in adopting such a system in clinical practice. The requirement for sizable annotated datasets for efficient model training is one major obstacle. The availability of sufficiently diverse, well-annotated datasets that cover different demographic groups, imaging conditions, and disease severities is a significant barrier. Without such data, the model's accuracy and generalizability may be limited. Additionally, there is a risk of over-reliance on automated systems, potentially leading to misdiagnoses if the model’s predictions are not cross-validated by experienced ophthalmologists. Clinicians must maintain a supervisory role in the diagnostic process, using the system as a supportive tool rather than as a replacement for expert judgment. To mitigate this, hybrid systems that combine machine learning outputs with clinician insights would be the most effective approach.

The primary limitation of this study is the small, imbalanced dataset, particularly with only 10 Purtscher-like Retinopathy (PR) cases (9 women and 1 man), which restricts the model’s generalizability. The model may also suffer from overfitting due to the limited number of PR cases and class imbalance. Additionally, the model is tailored for PR and may struggle to generalize to other retinal diseases, which often present different lesion types and severities.

To address these limitations, future studies should focus on collecting a larger, more diverse dataset with balanced gender, age, and disease severity. Incorporating OCT alongside fundus photography to improve lesion detection. Exploring transformer-based models and hybrid CNN-transformer architectures for better feature extraction. These advancements will enhance the model's accuracy, generalizability, and clinical applicability.