Advanced Computer Vision Alignment Technique Using Preprocessing Filters and Deep Learning

One of the most noteworthy problems in image processing field is image alignment, or IA. Image alignment is done for a variety of reasons, including getting region of interest images from various perspectives or distances, using various modalities, getting them with various techniques, and getting them at various times. Many industries, including medicine and radiology, defense sector, cartography, remote sensing, and computer vision, use the IA technique. In realm of image processing, image alignment is frequently used for tasks like image-guided operations and illness monitoring and diagnosis [1]. Image alignment algorithms are capable of identifying the connections between images that have different levels of overlap. These algorithms are particularly useful for tasks like video stabilization, summarization, and generating panoramic mosaics [2]. Two photos are used as the alignment process's input. A reference or fixed image is one of input images. We refer to the other image as moving one. The altered image that has been created by applying different transformations, like affine, rigid, or deformable, is represented by moved image. Moving and fixing image pairings into a single coordinate system is the process of image augmentation, or IA. The mapping process aims to align specific features present in both photos. Finding parameters which would result in maximizing similarity between the 2 images is the goal of the image alignment process [2, 3]. As demonstrated in Eq. (1), the image alignment problem may be viewed as optimization problem. Where $\hat{T}$ represents desired spatial transformation, IF stands for reference or fixed image, and IM for moving image in Eq. (1). (IM) is the distorted image produced by applying transformation T on IM. The dissimilarity or the similarity between IF and the warped images (IM) is assessed by function S(). The objerctive of image alignment is finding the transformation T that, given a cost function, maximizes image similarity or reduces image dissimilarity [4].

$\hat{T}=\operatorname{argmi}\left(I_F, T\left(I_M\right)\right)$       (1)

DL networks like CNNs were known as state-of-art in a broad range of applications, especially in the field of computer vision like classification, segmentation, and alignment due to exhibited promising results. When compared to classical methods, DL methods yield better results. Most DL-based models are used in segmentation and classification applications [5, 6]. The alignment challenge differs significantly from the standard classification problem. Finding the image similarity and transformation parameter values (rotation, translation, and scaling) maximizing similarity between fixed and moving images is critical in the image alignment problem. Consequently, it is not possible to directly adapt CNN networks to the alignment issue [7]. Numerous successful research has applied DL techniques on registered image pairs. To evaluate the position and posture of implemented subject during surgery, Miao et al. [8] employed CNN regression. The goal of their CNN model was to forecast 6 inflexible parameters of transformation. The vast majority of realistic ground-truth training material for DL-based image alignment was synthesized by Uzunova et al. [9]. From the available training images, they essentially learned a statistical appearance model, which they then used to synthesis a random number of new images. The study of Eppenhof et al. [10] included employing CNN to record deformable medical computed tomography image pairings. Transformation parameters are learned immediately by their architecture. Utilizing fully convolutional network, some researchers [11] provide an unsupervised learning technique for registering CT image pairings. Their network warps a moving image to a map-fixed image by estimating a dense displacement vector. De Vos et al. [12] used unsupervised learning to train their CNN regressor end-to-end in order to maximize the metrics of similarity between the image pairs. They have used the MNIST data collection and images to test their network. The spatial transform network, which creates the vector of displacement, makes use of the network's output parameters. CNN-based unsupervised stiff image alignment has been proposed by Liu et al. [13]. The network output is the fixed and moved images attributes. Between registered image and fixed image, loss is computed. The stiff motion was represented by ultrasound images. They claimed that compared to traditional procedures, their approach produced faster and more successful results [14]. The key contribution of this work is the innovative, supervised, CNN-based rigid image alignment technique that has been suggested. By experimenting with various parameters, it has been attempted to find the ones that provide the best outcomes for the problem. Furthermore, we examine how the state-of-the-art convolutional network performs in a job of image classification in relation to the quality of images, and we propose a unique architecture to mitigate the consequences. We select common noise forms and JPEG lossy compression, which are immediately recognizable as bad quality in real-world photographs. We utilize the most prevalent noise sources that are observed in actual world—Gaussian—due to the fact that there are other causes which can generate different noise types. Next, in order to improve image alignment performance in the noisy environments, we explore how popular models of NNs perform worse and experiment with the most popular input image pre-processing. We create the neural network model using the most popular input image denoising technique.

This work aims to extract spatial transformation parameters between two photos using CNN regression architecture. Figure 1 displays the model's overview. When estimating transformation parameters as an output, CNN regression uses the model's input of fixed and moving images from prepressing stage. Concatenation of the fixed and moving denoising images occurs before the feeding network, after which they are fed as a single image to the alignment network. Since the supervised technique was employed as a learning strategy, ground-truth values were utilized throughout training phase. Two images are processed by the CNN alignment network using a sequence of convolutional and pooling layers. Utilizing the variation in the parameter of transformation that has been generated by the network has been trained by using loss value in conjunction with ground-truth values. The ground-truth value and the model's anticipated output value are the two inputs needed by the loss function.

3.1 Dataset and preprocessing

The Stanford Streaming MAR Dataset has been utilized in order to put the suggested techniques into practice.

The dataset has four categories with 23 distinct objects of interest, each represented by a single film in which the object remains stationary while the camera moves. Every video has 100 frames with a 128×256 resolution. The visuals from the videos in the MAR dataset are shown in Figure 2.

The Figure 3 represents the some samples of fixed and moved images in MAR dataset .Even though it was rarely noticeable, the present CNN produces images with quality loss or noise with decreased accuracy. Historically, noisy images have been processed using image-denoising techniques. Denoising sharpens the edges while reducing fine details and disturbances. This process can be conceptualized as essentially blurring the image and then sharpening its edges. Consequently, the resulting image suppresses noise by emphasizing edges and suppressed features. Denoising techniques have been shown to be a reliable means of suppressing noise, but whether or not they would function well when applied to a deep neural network remains to be seen. Because of this, before grafting the input image onto the neural network, we will first conduct denoising on it. Along with this single-model model, we empirically demonstrate that using the denoising method for training and testing can result in a model that is resistant to certain kinds of noise. We also produced JPEG compression loss because it might potentially be resistant to common quality-distortions.

Have been applied Several rotations between [-20, 20] to original images in order to create synthetic images for the stiff transformation. The fixed image was rotated counterclockwise (positive value of rotation) and clockwise (negative value of rotation) to create synthetic images. The 750 and 250 photos from the MAR data sets are used to train the network.

1.png

Figure 1. Proposed model overview

2.png

Figure 2. MAR dataset

3.png

Figure 3. MAR dataset, fixed and moved images

3.2 Transformation model

The stiff technique served as the study's spatial transformation model. The parameters of the rotation transformations used to make the photos were attempted to be calculated. Mapping every one of the pixels in the image from one position (x1, y1) to the other (x2, y2) is one way to communicate geometric transformations. The rotation amount (θ) is the network's output. Rotation is the process of rotating a point at specific angle (θ) around an (x1, y1) coordinate in the input image to (x2, y2) point in the output image. A transformation matrix is a rotation matrix, as seen in Eq. (3). It is used to turn something in Euclidean space

$\begin{aligned} & x 2 \\ & y 2\end{aligned}=\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta\end{array} \times \begin{gathered}x 1 \\ y 1\end{gathered}$         (3)

The two-dimensional Cartesian coordinate system's origin is rotated counterclockwise by this matrix at an angle of θ with respect to positive x axis.

3.3 Network archıtecture

The procedures involved in network design contain a few crucial points. The alignment challenge differs significantly from the standard classification problem. Finding the image similarity and transformation parameter values (translation, scaling and rotation) maximizing similarity between fixed and moved images is critical in the image alignment problem. Thus, it is not possible to directly adapt CNN networks-which are regarded as state-of-the-art in the studies of classification and segmentation—to the alignment problem. CNN can be created as regression network for use in IA applications. A continuous numerical value is returned by the single neuron in the last layer of the CNN. The concatenation of the images' moving and fixed positions serves as the network's input, and outputs of spatial parameters θ represent the network. The learning process was guided by ground-truth/target values. Ground-truth values are produced using simulated photos. For the estimation of rigid transformation parameters needed for warping moving to fixed images, a supervised DL network is employed. Loss between the ground-truth labels and projected parameters is estimated next. During the data augmentation phase, a significant quantity of artificial images and ground truth are produced by randomly rotating the original fixed images. There was no similarity metric applied in this study. The network has been trained with the use of loss function rather than image similarity metric. This approach’s drawback is the fact that ground-truth data cannot be gathered. Getting ground-truth data can be done in two ways: the first, using classical alignment algorithms to create it synthetically, and second, using synthetic data. The study employed the second strategy because there weren't enough images.