Effects of Different Pre-Training Algorithms on Video Classification Results

In this study, 600 videos collected from different accounts on social media (Instagram) are used. These videos are in 6 different classes and there are 100 videos in each class. These include videos titled mukbang, fitness, makeup, dance, swimming, and driving car. Since the videos are collected from different accounts, different people perform the same action. Thus, a wider range of data is obtained. The size of each video is 720*720, and each video consists of approximately 1000 frames. Each video contains 30 frames per second. Certain factors were taken into account to include 600 videos in the study. The videos consist of 6 different classes. The reason for choosing these 6 classes is that the effects of performing different actions in completely different areas are investigated and this number is sufficient and necessary to describe many actions. In other words, it was aimed to investigate the effect of different actions such as eating, applying make-up, and driving on classification. During the classification phase, factors such as the fact that videos in each category contain clear visuals of the action performed, videos with unnecessary details are not included in the classification, image sizes are compatible with each other, and there are videos of more than one type of action in the same category in the data set are taken into account so that learning is sufficient in every class. For example, when choosing makeup videos, the goal is to introduce different types of actions to the network, such as applying lipstick, applying mascara, applying foundation. or performing the same action by more than one person and in more than one way makes it easier to obtain more detailed data. Thus, the resulting data set contains different details. In the first stage, pre-training algorithms come into play to obtain features from the video collection stage.

Three different pre-training networks consisting of ResNet50, ResNet101, and GoogleNet, were used to obtain information from each framework. The main reason for using these three pre-training networks is the large number of layers in different numbers and the different depths of the 3 different networks. It was wanted to investigate what the effect of having different networks that have many layers and varying depths would be on classification. Then, the features obtained with pre-training algorithms were reduced in size.

GoogleNet; It has 22 deep layers. Regardless of the initial size of the images entering the layers, they are converted to 224*224*3 by the algorithm. And it has a total of 144 layers and 170 connections [8]. ResNet 50 has 50 deep layers. Regardless of the initial size of the images entering the layers, they are converted to 224*224*3 by the algorithm and it has a total of 177 layers and 192 connections [9]. ResNet 101; It has 101 deep layers. Regardless of the initial size of the images entering the layers, they are converted to 224*224*3 by the algorithm and it has a total of 347 layers and 379 connections [10]. In other words, there are three different network structures with different depth, connections and layer numbers. At the classification stage, 80% of the features obtained are allocated for training and 20% for testing, this is done using a 5-fold. All data is multiplied by 5 tests and training data are divided by each multiplier. During this division process, different data is trained on each fold.

2.1 t-SNE analysis

t-SNE is a linear technique used to reduce feature dimension [11]. It is especially used for visualizing high-dimensional feature matrices. It is widely applied in image processing and audio processing. t-SNE expresses the closeness and distance of data probabilities. Distances in space are expressed with a Gauss distribution, while distances in embedded space are expressed by t-distributions [12]. This makes the structure of t-SNE sensitive. t-SNE allows the structure to be revealed very broadly on a single map. Multiple, distinct, manifolds or data sets emerge. Collecting some points in the center reduces the total. While Local Linear Embedding and robustities are suitable for extracting a single continuous low-dimensional manifold, t-SNE extracts clustered groups of local samples by focusing on recorded local data [13]. The ability to group samples locally can be useful for visually resolving a data setup that contains multiple manifolds at once, as in a digital data setup. The most current and best solution for reducing dimensions is the t-SNE technique [14]. Therefore, the t-SNE dimensionality reduction algorithm was used in this study.

Dimension reduction analysis is a data visualization problem, which is the visualization of these reduced dimensions as a result of reducing the features obtained from the frames of each video into two dimensions. The main idea of the t-SNE algorithm is to find a low-dimensional representation that preserves the distances between points as much as possible. t-SNE starts with an arbitrary low-dimensional representation for each data point and tries to keep points close together and points far apart in the original space. t-SNE gives more importance to points that are close together rather than maintaining distance between points that are far apart. Due to all these advantages, 1000 features were extracted from each image, frame in the videos for all three pre-training algorithms. The extracted features were visualized by reducing them to two dimensions [15]. Results in Figure 1; It showed that after the dimensions of the extracted features were reduced and visualized, 100 videos from 6 classes were grouped within themselves. It was also observed that each of the 600 videos could be classified within itself.

1a.jpg

(a)

1b.jpg

(b)

1c.jpg

(c)

Figure 1. t-SNE analysis results (a) ResNet101, (b) ResNet50, (c) GoogleNet

2.2 Pixel analysis

Pixel analysis is a numerical evaluation of the density of the pixels that make up the image. That is, there is a numerical value corresponding to the pixel position. Therefore, these values are also very effective in classification. because one of the parameters that make up the obtained features is pixel values. Pixel analysis is the graphing of the values as a result of dividing each video into images and taking the average of the values corresponding to the pixel information taken in the same region of the image frames. This reveals that the mathematical results corresponding to pixels are of great importance in the feature determination process, as they create different and very specific images. Hence, pixel analysis is another data visualization technique used to check whether the collected data is suitable for classification. Thus, a point close to the center of the 100 videos in each class was selected and the pixel values corresponding to this point were analyzed. First of all, each video is divided into individual images and the images are stored for each video.

To select the pixel to be analyzed, the pixel (692, 832) was selected from the center of the images taken from the video using the image analyzer from the MATLAB applications section. and the values corresponding to this pixel were kept in the Excel file with .csv extension, and then the graphics in picture 2 were obtained by averaging the pixel values obtained for 100 videos. In other words, the pixel values of 100 videos in the first class were averaged, and the average values of 100 videos in the second class were taken.

This process was done for 6 classes, that is, 600 videos. The values corresponding to the pixels of the first 700 frames are shown for 6 different classes in Figure 2. The vertical axis represents the pixel value and the horizontal axis represents the frame number. Values range from 0 to 250 for each frame. The fact that the results are different for each class is a desirable situation and constitutes one of the components in distinguishing the videos.

2a.jpg

(a)

2b.jpg

(b)

2c.jpg

(c)

2d.jpg

(d)

2e.jpg

(e)

2f.jpg

(f)

Figure 2. Pixel analyzes of 6 different classes

Figure 2 shows the average pixel analysis of 6 different classes. It is seen that quite distinct and characteristic results are obtained for each class. Thus, it can be seen that conditions such as color and intensity of the videos to be classified will be very effective in extracting features.

2.3 Video classification

The first step in video classification is to line up each image frame that makes up the video. Then, features are retrieved from each of the sequenced frames. The number of these feature retrievals is 1000 for each of the three pre-training algorithms [16]. 1000 features are extracted from the FC1000 layer of ResNet 50 and ResNet 101 and the loss-3 classifier layer of GoogleNet. In the last stage, it is sent to the classifier and classified.

3.jpg

Figure 3. Algorithm structure from video classification

In Figure 3, we see a video classification algorithm. First of all, the frames of the videos belonging to each of the 6 classes we have are arranged one after the other. Consecutive frames are sent to a pre-training algorithm so that their features can be extracted. The obtained features are classified in MATLAB by sending them to the Bi-LSTM classifier. The Bi-LSTM classifier is used because it is a time-dependent classifier. Thus, it is a widely used algorithm for the classification of video in close pattern with time information. Let the attribute entering the network be denoted by X, and let the resulting label value be denoted by Y. Our first goal is to find the f function that provides the relationship between X and Y. Our second goal is to predict Y data in response to new X data. The function obtained from here is it is nonlinear and has an iterative solution.

$h w(x)=(w 0 * x 1 / w 1+x 2)+w 3 * x 3$             (1)

$\mathrm{Y}=h w(x)+\dot{\varepsilon}$               (2)

$\dot{\varepsilon}=\mathrm{Y}-h w(x)$               (3)

Here hw(x) constitutes the prediction feature, while έ represents the error data. The combination of prediction and error results in recorded actual values. The extracted features are turned into a matrix and sent to the Bi-LSTM classifier. Choosing the classification parameters correctly is important in order to avoid overfitting or underfitting of the system and to ensure that the modal gives correct results. One of the important parameters here, MiniBatchSize, is the size of the minibatch to be used for each training iteration, specified as a comma-separated pair of a positive integer.

Additionally, the epoch number expresses how much of all the training data is exposed to the network during the training process. The learning rate determines how fast the network can learn from data. This rate affects how quickly the network can adapt to new information and how quickly it can forget existing information. Low learning rates cause the network to learn slowly. The classification process was carried out with the parameters in Table 1 for all three pre-training algorithms.

Classification is done in 5 folds, the purpose of doing this is to review all the data by dividing it into 5 parts and to prevent any data that does not fall into the classification. In the context of Machine Learning (ML), Cross-Validation is a critical statistical technique used to evaluate the performance and generalizability of a particular predictive model or algorithm [17]. This method makes a model capable of performing extremely well on training data while also trying to minimize problems such as overfitting that occur if it performs poorly on new data [18]. Given the important role predictive models such as language classification and image classification play in AI applications, cross-validation is an essential component of the model evaluation process that ensures high-quality performance across different datasets and scenarios. Figure 4 shows the visual structure of cross-validation.

Table 1. Classification parameters used in feature classification

4.jpg

Figure 4. The schematic structure of cross-validation

It involves splitting the existing dataset into two or more distinct subsets called “folds.” The train data is processed for the number of floors here and tested on the remaining floors. By repeating this process many times. In this way, a more accurate and consistent model will be created. It is 5-fold cross-validation where the data is divided into 5 equal subsets and the network is trained and tested 5 times using a dissimilar subset as test data each time. After all iterations are completed, the results are averaged to determine the final model performance [19]. In addition, the features taken in the pre-training algorithms are sent to the Bi-LSTM classifier in certain proportions and amounts.

5a.jpg

(a)

5b.jpg

(b)

5c.jpg

(c)

Figure 5. Feature distribution histograms (a) ResNet101, (b) ResNet50, (c) GoogleNet

If overfitting occurs, the training error decreases and the test data increases. Moreover, the data is not learned but memorized. For this reason, when the test data arrived, it was not similar to the train data they had memorized, so it could not be classified correctly. It is possible to talk about high variance. The main reason for this is to prevent overfitting by introducing too much data into the network at once. Figure 5 shows the orientation histograms of the sent features in size 1*144. As can be seen, it is sent to the network in series between 200-300. The vertical axis (frequency axis) of the graphics in picture 3 expresses how large series of attributes are sent to the network. The horizontal axis represents the number of attributes sent in each cycle. This adjustability allows attributes to be sent in quantities that the network can handle. Thus, all data is reviewed. It also prevents problems such as overfitting.

In the classification phase, Bi-LSTM classification algorithm consisting of 780 hidden layers and 0.5 dropout is used. Bi-LSTM is an algorithm consisting of a bidirectional LSTM classifier [20]. Bidirectional LSTM is composed of two different LSTM layers that process input data forward and backward. As seen in Figure 6, both outputs of these two layers are used to give the final outputs. Additionally, LSTM often encounters poor performance due to the vanishing gradient problem. To overcome this problem, a two-way LSTM structure has been proposed. In the classification algorithm structure in one-way forward LSTM, only previous features are reviewed, but subsequent features are not taken into account. However, with the Bidirectional LSTM classifier, it is possible to use previous and future features effectively in feature classification. Unlike the LSTM network, the Bi-LSTM network has two equidistant structures in the forward and reverse directions [21]. Additionally, the number of layers and dropout value can be changed to enable classification with the Bi-LSTM classifier. Bi-LSTM can be expressed as the following equations; st is the hidden state output; and G is the weight matrix. Also, G_ys is a weight connecting input y to hidden layer s. b, bias; tanh is the activation function.

$\mathrm{{s}_\mathrm{t}^{\mathrm{f}}=\tanh \left(G_{y s}^f y_t+G_{s s}^f S_{t-1}^f+b_s^f\right)}$           (4)

$\mathrm{s}_{\mathrm{t}}^{\mathrm{b}}=\tanh \left(\mathrm{G}_{\mathrm{ys}}^{\mathrm{b}} \mathrm{y}_{\mathrm{t}}+\mathrm{G}_{\mathrm{ss}}^{\mathrm{b}} \mathrm{b}_{\mathrm{t}+1}^{\mathrm{b}}+\mathrm{b}_{\mathrm{s}}^{\mathrm{b}}\right)$                   (5)

$x_t=G_{s x}^f S_{\mathrm{t}}^{\mathrm{f}}+G^{\mathrm{b}}{ }_{\mathrm{sx}} \mathrm{s}_{\mathrm{t}}^{\mathrm{b}}+b_x$               (6)

6.jpg

Figure 6. Structure of bidirectional LSTM

The dataset with 6 classes and 100 videos in each class, collected from different accounts on Instagram, was created by us. Since the video durations on Instagram are specific, the frame numbers of the videos in the dataset are almost close to each other. While collecting data, attention was paid to the image quality and size of the videos. Moreover; It is important that there are different people performing the same action in the videos. Because what is intended to be done here is to ensure that the action taken is learned in detail by the network. For example; In mukbang videos, different people eat and the foods eaten differ from each other. These differences allow us to access more detailed and diverse information about the videos and in make-up videos, different people wear make-up and the make-up material applied to the face varies. In some videos, lipstick, eye shadow, foundation, and more than one make-up material are used together. After data was collected, 1000 features were obtained from each frame with 3 different pre-training algorithms (ResNet101, ResNet50, GoogleNet). By performing dimensionality reduction (t-SNE) analysis of these features, 600 videos belonging to 6 classes could be classified and successful results were obtained for the features obtained from all 3 pre-training algorithms. In addition, the average values of 100 videos in 6 different classes of the pixels taken from a point close to the center of the images were evaluated and it was observed that the values of each video gave quite different results. This difference has a positive effect on the distinguishability of the videos. After feature and pixel analysis, in the video classification stage, a Bi-LSTM classifier was used. The number of layers of the Bi-LSTM classifier can be increased or decreased at the desired rate, and overfitting can be prevented by making dropouts at the desired rate [27]. It was preferred due to these advantages. As a result of classification, the most successful results were obtained from the ResNet101 algorithm. Both the depth of the network and the number of layers are greater than the other two algorithms. Positive results have also been observed. It is understood from this that as the structure of the network improves, the success of the system and the received attributes increases [28]. Then, accuracy values of ResNet50 and GoogleNet were obtained. From the results obtained, it is seen that as the depth of the network and the number of layers increases, the obtained features are more accurate and more selected, as well as the accuracy and other parameters obtained. As a result, in this study, the effects of different feature extraction algorithms with the same dataset and the same classification parameters on the system outputs were investigated.