A Deep Transfer Learning Based Visual Complexity Evaluation Approach to Mobile User Interfaces

In recent years, people's access to mobile devices such as smartphones and tablets has become easier, and with regard to this technological change, the amount and use of mobile applications has grown dramatically. Considering that there are millions of mobile applications being developed in areas like online shopping, food delivery, online banking, social media, entertainment and online learning, it is obvious that mobile applications are the main gateway to the virtual world. Hence, for service/product providers or application developers, users interface complexity perception is one of the most important components to reach large audiences. Therefore, user interfaces that are perceived less complex, easy to use and in line with user`s mental representation are becoming more in demand.

On the other hand, mobile applications with less intuitive, distractive or complex user interfaces (UI) may hinder user acceptance. From the user's point of view, almost every user screen of the application should be easily comprehensible but also respond to user expectations. From the application developer's point of view, it is a tedious process to develop low-complex graphical user interfaces especially for mobile applications taking into account constraints such as limited screen sizes, on screen keys and small input fields.

Many different factors effect visual complexity, such as graphical design of the UI, the layout, the number and variety of components and the relationship between these components and so on. Complexity analysis of visual interface and its understandability has long been the focus of attention of researchers. While in earlier studies [1-3], perceptual visual complexity had been tried to be defined based on perceptual attributes, in later studies, visual complexity of UIs has been measured by user evaluation with surveys, developing formal metrics and models [4-6], heuristics [7] to machine learning based [8] methods. Although formal metric and model based methods are mostly used, due the need of a blend of different metrics for different attributes and their potential complex interaction this problem is still not fully resolved. One possible reason for this is that the success of this method is tightly dependent various assumptions such as number and combination of metrics and metric coefficients used in the conceived predictive model [4]. Another reason is that there may be other factors that affect the perception of visual complexity but still cannot be formulated precisely [5].

For the above stated reasons, as opposed to user evaluation, metric/model based methods; machine learning based approaches focus on visual complexity evaluation rely on machine learning models` understanding. Although, previously deep learning models for other type of general image visual complexity problems [9-11] and even for the quality assessment problem for 3D images [12] produced promising results, deep learning models for visual complexity of mobile UIs has not been studied thoroughly. The absence of latent factors that can be utilized in addition to existing metrics without directly involving human in the visual complexity analysis process encouraged us to explore pre-trained models that are known to be effective for computer vision tasks.

Therefore in this study, we explored the effectiveness of pre-trained deep learning models to evaluate mobile UI visual complexity. Various mobile applications are used in the data set gathering process and during experiments. However, at the time we preferred to include UIs that exist for common purposes/features in many applications. For this reason, four different widely used mobile UI types have been determined as login, search, menu and settings. A dataset containing these UIs has been prepared and classification models with five different state of the art deep neural networks for visual tasks have been adapted. Pre-trained deep learning models, namely VGG16 [13], DenseNet121 [14], MobileNetv2 [15], GoogleNet [16] and ResNet152 [17], were trained with 3635 different mobile UI images as login, menu, search and settings. We aimed to explore deep learning models for mobile UIs to see its potential for mobile UI visual complexity measurement. The three main contributions of the study are as follows: 1) A dataset, which includes different mobile UI types as login, menu, search, settings and registration, was distilled from the RICO dataset for this study, this can be used for other similar studies. 2) Five state of the art pre-trained deep learning models are experimented with and their accuracies are compared for mobile UI complexity evaluation to identify the most suitable for this task 3) In order to validate the effectiveness of the approach, a new validation dataset was collected from different mobile applications and formed then an application is developed to be used in the evaluation study. The evaluation was conducted with 98 participants where 7309 comparison result were obtained. It was found that the agreement rate between the deep learning models results and the user evaluations was up to 78% on the average 74%. The high to moderate agreement rate between the results of deep learning models and the user evaluations reveals that this approach can be useful for automatic visual complexity evaluation of mobile user interfaces.

The rest of the paper is as follows: section II reviews the visual complexity analysis methods. In section III, the proposed deep learning model is described. The validation process and the results of the proposed model are presented in section IV. Discussions and the potential future works are given in section V. The potential threats to the validity of this research are discussed in section VI, followed by conclusions in section VII.

In this section, the method for visual complexity evaluation of mobile UIs is discussed. To this end, the dataset, the deep neural networks, parameters and the training process of the networks are explained respectively.

3.1 Dataset

Among the datasets mentioned in the previous section, the most suitable for our study was the RICO dataset. RICO is created from over 9300 android applications in 2017 for data centric research to be conducted by data miners, mobile application developers and UI/UX designers. It provides the largest mobile UI data with an approximate size of more than 200 GB (UI screenshots and metadata, user ratings, animations and user interactions etc.) needed by researchers to perform data centric applications such as UI design and code generation, user interaction modeling, user perception prediction, design search.

We can list the reasons for preferring the RICO dataset as follows: 1) Instead of collecting and creating our own therefore potentially limited data samples and resolution, we preferred to use a dataset which is publically available, has many samples and also used in other similar studies for more generalizable results. 2) Among the other potential datasets SAVOIAS is not a mobile UI dataset and Appsthetics is the subset of images randomly taken from the RICO dataset for labeling the mobile UIs. 3) RICO is an extended version of ERICA. Therefore, a publicly available dataset, the RICO dataset, containing 72000 mobile UI images in different categories was used in this study.

Classification of mobile UIs is not a trivial process when there are applications from various categories. For this reason, the dataset was preprocessed in three steps before using it. In the first step, types of mobile UIs to be included in the classification process were determined. For this, four UI types have been listed such as; login, search, menu and settings by paying attention to the UIs used in almost every application regardless of the mobile application category. Secondly, suitable mobile UI images were selected among 72000 images in the RICO dataset. With the preliminary study carried out for this purpose, all images reside in the dataset were examined one by one and a total of 3635 images were extracted. In the last step, the resulting distilled images were organized to be used as training data. In this context, the images of each UI type were kept in a hierarchical structure and made ready to be used in the training of deep neural networks. Then, the obtained data was included in the training process of deep learning models such that 80% for training data, 10% for validation data and 10% test data (Table 1).

Table 1. Number of images in the dataset created for four different mobile UI categories

Apart from the dataset used in the training of classification models, another dataset was created by us to be used for the validation phase of our study. That is we developed a dataset this time to validate the visual complexity analysis results obtained by deep neural networks with the real users. This dataset consisted of UI images of mobile applications available in Android or IOS markets that are not included in the RICO dataset. In order to keep the UI diversity as high as possible, there exist 51 different UI images of 30 mobile applications in many categories ranging from online newspaper, online banking, and online shopping to entertainment. As in the training dataset, UI images in this dataset were located under the categories of login, search, menu and settings. In Figures 1-4, sample images of login, search, menu and settings UIs of the dataset are shown, respectively.

1.png

Figure 1. Sample images for login UI

2.jpg

Figure 2. Sample images for search UI

3.jpg

Figure 3. Sample images for menu UI

4.jpg

Figure 4. Sample images for settings UI

3.2 Transfer learning with pre-trained architectures

Training process for UI image classification task was carried out with five state of the art pre-trained deep neural networks: VGG16 [25], ResNet-v2-152 [26], MobileNetv2 [29], GoogleNet [30] and DenseNet-121 [28]. The development environment used during the training of deep neural networks is based on Python v3.8.5 and the rest of scientific computing libraries were obtained through anaconda, the most widely used python distribution. All deep neural networks used in the study were acquired using python's torchvision v0.8.2 library, which was developed for computer vision tasks and contains model architectures.

We decided to experiment with both with low parameter count (MobileNetv2, GoogleNet, DenseNet-121) and high parameter count (VGG16, ResNet-v2-152) together. We wanted to determine whether deep networks with a small number of parameters are efficient, in line with the need to work on mobile devices such as tablets and mobile phones with low processing power.

In the training of the pre-trained models, transfer learning technique was used, specifically, only the classification layer were retrained on each model that was previously trained on the “ImageNet” dataset. Therefore, generic features of the UI images are extracted with the help of pre-trained networks. However, pre-trained models have high parameter count to classify 1000 classes by default. Since there were four different classes in our study, we replaced pre-trained classification layer of each model with a lightweight and randomly initialized classification layer to classify four classes of mobile UIs (login, search, menu, settings) from scratch. In this way, training process was achieved in a shorter time as we had aimed. One of the precautions we took to avoid overfitting of the networks we did not use the entire dataset in training phase. We used randomly selected 3271 mobile UI images as shown in Table 1 to train the networks. We reserved the remaining 365 screenshots for testing purposes after the training of the networks was completed.

Table 2. Hyper parameters related to the deep neural networks used and the results of training process

Hyper parameters for each deep network and the top accuracies for the classification task are shown in Table 2. During the training process, the hyper parameters were improved over time by trial and error to find the optimum for each parameter. For this purpose, firstly, the average number of epochs giving the highest accuracy was determined for the initial configuration of each model such as “Stochastic Gradient Descent-SGD” as an optimization algorithm and 0.002 as learning rate. This experiment was carried out as follows; 1) In order to prevent overfitting, the training was stopped at the epochs where the accuracy in Table 2 were reached. 2) The average 10 epochs obtained from the training for each model was determined. Since the optimum learning rate was observed to be 0.002 with various trials in order for the model to achieve high accuracy with less computation time, this value kept constant for all models. Then, SGD with momentum were tried as an optimization algorithm without changing the rest of parameters. It was observed that by using SGD with momentum, the classification accuracy of the model increased while average epoch number decreased. On the other hand, using the Adam optimization for GoogleNet, among other models trained with a high number of epochs; it was observed that the same accuracy was achieved with lower total loss and fewer epochs.

Perceptual visual complexity analysis problem for mobile UIs has been reduced to a classification problem. The classification is done by pre-trained deep learning models with the transfer learning method on mobile UI datasets we have formed. Following the model training, the type of a mobile UI image was determined by the trained models and then the feature maps for each model are taken into account. Previously [33], it was observed that the scores obtained from the feature maps of the first and last layers were not compatible with participant evaluations. Based on this, we also used feature maps in the intermediate layers of each model. Accordingly, the architectures of deep learning models, the layers from which feature maps are used are shown in Figure 5. Moreover, as an illustrative example, the feature maps of each layer corresponding to the sample mobile interface images given as input to the VGG16 network are shown in Figure 6.

5.jpg

Figure 5. The architecture of deep learning models used; the feature maps are obtained from the intermediate layers (green boxes) for visual complexity calculation

For each deep learning model, the formula in Eq. (1) [33] was applied to the feature maps and a scaler value which is named visual complexity value of the input image was obtained.

$V C(l)=\frac{1}{h * w * d} \sum_{i=1}^h \sum_{i=1}^w \sum_{k=1}^d F_l[i, j, k]$                   (1)

where, VC is visual complexity value, h, w, d, l and F are height, width, depth, layer number and feature map, respectively.

After obtaining the VC values for experimented mobile UI images, as we wanted to see the agreement with the human perception a validation dataset using the screenshots of different mobile UIs was prepared and a survey study was conducted for labeling the newly prepared dataset.

6.png

Figure 6. The images of the feature maps in VGG16 corresponding to example inputs for each of mobile UI types. In the first row, input images are login, search, menu and settings, respectively. In the following rows below, feature maps for each convolution layer are shown

The results obtained within the scope of this study could be evaluated from different perspectives. First, with the diversity of the participants included in the survey indicate that the results of the state of the art deep learning models for visual complexity analysis agree to a certain degree with human perception. The highest match percentage between the deep learning models and participant evaluations for in-category comparisons belongs to VGG16 and it is 67.4%. This result shows that the visual complexity difference between the UIs belonging to the same category can be perceived by the participants and this complexity difference can be revealed numerically by deep learning models. In addition, it is understood that the participants could not make a decision when the visual complexity difference between two mobile UIs compared is small. From this point of view, deep learning models can still produce a visual complexity score even when the participants cannot decide (29.7% of total comparisons marked as “No Idea”) which UI is more complex.

For mobile UIs belonging to specific categories, the results show that the participants can make comparisons more easily. In other words, the undecided comparison rate decreased to 18.1%. In addition, the difference in visual complexity score was ensured to be more pronounced, and in this case, the agreement is 74% by reflecting on the results.

In addition, participant groups’ perception of visual complexity varies according to the participant profile. This is already the main factor that makes the visual complexity analysis problem complicated. The reason why group 2 participants' evaluations have lower compatibility percentage with the proposed method compared to group 1 and group 3 participants is that the participants in this group have different perceptions compared to the others.