Exploring Attentive User Interface Input via Raspberry Pi, Based on Face Landmark Detection, Eye Open-Closed Detection and Head Movements Detection

2.1 Hardware design

To build and test the proposed AUI input system, various supporting hardware components are required, as illustrated in Table 1. The system design block diagram is depicted in Figure 2.

Table 1. Hardware

As depicted in Figure 2, the camera employed to capture visual attention is a C525 Logitech webcam. The Raspberry Pi 4 Model B serves as a single-board computer (SBC), operating on the Raspbian 10 (Buster) operating system, with a 5-volt 3-ampere voltage source. A 23-inch monitor functions as the display screen, while wireless keyboard and mouse are utilized as interface media for program creation, execution of algorithms, and cursor navigation on the Raspberry Pi.

2.png

Figure 2. Hardware design

2.2 System design

The system in this research runs on the Raspberry Pi 4 embedded system by developing algorithms and programs that extensively utilize the Python 3.7 programming language, OpenCV 4.1.0 library, Dlib machine learning (ML) library, and Mediapipe ML library. The OpenCV library is the most well-known computer vision library, which builds on algorithms that run in real-time, such as the Viola-Jones technique based on Haar fature-based cascade classifiers for object detection [22, 23]. Dlib is a cross-platform toolkit based on C++ programming that can be used as ML tools and for the creation of algorithms. Dlib includes several modules mainly focused on ML, deep learning, and image processing [24], while Mediapipe is a set of libraries, models, and methods trained using ML so that it can be used for face identification in images, facial landmark detection, body pose tracking, and object recognition [25, 26]. In its current development, the OpenCV library, Dlib ML library, and Mediapipe ML library are widely used as tools in designing computer vision-based systems [27-31].

In this research, various methods employed in the design of facial landmark detection, open and closed eye condition detection, and head movement detection will be compared. These methods include face detection using Haar cascade classifiers and Dlib [32, 33], Dlib face landmark detection [34-36], and Mediapipe face landmark detection [37, 38]. Facial landmark detection plays a crucial role in the design of this AUI input system. Therefore, a comparison between Dlib face landmark detection and Mediapipe face landmark detection will be conducted. This comparison is necessary as the number of facial landmark detections between Dlib and Mediapipe differs. Dlib-based landmark detection results in 68 face landmark detections, while Mediapipe landmark detection results in 468 face landmark detections. However, it is worth noting that the utilization of Dlib face landmarks requires prior face detection, which is different from Mediapipe-based facial landmark detection. Figure 3 illustrates the AUI input design consisting of facial landmark detection, eye open-close detection, and head movement detection.

As depicted in Figure 3, there are 8 AUI inputs, each assigned a detection code (a to h), indicating the outcome of the AUI input detection process. Facial landmark detection holds a significant role in the proposed design for AUI inputs. Two prominent methods of facial landmark detection, namely using Dlib and Mediapipe, are employed and tested due to their popularity in the field of computer vision. In this study, the outcomes of facial landmark detection serve as input information for AUI detection. Figure 4 illustrates facial landmark detection based on Dlib, while Figure 5 showcases facial landmark detection based on Mediapipe.

3.png

Figure 3. Design of the AUI input

4.png

Figure 4. Dlib based 68 landmarks [36]

5.png

Figure 5. Mediapipe based 468 landmarks [38]

Since the study involves the utilization, testing, and comparison of face landmark detection based on Dlib and Mediapipe as inputs for the AUI, the experiment has been categorized into several scenarios, including:

Scenario 1: AUI input based on Dlib detection of 68 landmarks ("shape_predictor_68_face_landmarks.dat") using Haar cascade Classifier for face detection.

Scenario 2: AUI input based on Dlib landmark detection using Dlib HOG (histogram of oriented gradients) for face detection.

Scenario 3: AUI input based on Mediapipe detection of 468 landmarks.

Both Scenario 1 and Scenario 2 involve the detection of 68 landmarks based on Dlib. However, in this context, the study aims to compare the impact or differences of using face detection methods, namely Haar cascade Classifier and Dlib, as the foundation for the 68 landmark detection. This is due to the fact that Dlib's 68 landmark detection relies on face detection. Additionally, each scenario undergoes AUI input testing as depicted in Figure 3.

Upon successful landmark detection, the subsequent phase involves designing the AUI input for detecting open and closed eye conditions. For the eye condition detection, the researchers utilize the eye aspect ratio (EAR) method. This method calculates the height and width of the eye region [34, 39, 40]. As illustrated in Figure 6 and Equation 1, the EAR method is chosen based on previous research demonstrating its success in detecting open and closed eye conditions, particularly in drowsiness applications [34, 41].

$E A R=\frac{\left\|p_2-p_6\right\|+\left\|p_3-p_5\right\|}{2\left\|p_1-p_4\right\|}$    (1)

As shown in Figure 6 and Eq. (1), the computation of the detected eye landmark region utilizes the eye aspect ratio (EAR), which calculates the ratio between the height and width of the eye region. The values P1 through P6 represent the locations of the detected eye landmark points. The EAR tends to remain relatively constant when the eyes are open and approaches a value close to zero, allowing this method to be used for estimating open and closed eye conditions [34, 41]. In this study, the detection of eye conditions serves as input information for AUI in discerning open and closed eye states.

Regarding head movement detection or monitoring, this experiment serves as the AUI input for view direction detection, encompassing detection of views to the right or forward, views to the right or left, and views up or down. Head movement monitoring contributes to the study of human behavior, enhancing our understanding of a person's attention, focus, and emotional state [42]. For head movement detection, the researchers employ the head pose estimation method, correlating the 2D facial landmark detection results with the 3D face shape [43-45], based on the detection of landmark points on the face, as depicted in Figure 7.

In this experiment, 2D landmark points are derived from landmark detection using Dlib and Mediapipe. The 3D face shape is expressed in world coordinates, with the ability to convert world coordinate points to camera coordinates. In camera coordinates, these points are projected onto the image plane using camera intrinsic parameters, including focal length, optical center, and radial distortion, as outlined in Eq. (2). Matrix A within the equation represents the camera matrix containing intrinsic parameters and key points, while matrix [R|t] signifies the joint rotation-translation matrix that transforms points from the 3D world coordinate system to the 3D camera coordinate system [46].

$p_i=A[R \mid t] P_i$    (2)

The head pose estimation at 2D-3D landmark points is conducted through the solvePnP (Perspective-n-Point) algorithm in the openCV function. SolvePnP is an algorithm that extracts rotation vectors and translation vectors using DLT (direct linear transform) and Levenberg-Marquardt optimization [46, 47]. Through solvePnP 2D-3D, head movement estimation is obtained, encompassing the roll, pitch, and yaw rotation axis values. However, the experiment primarily focuses on the pitch (vertical) rotation value, representing forward, up, and down views, and the yaw (horizontal) value, representing left and right views. This movement aligns with the head movement classification illustrated in Figure 8 [42].

As depicted in Figure 8, head movement classification serves as a reference for head movement detection, representing the direction of the view. In this experiment, it is utilized as the basis for determining head movement detection.

The comprehensive design of the proposed Attentive User Interface (AUI) input, incorporating facial landmark detection, open and closed eye conditions, and head movement detection, is illustrated in the block diagram presented in Figure 9.

As seen in Figure 9, there are 3 sections labeled A, B, and C that consist of parts of the detection process. As previously explained, face landmark detection is a very important initial part in the process of designing this AUI input system because, through the results of face landmark detection, eye open-close detection and head movement detection can be modeled. On the label of section “A”, there are three scenarios of face landmark detection processes that have different methods, and later each face landmark detection result in each scenario will be continued in section “B”, namely eye open-close detection and section “C” head movement detection. The purpose of making these three scenarios is to test, know, and compare the performance of each detection scenario when the AUI input system is running on Raspberry Pi 4. Each detection process, including face landmark detection, eye open-close detection, and head movement detection, is programmed with the same algorithm the difference lies only in the face landmark detection method based on the three proposed scenarios. Later, the design of this AUI input system resulted in three application-based Python forms based on scenarios 1, 2, and 3.

6.png

Figure 6. Eye landmark indicators [34]

7.png

Figure 7. 3D landmarks vs 2D landmarks [44]

8.png

Figure 8. Classification of head movements [42]

9.png

Figure 9. AUI input block diagram: A. Face landmark detection, B. Eye open-close detection, C. Head movement detection

The presentation of the results for the tested AUI input proposed through the Raspberry Pi 4 is illustrated in Figure 11. It is evident that the three AUI input scenarios are functional, as indicated by the detection codes representing various AUI input indicators, so that scenarios 1, 2, and 3 can display the AUI input detection result indicator with the participant facing front or facing the monitor.

Figure 11 provides insights into the functionality of the system based on the proposed scenarios tested on the participants. The AUI input indicators are displayed in the upper left corner, presenting detection result codes in alignment with the AUI input design depicted in the previous Figure 3. The information in Figure 11 indicates "AUI: a, b, d" with the following meanings:

-Code “a” provides information on detected facial landmarks, enabling the detection of AUI input information related to face detection and recognizing the user's face.

-Code “b” offers information on eye condition detection, with the detection result indicating an open eye. Thus, the AUI input system successfully detects information related to eye condition.

-Code “d” furnishes details on head movement conditions associated with the viewing direction. The result indicates forward view detection, facilitating the AUI input system to detect information on viewing direction.

11.png

Figure 11. AUI input results using Raspberry Pi 4

The monitoring results are observed through the Raspberry Pi task manager when executing three AUI input system scenarios. The primary focus of the monitoring process is directed towards examining central processing unit (CPU) usage and memory usage. This approach facilitates an assessment of the performance of the proposed AUI input system, as depicted in Table 4.

Table 4. Raspberry Pi 4 task manager results

Table 5. FPS comparison on tre AUI input scenarios

Based on Table 4, Scenario 1 of the AUI input system exhibits the highest average CPU and memory usage values when compared to Scenarios 2 and 3. Specifically, Scenario 2 demonstrates the lowest average CPU usage, while Scenario 3 shows the lowest memory usage among the proposed AUI input system scenarios. In addition to task manager monitoring, we also observed the frames per second (FPS) values, as shown in Figure 12 and Table 5.

As shown in Figure 12 and Tables 5, based on the experimental results, scenario 3 has the highest FPS value with an average value of 10 FPS when running on Raspberry Pi 4, while the minimum FPS value is 9 FPS and the maximum FPS value is 11 FPS, so when compared to the FPS values of scenarios 1 and 2, the FPS value of scenario 3 using the Mediapipe-based facial landmark detection method is much better. On the other hand, scenario 2 recorded the lowest FPS value in this test, with an average value of 3 FPS, while the FPS value of scenario 1 is in the middle with an average value of 5 FPS. From the results of this test, it was found that differences in facial landmark detection methods affect the FPS value. Besides that, we found that the FPS value affects the AUI input results in face landmark detection, eye open-close detection, and head movement detection. Higher FPS values correspond to faster AUI input detection results when executed on a Raspberry Pi 4, while lower FPS values result in slower or delayed AUI input detection. In addition to the superiority FPS value, the average CPU and memory usage is also quite low when running on the Raspberry Pi 4, so in terms of FPS values and task manager results, scenario 3 is superior.

Regarding accuracy testing results, this experiment refers to the tasks outlined in Table 2 and detection indicator in Table 3. Each task was tested in three scenarios in turn by the participants, and the pitch and yaw detection angles are measured based on head movements, as illustrated in Figure 8. For pitch detection angles, representation includes front (0°), up (0° to 80°) and down (0° to -80°), while yaw detection angles represent front (0°), right (0° to 90°) and left (0° to -90°). The AUI input test results for facial landmark detection based on Task 1 in the pitch and yaw sections are presented in Tables 6 and 7.

12.png

Figure 12. Real-time FPS readings on the Raspberry Pi 4

Table 6. AUI input results on face landmark detection based on task 1 on the pitch side

Table 7. AUI input results on face landmark detection based on task 1 in yaw side

As shown in Tables 6 and 7, when 15 participants performed the test with the position facing the monitor screen (front), in turn, scenarios 1, 2, and 3 demonstrated effective detection of facial landmarks, each achieving an accuracy of 100%. This proves that in the front-facing position, all the proposed scenarios can work to display the face landmark detection indicator as AUI input information related to face detection. However, when testing moving the head up and down (pitch), as well as testing moving the head right and left (yaw), there are differences in facial landmark detection results. This test is carried out in stages based on each detection angle accuracy reading result, as listed in the table.

When 15 participants tested moving their heads up and down (pitch) using scenario 1, the accuracy of detecting facial landmarks was only optimal at a reading angle of 0° to 30° for head movement up and 0° to -30° for head movement down, so if the reading of head movement up exceeds an angle of 30° and head movement down exceeds an angle of -30°, the accuracy of scenario 1 is not optimal for detection and cannot even detect facial landmarks. While the accuracy results for scenario 2 testing are better when compared to scenario 1, when testing moving the head up and down based on scenario 2, the accuracy of detecting facial landmarks can be more optimal at the reading angle of 0° to 60° for head movement up and 0° to -60° for head movement down. In contrast to testing the detection of facial landmarks using scenario 3, scenario 3 is superior in detection accuracy when compared to scenarios 1 and 2, because when 15 participants tested with scenario 3, the accuracy of detecting facial landmarks could be optimal at reading angles of 0° to 80° for head movements up and 0° to -80° for head movements down, so there were no facial landmark detection failures when participants tested with scenario 3.

Similar to the test of moving the head right and left (yaw) as shown in Table 6, the accuracy of detecting facial landmarks using scenario 3 is superior when compared to scenario 1 and scenario 2, the accuracy of detecting facial landmarks can be optimized up to a reading angle of 0° to 90° for right head movements and 0° to -90° for left head movements, and there were no facial landmark detection failures when 15 participants moved their heads left and right when testing using scenario 3. Therefore, based on Task 1 experiments, AUI input using scenario 3 exhibits superior face landmark detection (Mediapipe based) compared to scenarios 1 and 2, as illustrated in Figure 13. This figure provides an example of a sample comparison of scenarios 1, 2, and 3 at a detection angle of -90° yaw, indicating that scenarios 1 and 2 fail to detect the AUI input on the side of the -90° yaw detection angle, while scenario 3 can still detect the AUI input.

In testing the accuracy of eye condition detection, the test stages are not much different from testing face landmark detection. When participants are asked to move their heads up and down (pitch) and move their heads right and left (yaw), we observe the results of open eye condition detection and closed eye condition detection as described in task 2. The EAR method applied to detect eye conditions was tested in scenarios 1, 2, and 3. The test results show that the accuracy of open eye condition detection in scenario 3 is superior when compared to scenarios 1 and 2, when participants move their heads up and down and move their heads right and left, as shown in Tables 8 and 9.

As for the results of testing the detection of closed eye conditions, scenarios 1, 2, and 3 have the same accuracy value when the participant moves the head up and down and moves the head right and left, as seen in Tables 10 and 11, so that when the head movement up exceeds the angle of 30° and the head movement down exceeds the angle -30°, as well as the right head movement exceeding the angle 30° and the left head movement exceeding the angle -30°, the system cannot detect closed eye conditions.

The use of the EAR method and participant testing enable the classification of open and closed eye conditions. By setting the EAR threshold value to 0.25, a detection result indicating a closed eye condition is established if the EAR value is ≤ 0.25, whereas an open eye condition is indicated if the EAR value is ≥ 0.25, as shown in Figure 14. Additionally, despite the presence of five participants wearing glasses, experimental results indicate that this factor does not impact the accuracy of eye condition detection. The system is thus proposed to reliably detect both open and closed eye conditions as AUI input, even when participants are wearing glasses.

Table 8. AUI input results on open eye condition detection based on task 2 on the pitch side

Table 9. AUI input results on open eye condition detection based on task 2 in yaw side

Table 10. AUI input results on closed eye condition detection based on task 2 on the pitch side

Table 11. AUI input results on closed eye condition detection based on task 2 in yaw side

13.png

Figure 13. An example of scenario 1, 2 and 3 comparison at -90⁰ yaw detection angle

Concerning AUI input testing for head movement detection, representing the direction of view encompassing forward, upward, and downward views (pitch), as well as left and right views (yaw), it is observed that scenario 3 outperforms scenarios 1 and 2. This is evident in Tables 12 and 13, which present the results of participant testing for Task 3, aligning with the results obtained from Task 1 testing on face landmark detection testing on face landmark detection testing.

The use of the 2D-3D head pose estimation method and analyzing pitch and yaw rotation axis values, the researchers successfully classified head movements as AUI input for view direction detection, especially by using scenario 3. The researchers defined front view direction detection when both the pitch and yaw rotation axis values were ≤ 15 and ≥ -15. For upward viewing direction detection, they considered pitch axis values ≥ 15, while downward viewing direction detection was defined for pitch values ≤ -15. Similarly, leftward viewing direction detection was designated for yaw values ≥ 15, and rightward viewing direction detection for yaw values ≤ -15. Figure 15 illustrates one of the sample head movement detection results, representing the detection of view direction as an AUI input.

From the experiment results, all three proposed scenarios proved capable of detecting viewing direction as AUI input. Although the scenarios varied in accuracy, scenario 3 demonstrated superior performance in reading head movements as the direction of view in pitch and yaw angle readings, so that Mediapipe-based facial landmark detection with scenario 3 has the advantage in every test that has been done.

14.png

Figure 14. One of the sample readout graphs of open and closed eye condition detection via the EAR method

15.png

Figure 15. One of the sample head movement detection results representing view direction detection as AUI input

Table 12. AUI input results on head movement detection representing the view direction based on task 3 on the pitch side

Table 13. AUI input results on head movement detection representing the view direction based on task 3 on the yaw side

The exploration of AUI input through the Raspberry Pi 4 covering facial landmark detection, eye opening and closing detection, and head movement detection yielded 8 functional AUI inputs aligned with indicators and predefined tasks. The performance of the Raspberry Pi 4 embedded system exhibited variations based on the employed methods and scenarios. Notably, the test results highlighted that the AUI input system in scenario 3 (Mediapipe-based) outperformed scenarios 1 and 2 in terms of both performance and accuracy. Therefore, the exploration of AUI input through Raspberry Pi 4 with scenario 3 can serve as a reference for designing AUI in diverse forms of interaction, particularly to support the visual attention system design in computer vision-based systems.

Based on the conducted experiments, although the Raspberry Pi 4 embedded system can support input AUI computer vision-based systems, several experiments are needed to obtain a method that can work optimally on the Raspberry Pi, as has been done in this study, so as to obtain the best performance of the system. In addition, the results of this study prove that the AUI system using Raspberry Pi has the advantage of designing a visual attention-based AUI system that can support various systems with small media, limited voltage sources, and low cost. This is crucial for utilizing it as a parameter for user interface design in various forms of interaction, including HCI, HRI, and various other interaction systems. Because, in 2024, the price of the Raspberry Pi 4 from $35 [49], it can be a consideration besides using desktop computers. However, it cannot be denied that desktop computers have a larger CPU and memory when compared to the Raspberry Pi, especially desktop computers equipped with GPU.