Multimodal System of Ambient Assistance Services for Human Activity Monitoring

Activity detection and classification using various sensors has become a revolutionary technology for real-time, autonomous monitoring in behavior analysis, assisted living, activities of daily living, elderly care, rehabilitation, entertainment and surveillance in smart home environments. Wearable devices, smartphones and ambient environment devices are equipped with various sensors such as accelerometers, gyroscopes, magnetometers, presence sensors, heart rate sensors, pressure sensors and wearable cameras for activity detection and monitoring.

Several contemporary researchers, have published studies on or developed fall detection systems [3]; these have been categorized from the perspective of "portability" or "sensor type". As reported in the study [4], 186 out of 197 published works on fall detection systems develop wearable devices. Most of these devices are accelerometer-based, while others incorporate gyroscopes to obtain information on the patient's position. Biomedical sensors can also be included for fall detection. Currently, with the development of Smartphones incorporating accelerometers and gyroscopes, the latter are also being used in a number of research projects.

2.1 Single sensor systems

Chen et al. [5], Wu et al. [6] have developed similar wireless, triaxial accelerometer-based, waist-worn fall detection systems. The latter automatically distinguishes between falls and ADLs and communicates with a care or monitoring center. Similarly, Palmerini et al. [7] have exploited acceleration signals recorded by an inertial sensor, from continuous real-world monitoring of subjects at moderate to high risk of falling. Of the various methods proposed, the most promising, based on SVM and a multi-phase fall model, achieved a sensitivity of over 80%. These results expand the boundaries of knowledge about fall detection in real-life environments, while proposing relevant metrics for evaluating fall detection systems in real-life contexts.

More recently, the wearable camera has been the subject of research with a view to its integration into fall detection systems-monitoring is then not limited to confined areas and extends wherever the subject may move (indoors and outdoors) while reducing the problems of user privacy intrusion [8]. Ozcan et al. [9] proposed a fall detection and activity classification system using a camera placed at the person's belt; the proposed fall detection algorithm uses histograms of oriented gradients (HOG) and an optical flow-based activity classification method. In addition, Boudouane et al. [10] proposed a fall detection system using a camera (Rasbcam) worn at the user's hips. The introduction of the optical flow technique reduced the false positive rate, observed mainly for rotation scenarios using the HOG alone.

2.2 Multisensor systems

New trends are emerging in the implementation of fusion strategies to combine sensor data, features and classifiers to bring diversity, offer better generalization and solve challenging problems. In the study [11], the authors provide a thorough and comprehensive review of data fusion techniques and multiple classification systems for human activity recognition, with a focus on mobile and wearable devices. Sensor fusion can be performed on data from wearable sensors (accelerometers, gyroscopes), ambient sensors (cameras, vibration, microphones, IR), or both (camera, IR, accelerometer).

Wang et al. [12], proposed a low-power fall detector using triaxial accelerometry and barometric pressure detection, worn around the neck using a lanyard. Using an RF link, the system can transmit the fall alarm to a remote operator installed in the user's home. The latter then relays the alarm to a remote care center to trigger a care intervention. Huynh et al. [13] and Guo et al. [14], used a wearable device with an integrated accelerometer and triaxial gyroscope for fall detection; they used an algorithm based on a simple threshold method and was implemented on a microcontroller, to check whether the person has fallen or not. Jefiza et al. [15] designed a fall detection device to minimize post-fall risk based on an MPU6050 sensor with a 3-axis accelerometer and 3-axis gyroscope. This research recognizes different types of falls as well as different ADLs such as sitting, sleeping, crouching, going up and down stairs and praying with an accuracy of 98.18%. Kerdjidj et al. [16] propose an efficient system for automatic detection of falls and various ADLs. The system is based on a portable Shimmer device, with an on-board accelerometer and gyroscope, which transmits inertial signals to a computer via a wireless connection; the resulting system achieved 99.8% accuracy.

2.3 Smartphone based systems

With the development of Smartphones integrating accelerometers, gyroscopes, cameras and communication tools, these are also used in several research works [17, 18]. In the study [19], the authors propose a fall detection system based on a Smartphone's accelerometer data from two publicly available databases. An eccentric approach with a one-class classification based on SVM was used, resulting in a sensitivity of 98.81% and a specificity of 98.65%. In this context, Hakim et al. [20] detect a fall using the inertial sensors of a body-worn smartphone, with signals transmitted wirelessly to a remote PC for processing. Chen et al. [21] proposed a fall detection system based on a wrist-worn accelerometer. Experiments conducted on simulated falls and real ADL data from two different groups of volunteers yielded an accuracy of 97.45% and 97.82%, respectively. Sanchez and Muñoz [22] presented a multi-sensor, wrist-worn fall detection system that uses an artificial neural network (ANN) to differentiate between falls and ADLs. Three types of sensor - an accelerometer, a gyroscope and a magnetometer - were used in the prototype. Tests carried out under laboratory conditions gave an accuracy, sensitivity and specificity of 98.10%, 98.10% and 98.10% respectively. The results suggest that reliable fall detection is achievable using the generalization capability of the neural network, eliminating the need for complex feature extraction algorithms. This seems a promising approach with the potential to improve user safety.

2.4 Systems with ambient sensors

Ambient devices enable the monitoring of people in a confined environment such as the home, care homes or even smart housing for health; They collect data on the user's daily activities and detect any falls. These devices include: cameras, infrared sensors, vibration sensors, acoustic sensors, contact detectors placed on doors/windows and temperature sensors [23, 24].

In the work developed by Chaccour et al. [25], falls are estimated by monitoring the pattern of recorded vibrations and comparing it with those of habitual movements such as walking; the alarm is triggered if a fall is detected. Li et al. [26] use a multi-microphone system for fall detection based on sound recognition.

In addition, there are other localization methods, such as the installation of infrared sensors [27]. Sixsmith et al. [28] installed pyroelectric infrared sensors on walls to detect subject activity.

2.5 Biomedical sensor based systems

Continuous monitoring of heart rate can reduce the risk of accidents due to heart failure. To achieve this objective, several heart rate sensors have been developed. Mahmood et al. [29] developed a system that monitors an athlete's heart rate during training or exercise using a finger pulse meter. The pulsometer sends the collected data to a wrist strap, and is used as a heart rate monitor so that athletes and coaches can monitor their fitness level.

Biomedical sensors can also be integrated for fall detection; a wearable fall detection system using a triaxial accelerometer and pulsometer was developed in the study [30]. Wang et al. [31] proposed an improved fall detection system for monitoring the elderly, based on body-worn smart sensors and operating via consumer home networks. Using information gathered by an accelerometer, cardiotachometer and smart sensors, fall impacts can be recorded and distinguished from normal daily activities.

Emerging trends in fall detection systems include the growing adoption of machine learning for improved accuracy, the integration of diverse sensor technologies for more comprehensive data, and the increasing use of wearable devices for unobtrusive monitoring and effective alerts. These advances aim to improve fall detection systems' reliability and accessibility.

2.6 Discussion

As previously reported, portable devices are the most commonly used in the field of fall detection because of their ability to be used both indoors and outdoors. The accelerometer is a good fall detector with a low cost and it is better accepted by users due to its small size. A portable camera can also be used to detect a fall. Ambient devices are also characterized by low cost and are much less cumbersome than portable devices, since they are placed in the environment; however, when using them, the parameters measured are not specific only to the person being monitored.

Heart rate monitoring can be done for all types of people, healthy people, athletes and people with heart disease. As well as this monitoring is done for any activity such as: sleeping, exercising, walking and falling. The most commonly used devices for this monitoring are finger pulse meters, although their constant wearing by users is inconvenient. Belts worn around the chest are increasingly improved to give more accurate information but their cost remains high. Oximeters placed on the ear can reduce the discomfort caused by portable devices with a low cost.

Integrating fall detection systems raises important ethical issues, particularly as relates to privacy, especially for the elderly and vulnerable populations. Wearable and ambient devices collect sensitive data, requiring rigorous privacy policies and clear consent mechanisms, while aiming to minimize privacy intrusion and maximize security.

We can remark that the work on heart rate monitoring and posture detection is mainly based on the use of accelerometers and pulsometers. The systems developed make it possible to detect falls and/or heart disorders, but do not allow their simultaneous detection. The system proposed in this paper allows the simultaneous detection of heart rate disorders and falls from a standing position, and the location of the person in his home. When designing the system, we particularly focused on its acceptability by the user, as well as its comfort when worn. The fall detection service uses a portable device based on a tri-axial accelerometer placed at the belt. The heart disorder detection service monitors the heart rate using a pulsometer placed at the person's ear. The location is detected with photoelectric sensors placed at the home doors.

The fusion system processes the data from the various portable and ambient sensors. Fusion is applied at two levels, a low level and a high level. The "low level" of fusion is the processing of data from each modality separately to generate three ambient assistance services: the fall detection service, the heart disorder detection service and the localization service. The "high-level" fusion is performed on the information from these three services, and generates the emergency service.

4.1 Low-level fusion

By collecting data on absolute body acceleration, heart rate and presence detection, we can detect accidents that may occur during ADLs. Input modalities represent the first module of our multimodal architecture. This module contains three pieces of sensory information collected from two devices worn by the user and eight sensors placed in the environment. This information constitutes the three input modalities: absolute acceleration, heart rate and presence detection.

4.1.1 Fall detection

- Modality I1: Acceleration

In this work, the fall is considered as a "sudden event" bringing the person to the ground. When a real fall occurs, the collision between the human body and the ground produces an obvious peak value of absolute acceleration; we exploit this feature to determine falls using an accelerometer. We chose the ADXL362 triaxial accelerometer from Analog Devices, which we placed at hip level using a belt; the choice of this location is justified by the fact that the hips, unlike the rest of the trunk, constitute the zone closest to the center of mass [32].

The magnitude of the linear acceleration A is obtained, according to Shinde and Chawan [33], by relation (1) according to the values of the acceleration components Ax, Ay and Az, delivered by the triaxial accelerometer.

$|A t|=\sqrt{\left(A_x^2+A_y^2+A_z^2\right)}$             (1)

where, A_x, A_y and A_z represent the body's accelerations according to the orthogonal axes X, Y and Z respectively, and |At| is measured as a function of the Earth's surface acceleration 'g'.

- Fall detection service

The proposed system identifies a fall by evaluating the two phases, critical and post-fall, shown on Figure 2. During the critical phase of a fall, the body's vertical velocity increases linearly with time, and this phase contains a free-fall stage followed by the ground impact stage [34, 35]. We detect the impact of the body on the ground and the duration of the post-fall phase in order to determine whether a fall has occurred.

2.png

Figure 2. Acceleration prototype of a fall

Free fall (T0): the fall starts with a free fall where the acceleration is less than 1g and more precisely less than 0.7g.

Impact (T1): Impact occurs when the body hits the ground with an acceleration greater than or equal to 3g.

Extended position (t>T2): the body remains elongated with an acceleration of around 1 g and an orientation close to 90° with respect to the start of free fall (T0).

After a fall, the person often remains motionless in an extended position. Lack of movement can therefore be the consequence of a fall. This can be detected by calculating the time interval during which the person has remained in an extended position. In the study [36], the authors exploited the person's immobilization time during the post-fall phase to reduce the false positive rate. The proposed fall detection service receives accelerometer values At(t), At(t+1), ...... At(t+n), .... every 500 ms. and compares them with critical values.

These critical values are as follows:

- impact detection threshold=3g;

- free-fall detection threshold=0.7g;

- extended position detection=1g.

where, g=9.81 m/s represents the gravity acceleration on the earth's surface.

4.1.2 Heart disorder detection

- Modality I2: Heart frequency

The proposed device for continuous heart rate monitoring is a pulse oximeter. This is clipped to the user's ear, as shown in Figure 3, and is less bulky and intrusive than finger-level pulsometers.

We chose to use the Grove-Ear-clip heart rate sensor from "Seeed studio", consisting of an ear clip and a receiver module linked by a 120 cm long wire. We have placed the receiver module with the accelerometer at belt level to make the system less bulky.

The Grove-Ear-clip measures heart rate per minute (Fc). For each heartbeat, it delivers the number of heartbeats in the previous minute.

- Heart disorder detection service

We propose to monitor the user's heart rate using an oximeter, with the aim of detecting abnormal heart function, i.e., tachycardia and bradycardia, during activities of daily living. Detection of these disorders depends on the category of the population we wish to monitor; the heart rate norms for each category are given by the Table 1.

3.png

Figure 3. Pulse oximeter clipped to ear

Table 1. Heart rate standards [37]

Our system is dedicated to the elderly, with a minimum heart rate of 50 beats/min and a maximum of 100 beats/min. For this service, we offer to merge heart rate 'Fc' data from the pulse oximeter.

To avoid triggering false detections of heart disorders or overlooking true detections, we propose to evaluate the heart rate value for ten '10' seconds, i.e., for ten successive 'Fc' samples.

4.1.3 Presence detection

- Modality I3: Presence detection

The proposed architecture is dedicated to elderly people living alone, and is implemented in an environment composed of just four rooms: a living room, a bedroom, a kitchen and a bathroom, distributed around a hall. In order to detect the user's location in the home, we propose to place two photoelectric sensors at each door in the home [38], for a total of eight sensors. In order to detect that the person has actually passed through the door, as well as the direction of passage, we have adopted the configuration shown in Figure 4. We have placed two photoelectric sensors, each consisting of a transmitter and a receiver, at each door. The first sensor is located on the external side, while the second is on the internal side. In order to avoid any confusion between the detection of the person being tracked and that of a pet, for example, we have placed the detector at an average height of one '1' meter above the ground.

- Location service

This service detects a person's movement from one room to another, by merging data collected from photoelectric sensors placed around the home at door level. The transmitter of each sensor sends a light beam to the receiver, which sends a logic output equal to '1' if no object crosses the beam, and equal to '0' if the person passes through the door and cuts the beam. In our case, we use the detector in sampled mode, reading these values every 500 ms. We say that the person has passed through a room door on the way in if, and only if, the receiver on the external side has generated a logic output equal to '0' before the one on the internal side. Similarly, when the person has left the room through the door, the receiver on the internal side generates a '0' output first. After merging the outputs from the eight sensors, we can determine the room number (noted Rnb) where the person is located.

4.png

Figure 4. Positioning of photoelectric sensors on door frame

Sensor data from the accelerometer, oximeter and presence detector are transmitted via an interface card to a computer. An algorithm based on sliding average calculation is used to guarantee data reliability.

4.2 Height level fusion

After a fusion at the level of each of the three ambient assistance services, another fusion is applied to the results from the latter. The detection of a fall and/or a heart condition with location triggers an emergency service whose role is to merge the events from the three services and then produce the appropriate action.

4.2.1 Emergency service

We have chosen to perform decision-level fusion for this service. In our case, to solve all possible cases and give a better result, we propose to combine this fusion with rule-based conditional fusion (IF THEN ELSE) to derive outputs according to the detection or non-detection of one or more hazardous situations [39]. This method allows us to estimate moving objects such as a person during ADLs and ensure good synchronization between the different assistance services.

This service takes as input the results of the three ambient assistance services and as output the state of the person with his location in the habitat. The outputs can then be: "detection of a fall without detection of the cardiac disorder", "detection of the cardiac disorder without detection of a fall" or "detection of a fall with detection of the cardiac disorder". The information provided by this service, such as the detection of falls and/or cardiac disorders, is then sent to an emergency center, which is responsible for generating the output modalities according to the situation and the danger detected.

4.2.2 Output modalities

Once the alert has been sent by the emergency service to the emergency center, the operator decides to request an intervention or not. Exit modes that can be set are:

Modality 1: a call to the user's telephone is made by the emergency center operator to speak to the user and enquire about his or her condition. Depending on the response, the operator may decide to request an on-site intervention or activate other procedures.

Modality 2: the operator sends the message "Are you OK?" to the user; the response to this message can be "Positive" or "Negative". In the case of false risk detections, the user replies "Positive" and the operator considers the user to be in a normal state. If the response is "Negative", the operator activates other modalities, such as a call.

Modality 3: an ambulance is requested urgently by the operator in the event of anomalies being detected that are dangerous to the user's health, or if the user does not respond to calls.

Modality 4: the emergency order classification ranges from a minimum value of '0' for normal situations to a maximum value of '1' for dangerous situations. Between the two values, the operator can classify the urgency with a percentage between 0% and 100%, after a conversation with the monitored person.

In addition, our system records the history of any anomalies detected, and stores all data concerning the user's condition in a database so that the doctor can consult the file at any time.

Validation is an important step in the process of multimodal architecture development. It allows showing that the system responds positively to the specifications requirements, especially to the real-time response of the system. It also makes it possible to check whether the method used in this work is functional in the simulation environment.

6.1 Experimental context

In order to validate and evaluate the proposed system in terms of accuracy, we conducted several experimental tests. All of these tests were conducted in a non-hospital environment and by volunteers in good health. During the tests, we ensured the volunteers' safety by having them fall onto a mattress in the presence of a doctor, who instructed them on how to fall.

We solicited six (06) volunteers, who were 30 to 70 years old and had no heart disorders or physical handicaps, to wear the device. Each subject performed ten (10) tests, which were free-falling from a standing position or ADLs; some of which may be similar to falls such as picking up an object from the ground or sitting down suddenly.

This allowed us to achieve four scenarios:

Scenario 1: We detached the oximeter from the volunteer's ear to perform the "Oximeter detachment" scenario.

Scenario 2: We asked the volunteer to walk up and down the stairs, jump and run to have a heart rate above 'Fcmax' and detect tachycardia.

Scenario 3: The volunteer free-falled and stayed on the ground for more than two minutes before getting up again.

Scenario 4: After the volunteer fell, we asked the volunteer to get up quickly.

Tests involving final users of the developed system require it to be worn by individuals in care residences for the elderly or in hospital settings.

In what follows, we present, among the different scenarios carried out, the results relating to Scenario 4 " Fall detection without cardiac disorder ", introducing accelerometer and oximeter values into the models developed in CPNTools.

6.2 Experimental results

The values of the absolute acceleration 'At' provided by the accelerometer are extracted every 500 ms, which led us to have 1200 values for the ten minutes of the test. In Figure 9, we have represented only the part between the moments 500/2 s and 850/2 s in order to better visualize the critical event.

We can distinguish: free-fall at t=521/2 s, impact at t=522/2 s and start of extended position at t=523/2 s. The duration of the extended position is more than one minute (from t=523/2 s to t=805/2 s), which means that the volunteer actually remained on the ground after the fall.

9.jpg

Figure 9. Absolute acceleration 'At' generated by the accelerometer

The pulse oximeter generates one value every second, giving 600 values 'Fc' for the ten-minute test. In Figure 10 we present only the part in the interval 100s and 500s, when the fall occurred.

For the 600 generated values we did not obtain ten (10) successive values of 'Fc' greater than 'Fcmax' or less than 'Fcmin', so no disorder is detected. We can see that the values of 'Fc' slightly exceed 'Fcmax' for t=262s, which means that the fall occurred at the same instant.

10.png

Figure 10. Heart rate 'Fc' generated by the oximeter

During the tests, when the subject moved from one room to another, photoelectric detectors placed on the doors generated outputs, which, once processed, gave his location. We have represented in Figure 11, the 1200 values corresponding to the room number 'Rnb' where the person is located.

11.jpg

Figure 11. Room number 'Rnb'

As shown in the Figure 11, at each passage from one room to another, the location 'Rnb' takes the value '0' corresponding to the passage through the hall. At the time t=523/2 s corresponding to the fall, the volunteer is in the room with the number '1', corresponding to the living room.

According to the three Figures 9, 10 and 11, we can see that the fall point appeared at time 512/2 s on the acceleration graph, corresponding to time t=256 s on the oximeter graph, which coincides with the location of point Rnb3 in the room number graph. By analyzing these three quantities, we can conclude that a fall has occurred.

After retrieving the results extracted from the three ambient assistance services, the emergency service merges them to deliver the appropriate message. The result delivered by the emergency service, for the scenario considered, is shown in Figure 12.

Among the three possible codes for this scenario (Code 5, 6 and 7), the emergency service transmitted Code 6 for this case. This code was transmitted for sample 523, and the message sent is 'Fall without Heart problem, Person on the floor, Intervention, Emergency, Operator order, Code 6, Person in living room'.

Through this message we can deduce that at the moment t= 523/2 s, the volunteer was in the living room and made a free-fall. He remained on the ground for more than one minute. The operator decided, after a telephone conversation, to activate an emergency.

After testing each scenario several times, we were able to verify the four scenarios by obtaining the appropriate codes. An evaluation was carried out for each of the proposed services on the system.

12.jpg

Figure 12. Sent message and code number