Multimodal System of Ambient Assistance Services for Human Activity Monitoring

Multimodal System of Ambient Assistance Services for Human Activity Monitoring

Isma Boudouane* Amina Makhlouf Nacereddine Djelal Nadia Saadia Amar Ramdane-Cherif

LRPE Laboratory, University of Sciences and Technology Houari Boumediene, Algiers 16111, Algeria

LISV Laboratory, University of Versailles Saint-Quentin-en-Yvelines, Velizy78140, France

Corresponding Author Email:
11 August 2023
11 October 2023
21 October 2023
Available online: 
31 October 2023
| Citation

© 2023 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (



The global demographic has seen a significant surge in the population aged 65 or over in recent years, a trend projected to accelerate in the coming decades. This elderly demographic is progressively losing autonomy, becoming increasingly susceptible to domestic accidents such as falls and heart rhythm abnormalities. To address this, this article introduces a multimodal system designed for continuous monitoring of elderly or disabled individuals within their homes. The developed architecture hinges on a fusion system, integrating signals from acceleration, heart rate, and presence sensors to generate ambient services. These services enable simultaneous detection of heart rate irregularities and falls, as well as tracking the individual's location within their home. Our methodology proposes a conditional fusion, employing IF THEN ELSE rules to produce outputs correlated to the presence or absence of one or more critical situations. This strategy amplifies the accuracy of moving object estimation, particularly during activities of daily living (ADLs), and ensures synchronized assistance services. An emergency service is introduced to classify the urgency and initiate the appropriate action. Validation of the proposed architecture and performance analysis were conducted using the CPNTools tool. Experimental test-based results demonstrated a 78.33% accuracy in the fall detection service. Heart disorder detection service tests confirmed 100% success rate in detecting tachycardia, while the location service demonstrated a sensitivity of 90%.


ambient services, fall detection, heart disorder detection, human activity monitoring, multimodal system, portable system, sensors fusion

1. Introduction

According to data from World Population Prospects 2022 [1], the number of people aged 65 and over has increased significantly in recent years and is expected to accelerate in the coming decades. This elderly population is becoming less independent and more exposed to the risk of domestic accidents, especially indoors. Among these accidents, falling is a major public health disorder that affects several tens of millions of elderly in the world every year, with immediate, fatal consequences, but also disabling complications, both physical and psychological. In most cases, the elderly person cannot get up alone after a fall, so it is necessary to react very quickly and therefore to be able to detect this event as soon as possible. In addition, considering heart rhythm disorders as a risk factor for elderly falls would be appropriate [2].

These challenges are essential for developing effective systems with applications in health, safety and improving people's quality of life. It is therefore imperative to develop real-time systems for automatic recording of physical activity and heart rate variability.

In this context, a multimodal structure is proposed in this paper to enable the elderly and disabled to be monitored continuously in their homes. This structure provides three ambient assistance services: the fall detection service, the heart disorder detection service and the location service.

These services make it possible to detect simultaneously heart rate disorders and falls from a standing position and to locate the person in home. Indeed, cardiac arrhythmias can lead to sudden and significant spikes in blood pressure. These spikes can make individuals more susceptible to dizziness and loss of balance, increasing the risk of falls. A fourth service, called the emergency service, is also offered to categorize emergency situations and determine the appropriate action plan based on the person's situation. This may involve contacting the emergency services, activating a medical alert or implementing other specific measures, depending on the circumstances.

2. Background

Activity detection and classification using various sensors has become a revolutionary technology for real-time, autonomous monitoring in behavior analysis, assisted living, activities of daily living, elderly care, rehabilitation, entertainment and surveillance in smart home environments. Wearable devices, smartphones and ambient environment devices are equipped with various sensors such as accelerometers, gyroscopes, magnetometers, presence sensors, heart rate sensors, pressure sensors and wearable cameras for activity detection and monitoring.

Several contemporary researchers, have published studies on or developed fall detection systems [3]; these have been categorized from the perspective of "portability" or "sensor type". As reported in the study [4], 186 out of 197 published works on fall detection systems develop wearable devices. Most of these devices are accelerometer-based, while others incorporate gyroscopes to obtain information on the patient's position. Biomedical sensors can also be included for fall detection. Currently, with the development of Smartphones incorporating accelerometers and gyroscopes, the latter are also being used in a number of research projects.

2.1 Single sensor systems

Chen et al. [5], Wu et al. [6] have developed similar wireless, triaxial accelerometer-based, waist-worn fall detection systems. The latter automatically distinguishes between falls and ADLs and communicates with a care or monitoring center. Similarly, Palmerini et al. [7] have exploited acceleration signals recorded by an inertial sensor, from continuous real-world monitoring of subjects at moderate to high risk of falling. Of the various methods proposed, the most promising, based on SVM and a multi-phase fall model, achieved a sensitivity of over 80%. These results expand the boundaries of knowledge about fall detection in real-life environments, while proposing relevant metrics for evaluating fall detection systems in real-life contexts.

More recently, the wearable camera has been the subject of research with a view to its integration into fall detection systems-monitoring is then not limited to confined areas and extends wherever the subject may move (indoors and outdoors) while reducing the problems of user privacy intrusion [8]. Ozcan et al. [9] proposed a fall detection and activity classification system using a camera placed at the person's belt; the proposed fall detection algorithm uses histograms of oriented gradients (HOG) and an optical flow-based activity classification method. In addition, Boudouane et al. [10] proposed a fall detection system using a camera (Rasbcam) worn at the user's hips. The introduction of the optical flow technique reduced the false positive rate, observed mainly for rotation scenarios using the HOG alone.

2.2 Multisensor systems

New trends are emerging in the implementation of fusion strategies to combine sensor data, features and classifiers to bring diversity, offer better generalization and solve challenging problems. In the study [11], the authors provide a thorough and comprehensive review of data fusion techniques and multiple classification systems for human activity recognition, with a focus on mobile and wearable devices. Sensor fusion can be performed on data from wearable sensors (accelerometers, gyroscopes), ambient sensors (cameras, vibration, microphones, IR), or both (camera, IR, accelerometer).

Wang et al. [12], proposed a low-power fall detector using triaxial accelerometry and barometric pressure detection, worn around the neck using a lanyard. Using an RF link, the system can transmit the fall alarm to a remote operator installed in the user's home. The latter then relays the alarm to a remote care center to trigger a care intervention. Huynh et al. [13] and Guo et al. [14], used a wearable device with an integrated accelerometer and triaxial gyroscope for fall detection; they used an algorithm based on a simple threshold method and was implemented on a microcontroller, to check whether the person has fallen or not. Jefiza et al. [15] designed a fall detection device to minimize post-fall risk based on an MPU6050 sensor with a 3-axis accelerometer and 3-axis gyroscope. This research recognizes different types of falls as well as different ADLs such as sitting, sleeping, crouching, going up and down stairs and praying with an accuracy of 98.18%. Kerdjidj et al. [16] propose an efficient system for automatic detection of falls and various ADLs. The system is based on a portable Shimmer device, with an on-board accelerometer and gyroscope, which transmits inertial signals to a computer via a wireless connection; the resulting system achieved 99.8% accuracy.

2.3 Smartphone based systems

With the development of Smartphones integrating accelerometers, gyroscopes, cameras and communication tools, these are also used in several research works [17, 18]. In the study [19], the authors propose a fall detection system based on a Smartphone's accelerometer data from two publicly available databases. An eccentric approach with a one-class classification based on SVM was used, resulting in a sensitivity of 98.81% and a specificity of 98.65%. In this context, Hakim et al. [20] detect a fall using the inertial sensors of a body-worn smartphone, with signals transmitted wirelessly to a remote PC for processing. Chen et al. [21] proposed a fall detection system based on a wrist-worn accelerometer. Experiments conducted on simulated falls and real ADL data from two different groups of volunteers yielded an accuracy of 97.45% and 97.82%, respectively. Sanchez and Muñoz [22] presented a multi-sensor, wrist-worn fall detection system that uses an artificial neural network (ANN) to differentiate between falls and ADLs. Three types of sensor - an accelerometer, a gyroscope and a magnetometer - were used in the prototype. Tests carried out under laboratory conditions gave an accuracy, sensitivity and specificity of 98.10%, 98.10% and 98.10% respectively. The results suggest that reliable fall detection is achievable using the generalization capability of the neural network, eliminating the need for complex feature extraction algorithms. This seems a promising approach with the potential to improve user safety.

2.4 Systems with ambient sensors

Ambient devices enable the monitoring of people in a confined environment such as the home, care homes or even smart housing for health; They collect data on the user's daily activities and detect any falls. These devices include: cameras, infrared sensors, vibration sensors, acoustic sensors, contact detectors placed on doors/windows and temperature sensors [23, 24].

In the work developed by Chaccour et al. [25], falls are estimated by monitoring the pattern of recorded vibrations and comparing it with those of habitual movements such as walking; the alarm is triggered if a fall is detected. Li et al. [26] use a multi-microphone system for fall detection based on sound recognition.

In addition, there are other localization methods, such as the installation of infrared sensors [27]. Sixsmith et al. [28] installed pyroelectric infrared sensors on walls to detect subject activity.

2.5 Biomedical sensor based systems

Continuous monitoring of heart rate can reduce the risk of accidents due to heart failure. To achieve this objective, several heart rate sensors have been developed. Mahmood et al. [29] developed a system that monitors an athlete's heart rate during training or exercise using a finger pulse meter. The pulsometer sends the collected data to a wrist strap, and is used as a heart rate monitor so that athletes and coaches can monitor their fitness level.

Biomedical sensors can also be integrated for fall detection; a wearable fall detection system using a triaxial accelerometer and pulsometer was developed in the study [30]. Wang et al. [31] proposed an improved fall detection system for monitoring the elderly, based on body-worn smart sensors and operating via consumer home networks. Using information gathered by an accelerometer, cardiotachometer and smart sensors, fall impacts can be recorded and distinguished from normal daily activities.

Emerging trends in fall detection systems include the growing adoption of machine learning for improved accuracy, the integration of diverse sensor technologies for more comprehensive data, and the increasing use of wearable devices for unobtrusive monitoring and effective alerts. These advances aim to improve fall detection systems' reliability and accessibility.

2.6 Discussion

As previously reported, portable devices are the most commonly used in the field of fall detection because of their ability to be used both indoors and outdoors. The accelerometer is a good fall detector with a low cost and it is better accepted by users due to its small size. A portable camera can also be used to detect a fall. Ambient devices are also characterized by low cost and are much less cumbersome than portable devices, since they are placed in the environment; however, when using them, the parameters measured are not specific only to the person being monitored.

Heart rate monitoring can be done for all types of people, healthy people, athletes and people with heart disease. As well as this monitoring is done for any activity such as: sleeping, exercising, walking and falling. The most commonly used devices for this monitoring are finger pulse meters, although their constant wearing by users is inconvenient. Belts worn around the chest are increasingly improved to give more accurate information but their cost remains high. Oximeters placed on the ear can reduce the discomfort caused by portable devices with a low cost.

Integrating fall detection systems raises important ethical issues, particularly as relates to privacy, especially for the elderly and vulnerable populations. Wearable and ambient devices collect sensitive data, requiring rigorous privacy policies and clear consent mechanisms, while aiming to minimize privacy intrusion and maximize security.

We can remark that the work on heart rate monitoring and posture detection is mainly based on the use of accelerometers and pulsometers. The systems developed make it possible to detect falls and/or heart disorders, but do not allow their simultaneous detection. The system proposed in this paper allows the simultaneous detection of heart rate disorders and falls from a standing position, and the location of the person in his home. When designing the system, we particularly focused on its acceptability by the user, as well as its comfort when worn. The fall detection service uses a portable device based on a tri-axial accelerometer placed at the belt. The heart disorder detection service monitors the heart rate using a pulsometer placed at the person's ear. The location is detected with photoelectric sensors placed at the home doors.

3. Proposed Architecture

The system proposed in this article is related to the development of a multimodal architecture, as shown in Figure 1. This architecture is based on the work presented in [30], whose aim is to carry out continuous monitoring of elderly or disabled people in their homes. The method we have implemented on this architecture is based on two fusion levels: a low-level fusion processing sensor signals, which are then fused at the higher level using an algorithm based on IF, THEN, ELSE conditions, to trigger an emergency service in the case of a serious fall or cardiac problem. The proposed modular architecture detects simultaneously falls and heart disorders, and also allows the location of the person. This system collects data from portable and ambient sensors; this data will be processed and merged by a fusion system in order to generate the required actions. In order to validate the proposed architecture, we modeled it using Stochastic Timed and Colored Petri Nets. The CPNTools tool was used to simulate this architecture and generate fault or error detection scenarios.

Figure 1. Proposed multimodal architecture

As shown in Figure 1, the proposed architecture is modular and extensible for other input modalities, which can be other sensors (gyroscope, camera etc...); or modalities issued from artificial intelligence based machines. This architecture sends only emergency messages or phone calls that do not contain confidential or security information about the user. In addition, this information is transmitted only to the medical emergency center. The architecture is composed of three main modules: the input modalities, the fusion system and the output modalities.

- Module 1: Input modalities

Input modalities are the information collected from continuous monitoring of the daily activities of a person living alone. For this architecture, we consider data from three modalities: absolute acceleration, heart rate and presence detection. To generate these modalities, we propose to use light and less invasive sensors, which are:

·A tri-axial accelerometer to measure the absolute acceleration of the body, which will allow us to detect daily life risks such as a fall.

·A pulse oximeter to calculate the number of heart beats per minute to detect heart rate disorders.

·Photoelectric sensors to detect the presence of the user in his home.

- Module 2: Fusion system

The input modalities send the collected information to a computer for processing. This processing is carried out by a fusion system based on a centralized approach, consisting of two levels of fusion, a low level and a high level.

·The "low level" fusion is based on the signals from the sensors. It is carried out on several ambient sensors or on the data, taken over an interval of time, of a portable sensor. The data from each modality is processed separately to generate the three ambient assistance services.

·The data collected by these three services will, in turn, be merged at a high level to trigger an emergency service. The latter activates the appropriate output modality according to the user's current situation.

- Module 3: Output modalities

The emergency service is charged to provide the appropriate action according to the degree of risk. If an anomaly is detected, this service sends an alert to an emergency center, which produces the various output modalities: calls, messages, ambulance request and urgency order classification.

4. Fusion System

The fusion system processes the data from the various portable and ambient sensors. Fusion is applied at two levels, a low level and a high level. The "low level" of fusion is the processing of data from each modality separately to generate three ambient assistance services: the fall detection service, the heart disorder detection service and the localization service. The "high-level" fusion is performed on the information from these three services, and generates the emergency service.

4.1 Low-level fusion

By collecting data on absolute body acceleration, heart rate and presence detection, we can detect accidents that may occur during ADLs. Input modalities represent the first module of our multimodal architecture. This module contains three pieces of sensory information collected from two devices worn by the user and eight sensors placed in the environment. This information constitutes the three input modalities: absolute acceleration, heart rate and presence detection.

4.1.1 Fall detection

- Modality I1: Acceleration

In this work, the fall is considered as a "sudden event" bringing the person to the ground. When a real fall occurs, the collision between the human body and the ground produces an obvious peak value of absolute acceleration; we exploit this feature to determine falls using an accelerometer. We chose the ADXL362 triaxial accelerometer from Analog Devices, which we placed at hip level using a belt; the choice of this location is justified by the fact that the hips, unlike the rest of the trunk, constitute the zone closest to the center of mass [32].

The magnitude of the linear acceleration A is obtained, according to Shinde and Chawan [33], by relation (1) according to the values of the acceleration components Ax, Ay and Az, delivered by the triaxial accelerometer.

$|A t|=\sqrt{\left(A_x^2+A_y^2+A_z^2\right)}$             (1)

where, Ax, Ay and Az represent the body's accelerations according to the orthogonal axes X, Y and Z respectively, and |At| is measured as a function of the Earth's surface acceleration 'g'.

- Fall detection service

The proposed system identifies a fall by evaluating the two phases, critical and post-fall, shown on Figure 2. During the critical phase of a fall, the body's vertical velocity increases linearly with time, and this phase contains a free-fall stage followed by the ground impact stage [34, 35]. We detect the impact of the body on the ground and the duration of the post-fall phase in order to determine whether a fall has occurred.

Figure 2. Acceleration prototype of a fall

Free fall (T0): the fall starts with a free fall where the acceleration is less than 1g and more precisely less than 0.7g.

Impact (T1): Impact occurs when the body hits the ground with an acceleration greater than or equal to 3g.

Extended position (t>T2): the body remains elongated with an acceleration of around 1 g and an orientation close to 90° with respect to the start of free fall (T0).

After a fall, the person often remains motionless in an extended position. Lack of movement can therefore be the consequence of a fall. This can be detected by calculating the time interval during which the person has remained in an extended position. In the study [36], the authors exploited the person's immobilization time during the post-fall phase to reduce the false positive rate. The proposed fall detection service receives accelerometer values At(t), At(t+1), ...... At(t+n), .... every 500 ms. and compares them with critical values.

These critical values are as follows:

- impact detection threshold=3g;

- free-fall detection threshold=0.7g;

- extended position detection=1g.

where, g=9.81 m/s represents the gravity acceleration on the earth's surface.

4.1.2 Heart disorder detection

- Modality I2: Heart frequency

The proposed device for continuous heart rate monitoring is a pulse oximeter. This is clipped to the user's ear, as shown in Figure 3, and is less bulky and intrusive than finger-level pulsometers.

We chose to use the Grove-Ear-clip heart rate sensor from "Seeed studio", consisting of an ear clip and a receiver module linked by a 120 cm long wire. We have placed the receiver module with the accelerometer at belt level to make the system less bulky.

The Grove-Ear-clip measures heart rate per minute (Fc). For each heartbeat, it delivers the number of heartbeats in the previous minute.

- Heart disorder detection service

We propose to monitor the user's heart rate using an oximeter, with the aim of detecting abnormal heart function, i.e., tachycardia and bradycardia, during activities of daily living. Detection of these disorders depends on the category of the population we wish to monitor; the heart rate norms for each category are given by the Table 1.

Figure 3. Pulse oximeter clipped to ear

Table 1. Heart rate standards [37]


HeartBeats per Min

Adolescent and adult (≥13 years)


Elderly person (≥65 years)






Children (3-12 years)


Baby (1-2 years)


New born


Our system is dedicated to the elderly, with a minimum heart rate of 50 beats/min and a maximum of 100 beats/min. For this service, we offer to merge heart rate 'Fc' data from the pulse oximeter.

To avoid triggering false detections of heart disorders or overlooking true detections, we propose to evaluate the heart rate value for ten '10' seconds, i.e., for ten successive 'Fc' samples.

4.1.3 Presence detection

- Modality I3: Presence detection

The proposed architecture is dedicated to elderly people living alone, and is implemented in an environment composed of just four rooms: a living room, a bedroom, a kitchen and a bathroom, distributed around a hall. In order to detect the user's location in the home, we propose to place two photoelectric sensors at each door in the home [38], for a total of eight sensors. In order to detect that the person has actually passed through the door, as well as the direction of passage, we have adopted the configuration shown in Figure 4. We have placed two photoelectric sensors, each consisting of a transmitter and a receiver, at each door. The first sensor is located on the external side, while the second is on the internal side. In order to avoid any confusion between the detection of the person being tracked and that of a pet, for example, we have placed the detector at an average height of one '1' meter above the ground.

- Location service

This service detects a person's movement from one room to another, by merging data collected from photoelectric sensors placed around the home at door level. The transmitter of each sensor sends a light beam to the receiver, which sends a logic output equal to '1' if no object crosses the beam, and equal to '0' if the person passes through the door and cuts the beam. In our case, we use the detector in sampled mode, reading these values every 500 ms. We say that the person has passed through a room door on the way in if, and only if, the receiver on the external side has generated a logic output equal to '0' before the one on the internal side. Similarly, when the person has left the room through the door, the receiver on the internal side generates a '0' output first. After merging the outputs from the eight sensors, we can determine the room number (noted Rnb) where the person is located.

Figure 4. Positioning of photoelectric sensors on door frame

Sensor data from the accelerometer, oximeter and presence detector are transmitted via an interface card to a computer. An algorithm based on sliding average calculation is used to guarantee data reliability.

4.2 Height level fusion

After a fusion at the level of each of the three ambient assistance services, another fusion is applied to the results from the latter. The detection of a fall and/or a heart condition with location triggers an emergency service whose role is to merge the events from the three services and then produce the appropriate action.

4.2.1 Emergency service

We have chosen to perform decision-level fusion for this service. In our case, to solve all possible cases and give a better result, we propose to combine this fusion with rule-based conditional fusion (IF THEN ELSE) to derive outputs according to the detection or non-detection of one or more hazardous situations [39]. This method allows us to estimate moving objects such as a person during ADLs and ensure good synchronization between the different assistance services.

This service takes as input the results of the three ambient assistance services and as output the state of the person with his location in the habitat. The outputs can then be: "detection of a fall without detection of the cardiac disorder", "detection of the cardiac disorder without detection of a fall" or "detection of a fall with detection of the cardiac disorder". The information provided by this service, such as the detection of falls and/or cardiac disorders, is then sent to an emergency center, which is responsible for generating the output modalities according to the situation and the danger detected.

4.2.2 Output modalities

Once the alert has been sent by the emergency service to the emergency center, the operator decides to request an intervention or not. Exit modes that can be set are:

Modality 1: a call to the user's telephone is made by the emergency center operator to speak to the user and enquire about his or her condition. Depending on the response, the operator may decide to request an on-site intervention or activate other procedures.

Modality 2: the operator sends the message "Are you OK?" to the user; the response to this message can be "Positive" or "Negative". In the case of false risk detections, the user replies "Positive" and the operator considers the user to be in a normal state. If the response is "Negative", the operator activates other modalities, such as a call.

Modality 3: an ambulance is requested urgently by the operator in the event of anomalies being detected that are dangerous to the user's health, or if the user does not respond to calls.

Modality 4: the emergency order classification ranges from a minimum value of '0' for normal situations to a maximum value of '1' for dangerous situations. Between the two values, the operator can classify the urgency with a percentage between 0% and 100%, after a conversation with the monitored person.

In addition, our system records the history of any anomalies detected, and stores all data concerning the user's condition in a database so that the doctor can consult the file at any time.

5. Implementation

For the architecture modeling, we use the Timed and Stochastic Colored Petri Nets (CPN), simulated with CPNTools version V4.0.0 [40]. The programming language used for this modeling is CPN ML which is an extension of the standard ML language. The architecture composed of the three modules, input modalities, merge system and output modalities, is shown in Figure 5.

The CPN model developed contains twelve places (represented by ellipses or circles), six transitions (represented by rectangles), a set of oriented arcs connecting the places and transitions, and finally the textual expressions relating to the places, transitions and arcs.

To model this architecture under CPNTools, we used the hierarchical network based on the call of "sub-networks" in order to present a simplified view of it. The double rectangle transitions named 'Fall detection Service', Heart disorderdetection Service' and 'Location Service' represent sub-networks that take as input the input modalities and generate as output the detected anomalies and the user location for the same sample.

The 'Emergency Service' transition is a sub-network that takes the outputs from the three Ambient Assistance Services and generates specific codes according to the user's status.

Figure 5. Model of multimodal architecture

5.1 Falldetection service implementation

The fall detection service is charged with merging the user's absolute acceleration data 'At' for 'n' samples. This information is provided by the tri-axial accelerometer located on the user's belt. The algorithm in Figure 6 illustrates the fall detection service procedure.

Figure 6. Fall detection algorithm

The algorithm detects the occurrence of a fall when the acceleration decreases below the minimum threshold of 0.7g and then rises again to the value of 3g.

After the detection of the freefall phase and the impact, the person often remains in an extended position. To minimize the false positive rate, this service takes into account the duration of this phase (extended position). In our case, we have set this duration at one minute, i.e., 120 samples.

5.2 Heartdisorder detection service implementation

The heart disorder detection service (Figure 7) merges the user's heart rate value 'Fc' to detect tachycardia and bradycardia. The limits, minimum 'Fcmin' and maximum 'Fcmax' of a so-called normal heart rate depend on the health status of the person to be monitored.

After reading 'Fc' and entering 'Fcmin' and 'Fcmax', we can determine two distinct cases:

FcminFcFcmax: the number of heart beats per minute is within the limits of the standard.

Fc<Fcmin or Fc>Fcmax: following cases may occur.

- Fc>Fcmax: if for ten successive samples the value 'Fc' is greater than 'Fcmax', the service detects a tachycardia.

- 0 beats/min<Fc<Fcmin: corresponds to the detection of bradycardia.

- Fc=0 beats/min: if the service receives five 'Fc' values equal to '0 beats per minute', we consider the oxymeter to be incorrectly positioned.

Figure 7. Heart disorder detection algorithm

In the heartdisorder detection service, minimum and maximum heart rate limits are set according to the standard shown in Table 1 [37].

5.3 Location service implementation

In the proposed architecture (Figure 8), we have assigned a number to each room as shown in Table 2.

In order to detect the user's location in the habitat, we placed two photoelectric sensors at each door, for a total of eight sensors.

The location service merges the 'in-room' and 'out-room' ('room' refers to the living room, bedroom, kitchen or bathroom) outputs of the eight photoelectric sensors, placed respectively on the 'internal' and 'external' side of each door.

Figure 8. Proposed habitat with the presence detectors arrangement

Table 2. Numbering of habitat rooms







Room number






5.4 Emergency service implementation

The emergency service is responsible for merging the data from the three ambient assistance services, i.e., fall detection data, heart rate data and location data. During activities of daily living, a person may fall, have a heart disorder, or both can occur at the same time. The system allows to send the code corresponding to the action to be taken with the user's location. This code contains all the information about the user's condition, particularly the order of the emergency. We have proposed to classify the emergency from a minimum order "Order 0" to a maximum order "Order 1". We can then determine three distinct cases: detection of a fall without a heart disorder, detection of a heart disorder without a fall and detection of a fall with a heart disorder.

In order to achieve a better follow-up of the users, the system developed in this work allows the archiving of the results from the emergency service, in the objective of keeping a history of anomaly detection. For each situation, a summary message is sent to the doctor and stored in a database. Each message contains the anomaly detected, whether it is a fall, a heart disorder or both anomalies at the same time.

6. Experimental Validation

Validation is an important step in the process of multimodal architecture development. It allows showing that the system responds positively to the specifications requirements, especially to the real-time response of the system. It also makes it possible to check whether the method used in this work is functional in the simulation environment.

6.1 Experimental context

In order to validate and evaluate the proposed system in terms of accuracy, we conducted several experimental tests. All of these tests were conducted in a non-hospital environment and by volunteers in good health. During the tests, we ensured the volunteers' safety by having them fall onto a mattress in the presence of a doctor, who instructed them on how to fall.

We solicited six (06) volunteers, who were 30 to 70 years old and had no heart disorders or physical handicaps, to wear the device. Each subject performed ten (10) tests, which were free-falling from a standing position or ADLs; some of which may be similar to falls such as picking up an object from the ground or sitting down suddenly.

This allowed us to achieve four scenarios:

Scenario 1: We detached the oximeter from the volunteer's ear to perform the "Oximeter detachment" scenario.

Scenario 2: We asked the volunteer to walk up and down the stairs, jump and run to have a heart rate above 'Fcmax' and detect tachycardia.

Scenario 3: The volunteer free-falled and stayed on the ground for more than two minutes before getting up again.

Scenario 4: After the volunteer fell, we asked the volunteer to get up quickly.

Tests involving final users of the developed system require it to be worn by individuals in care residences for the elderly or in hospital settings.

In what follows, we present, among the different scenarios carried out, the results relating to Scenario 4 " Fall detection without cardiac disorder ", introducing accelerometer and oximeter values into the models developed in CPNTools.

6.2 Experimental results

The values of the absolute acceleration 'At' provided by the accelerometer are extracted every 500 ms, which led us to have 1200 values for the ten minutes of the test. In Figure 9, we have represented only the part between the moments 500/2 s and 850/2 s in order to better visualize the critical event.

We can distinguish: free-fall at t=521/2 s, impact at t=522/2 s and start of extended position at t=523/2 s. The duration of the extended position is more than one minute (from t=523/2 s to t=805/2 s), which means that the volunteer actually remained on the ground after the fall.

Figure 9. Absolute acceleration 'At' generated by the accelerometer

The pulse oximeter generates one value every second, giving 600 values 'Fc' for the ten-minute test. In Figure 10 we present only the part in the interval 100s and 500s, when the fall occurred.

For the 600 generated values we did not obtain ten (10) successive values of 'Fc' greater than 'Fcmax' or less than 'Fcmin', so no disorder is detected. We can see that the values of 'Fc' slightly exceed 'Fcmax' for t=262s, which means that the fall occurred at the same instant.

Figure 10. Heart rate 'Fc' generated by the oximeter

During the tests, when the subject moved from one room to another, photoelectric detectors placed on the doors generated outputs, which, once processed, gave his location. We have represented in Figure 11, the 1200 values corresponding to the room number 'Rnb' where the person is located.

Figure 11. Room number 'Rnb'

As shown in the Figure 11, at each passage from one room to another, the location 'Rnb' takes the value '0' corresponding to the passage through the hall. At the time t=523/2 s corresponding to the fall, the volunteer is in the room with the number '1', corresponding to the living room.

According to the three Figures 9, 10 and 11, we can see that the fall point appeared at time 512/2 s on the acceleration graph, corresponding to time t=256 s on the oximeter graph, which coincides with the location of point Rnb3 in the room number graph. By analyzing these three quantities, we can conclude that a fall has occurred.

After retrieving the results extracted from the three ambient assistance services, the emergency service merges them to deliver the appropriate message. The result delivered by the emergency service, for the scenario considered, is shown in Figure 12.

Among the three possible codes for this scenario (Code 5, 6 and 7), the emergency service transmitted Code 6 for this case. This code was transmitted for sample 523, and the message sent is 'Fall without Heart problem, Person on the floor, Intervention, Emergency, Operator order, Code 6, Person in living room'.

Through this message we can deduce that at the moment t= 523/2 s, the volunteer was in the living room and made a free-fall. He remained on the ground for more than one minute. The operator decided, after a telephone conversation, to activate an emergency.

After testing each scenario several times, we were able to verify the four scenarios by obtaining the appropriate codes. An evaluation was carried out for each of the proposed services on the system.

Figure 12. Sent message and code number

7. Evaluation

Evaluation involves quantitative analysis based on performance measures such as sensitivity, specificity and accuracy [24, 41]. These performance measures, represented by Eqs. (2), (3) and (4), are deduced from four main coefficients obtained from statistical analysis of experimental test results, namely True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN).

$ Sensitivity =V P /(V P+F N)$               (2)

$Specificity =V N /(V N+F P)$               (3)

$ Accuracy =(V P+V N) /(V P+F P+V N+F N)$               (4)

In order to evaluate the system developed in this work in terms of precision, we made a static analysis of the results obtained from the sixty experimental tests carried out by six healthy volunteers. Table 3 presents the results of the fall detection service performance evaluation.

Table 3. Performance parameters



Performance Parameters

Test (nbr)








Fall (30)








ADL (30)







The obtained results showed that the fall detection service is characterized by an accuracy of 78.33%. Indeed, this service can identify most fall events with a sensitivity of 80%. Similarly, that it correctly detects activities of daily life that are not falls with a specificity of 76.67%. Indeed, the fall detection sensitivity obtained, reaching 80%, leads to a false positive rate of 20%; this is acceptable but could be improved by integrating other sensors, for example. We have tested our system on healthy subjects who do not present cardiac diseases such as bradycardia. However, the system should be able to detect bradycardia if we tested it on subjects with cardiac disorders.

The tests of the heart disorder detection service confirm the detection of tachycardia and the detection of oximeter detachment with 100% success. However, we were unable to realize the case of the bradycardia detection. The cardiac disorder detection algorithm has been validated for tachycardia and bradycardia using Cpntools tests. However, evaluation of our algorithm regarding bradycardia could only be carried out in a real hospital environment.

The location service detected the user's location in the habitat with a sensitivity of 90%.

The emergency service was tested in simulation with experimental input data from portable and ambient devices. It was able to detect the oximeter detachment, the case of a fall without heart disorder detection, and the case of tachycardia without fall.

8. Conclusion and Perspectives

The system proposed in this article is based on the development of a multimodal architecture whose aim is to ensure continuous monitoring of elderly in their homes. The proposed modular architecture simultaneously detects falls and heart disorder, while locating the person in his home.

The system is based on three ambient assistance services, namely: the fall detection service, the heart disorder detection service and the location service. An emergency service is also developed to detect anomaly in the health of the persons monitored.

In order to validate the proposed architecture, we modeled it with the Timed and Stochastic Colored Petri Networks. The CPNTools was used to simulate this architecture and generate fault or error detection scenarios to validate and analyze the overall operation. The validation of the proposed system was made on the basis of experimental tests carried out on volunteers, who were 30 to 70 years old, who have no heart disorder or physical handicap. Tests have shown that the system allows the appropriate code to be sent for each case of detection of a fall and / or a heart disorder by locating the user in the home. Indeed, the results showed that the system is characterized by:

·An accuracy of 78.33%, a sensitivity of 80% and a specificity of 76.67% for the fall detection service.

·A 100% success rate for the detection of tachycardia.

·A sensitivity of 90% for the location service.

In addition to detecting heart problems and falls, our system offers other functionalities such as: locating the person in their home when they fall, making it easier for the emergency services to rescue them; and remote medical monitoring, enabling the doctor or emergency service operator to monitor the patient's condition in real time. Given the results, we consider the prospects for, firstly improve the response of the system, and secondly, use modularity to expand to other types of use. Then as a first step, we aim to validate the emergency service through real-life tests, ensuring accurate and quick responses during emergency situations. on the other hand, we plan to carry out a satisfaction survey to assess the acceptability of our system among final users, including the elderly and healthcare professionals Their feedback will help refine the technology to respond to real user needs and healthcare standards.

Furthermore, this kind of system could reduce strain on healthcare allowing more efficient home monitoring, which could reduce unnecessary hospitalizations. It could also be used to monitor patients with chronic diseases, improving their quality of life and reducing healthcare costs.

Finally, a distant detection system for falls and cardiac disorders could be a useful tool for healthcare professionals, enabling them to proactively monitor a large number of patients remotely. However, guaranteeing privacy and data security in such applications is essential to ensure user confidence and compliance with ethical standards.


[1] Statistique Personne Âgées.,accessed on 09-08-2023.

[2] Fall Safety: Take Steps to Remain Independent Longer.,accessed on 03-08-2023.

[3] Ranakoti, S., Arora, S., Chaudhary, S., Beetan, S., Sandhu, A.S., Khandnor, P., Saini, P. (2019). Human fall detection system over IMU sensors using triaxial accelerometer. In Computational Intelligence: Theories, Applications and Future Directions, I: 495-507.

[4] Igual, R., Medrano, C., Plaza, I. (2013). Challenges, issues and trends in fall detection systems. Biomedical Engineering Online, 12(1): 66.

[5] Chen, D., Feng, W., Zhang, Y., Li, X., Wang, T. (2011). A wearable wireless fall detection system with accelerators. In 2011 IEEE International Conference on Robotics and Biomimetics, pp. 2259-2263.

[6] Wu, F., Zhao, H., Zhao, Y., Zhong, H. (2015). Development of a wearable-sensor-based fall detection system. International Journal of Telemedicine and Applications, 2015: 576364.

[7] Palmerini, L., Klenk, J., Becker, C., Chiari, L. (2020). Accelerometer-based fall detection using machine learning: Training and testing on real-world falls. Sensors, 20(22): 6479.

[8] Boudouane, I., Makhlouf, A., Saadia, N., Ramdane-Cherif, A. (2019). Wearable camera for fall detection embedded system. In Proceedings of the 4th International Conference on Smart City Applications, pp. 1-6.

[9] Ozcan, K., Mahabalagiri, A.K., Casares, M., Velipasalar, S. (2013). Automatic fall detection and activity classification by a wearable embedded smart camera. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 3(2): 125-136.

[10] Boudouane, I., Makhlouf, A., Harkat, M.A., Hammouche, M.Z., Saadia, N., Ramdane Cherif, A. (2020). Fall detection system with portable camera. Journal of Ambient Intelligence and Humanized Computing, 11: 2647-2659.

[11] Nweke, H.F., Teh, Y.W., Mujtaba, G., Al-Garadi, M.A. (2019). Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions. Information Fusion, 46: 147-170.

[12] Wang, C., Lu, W., Redmond, S.J., Stevens, M.C., Lord, S.R., Lovell, N.H. (2017). A low-power fall detector balancing sensitivity and false alarm rate. IEEE Journal of Biomedical and Health Informatics, 22(6): 1929-1937.

[13] Huynh, Q.T., Nguyen, U.D., Irazabal, L.B., Ghassemian, N., Tran, B.Q. (2015). Optimization of an accelerometer and gyroscope-based fall detection algorithm. Journal of Sensors, 2015: 452078.

[14] Guo, H.W., Hsieh, Y.T., Huang, Y.S., Chien, J.C., Haraikawa, K., Shieh, J.S. (2015). A threshold-based algorithm of fall detection using a wearable device with tri-axial accelerometer and gyroscope. In 2015 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), pp. 54-57.

[15] Jefiza, A., Pramunanto, E., Boedinoegroho, H., Purnomo, M.H. (2017). Fall detection based on accelerometer and gyroscope using back propagation. In 2017 4th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), pp. 1-6.

[16] Kerdjidj, O., Ramzan, N., Ghanem, K., Amira, A., Chouireb, F. (2020). Fall detection and human activity classification using wearable sensors and compressed sensing. Journal of Ambient Intelligence and Humanized Computing,11(1): 349-361.

[17] Islam, M.M., Neom, N., Imtiaz, M.S., Nooruddin, S., Islam, M.R., Islam, M.R. (2019). A review on fall detection systems using data from smartphone sensors. Ingénierie des systèmes d Inf.,24(6): 569-576.

[18] Nedjai-Merrouche, I., Saadia, N., RamdaneCherif, A., Makhlouf, A. (2021). Outdoor multimodal system based on smartphone for health monitoring and incident detection. Journal of Ambient Intelligence and Humanized Computing,12(12): 10699-10721.

[19] Shrivastava, R., Pandey, M. (2021). Human fall detection using efficient kernel and eccentric approach. International Journal of E-Health and Medical Communications,12(1): 62-80.

[20] Hakim, A., Huq, M.S., Shanta, S., Ibrahim, B.S.K.K. (2017). Smartphone based data mining for fall detection: Analysis and design. Procedia Computer Science, 105: 46-51.

[21] Chen, L., Li, R., Zhang, H., Tian, L., Chen, N. (2019). Intelligent fall detection method based on accelerometer data from a wrist-worn smart watch. Measurement, 140: 215-226.

[22] Sanchez, J.A.U., Muñoz, D.M. (2019). Fall detection using accelerometer on the user’s wrist and artificial neural networks. In XXVI Brazilian Congress on Biomedical Engineering. Springer, pp. 641-647.

[23] Birku, Y., Agrawal, H. (2018). Survey on fall detection systems. International Journal of Pure and Applied Mathematics, 118(18): 2537-2543.

[24] Nooruddin, S., Islam, M.M., Sharna, F.A., Alhetari, H., Kabir, M.N. (2021). Sensor-based fall detection systems: A review. Journal of Ambient Intelligence and Humanized Computing, 1-17.

[25] Chaccour, K., Darazi, R., el Hassans, A.H., Andres, E. (2015). Smart carpet using differential piezoresistive pressure sensors for elderly fall detection. In 2015 IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pp. 225-229.

[26] Li, Y., Ho, K.C., Popescu, M. (2012). A microphone array system for automatic fall detection. IEEE Transactions on Biomedical Engineering,59(5): 1291-1301.

[27] Medjahed, H., Dorizzi, B., Istrate, D., Baldinger, J.L., Boudy, J., Bougueroua, L., Dhouib, M.A. (2012). A fuzzy logic approach for remote healthcare monitoring by learning and recognizing human activities of daily living. INTECH Open Access Publisher.

[28] Sixsmith, A., Johnson, N., Whatmore, R. (2005). Pyroelectric IR sensor arrays for fall detection in the older population. Journal de Physique IV (Proceedings), 128: 153-160.

[29] Mahmood, N.H., Uyop, N., Zulkarnain, N., Harun, F.C., Kamarudin, M.F., Linoby, A. (2011). LED indicator for heart rate monitoring system in sport application. In 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, pp. 64-66.

[30] Makhlouf, A., Boudouane, I., Saadia, N., Ramdane Cherif, A. (2019). Ambient assistance service for fall and heart problem detection. Journal of Ambient Intelligence and Humanized Computing, 10: 1527-1546.

[31] Wang, J., Zhang, Z., Li, B., Lee, S., Sherratt, R.S. (2014). An enhanced fall detection system for elderly person monitoring using consumer home networks. IEEE Transactions on Consumer Electronics, 60(1): 23-29.

[32] Bet, P., Castro, P.C., Ponti, M.A. (2019). Fall detection and fall risk assessment in older person using wearable sensors: A systematic review. International Journal of Medical Informatics,130: 103946.

[33] Shinde, B.A., Chawan, P.M. (2014). Dementia patient movement detection and fall detection using smart phone technology. International Journal of Advanced Technology in Engineering and Science, 2(7): 155-160.

[34] Noury, N., Rumeau, P., Bourke, A.K., ÓLaighin, G., Lundy, J.E. (2008). A proposal for the classification and evaluation of fall detectors. In 2014 10th International Conference on Reliability, Maintainability and Safety (ICRMS), pp. 157-159.

[35] Chen, J., Kwong, K., Chang, D., Luk, J., Bajcsy, R. (2006). Wearable sensors for reliable fall detection. In 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pp. 3551-3554.

[36] Boudouane, I., Makhlouf, A., Harkat, M.A., Saadia, N., Ramdane-Cherif, A. (2020). Post-fall time accounting for fall detection using a portable camera. In Proceedings of the 3rd International Conference on Networking, Information Systems & Security, pp. 1-7.

[37] Pouls et rythme cardiaque: Les valeurs normales.,accessed on 02-10-2019.

[38] Makhlouf, A., Nedjai, I., Saadia, N., Ramdane-Cherif, A. (2017). Multimodal system for fall detection and location of person in an intelligent habitat. Procedia Computer Science, 109: 969-974.

[39] Makhlouf, A., Saadia, N., Ramdane-Cherif, A. (2015). Services of ambient assistance for elderly and/or disabled person in health intelligent habitat. In ICAART, (2): 225-231.

[40] CPN Tools., accessed on Dec. 25, 2018.

[41] Fortino, G., Gravina, R. (2015). Fall-MobileGuard: A smart real-time fall detection system. In Proceedings of the 10th EAI International Conference on Body Area Networks, pp. 44-50.