Real-time Smart Auditory Assistive Wearable (RESAAW) for People with Different Degrees of Hearing Loss

ABSTRACT


INTRODUCTION
Assistive devices for the hard of hearing have come a long way in recent years, and there is ongoing research in this field to further improve their effectiveness and accessibility.These devices aim to improve communication, independence, and quality of life for people with hearing loss.The area of your hearing that is affected determines the sort of hearing loss you experience.Hearing loss can be classified into three categories: • Conductive hearing loss • Sensorineural hearing loss • Mixed hearing loss When sound cannot be properly transmitted from the outside or middle ear to the inner ear, conductive hearing loss develops.Because of this, the level of sound that enters the inner ear may be reduced, making it challenging to hear speech or subtle noises.Conductive hearing loss can affect one or both ears, and it can be either temporary or permanent.There can be several reasons [1] for conductive hear loss.
• Blockage of the ear canal can result in conductive hearing loss that could be brought on by earwax buildup, foreign objects, or tumors.
• Damage to the eardrum: Conductive hearing loss can be brought on by a perforated or burst eardrum.Trauma, illness, or variations in air pressure, such as those that occur during flight, may be the cause of this.
• Middle ear damage: The ossicles, three small bones in the middle ear that assist transfer sound from the eardrum to the inner ear, are damaged.Conductive hearing loss can result from damage to the ossicles, such as that caused by infection, trauma, or hereditary conditions.

•
Growths that aren't normal, including tumors, cysts, or bone growths, might obstruct the middle ear or the ear canal, causing conductive hearing loss.
Depending on how severe the problem is, conductive hearing loss can present with a variety of symptoms.Some typical signs include hearing quiet noises or speech with difficulty, distorted or muddled hearing, the experience of having a full or blocked ear, discomfort or pain in the afflicted ear and hearing ringing or tinnitus.
In case of sensorineural hearing loss, when the auditory nerve that transmits signals to the brain or the inner ear (cochlea) is damaged, hearing loss known as sensorineural hearing loss results.
Depending on how severe the problem is, sensorineural hearing loss can present with a variety of symptoms.Some typical signs include having trouble hearing speech or quiet noises, especially in busy situations, the requirement to increase the radio or television's loudness, ringing in the ears, called tinnitus, speech is difficult to comprehend, especially when spoken by toddlers pose a challenge in differentiating sounds, especially ones with identical frequencies.
The primary objective of this system is to help save the lives of the people with hearing loss problem.There are fatalities happening every year because of their hearing loss.Recently a middle aged woman who is having a hearing loss died on the spot while crossing the track inside a railway station.The train was not even faster but the impact of hitting her caused the death because there was no big noise.Such noise less movements can be identified and alert will be sent through the system proposed here.
Even though many small and ergonomic assistive devices are available for people with this problem, they are all expensive.They are not affordable for 50% of people (falling under below poverty line) in a developing country like India.
Resulting in many children growing up without any assistive devices and getting accustomed to the hearing loss problem in economically poor background areas.This leads to denial of education to them in a normal school.The work proposed here is about an assistive device which is economically affordable and made up of very simple technology.

RELATED WORKS IN ASSISTIVE TECHNOLOGY FOR HEARING AID
The need for having a device to assist in hearing for the people with hearing loss problem was there from 15th century.The first one was developed in the late 18th century.It was a device which is trumpet shaped.Some of them are simple cones.It would amplify the noise fed into it and gives the output, if kept near the ear.It was actually developed for the kings and for people in higher position.Later in the 19th century, electrical based, acoustic based devices were developed.Then many innovative hidden hearing assistive devices were also developed.Electronic and computer based hearing aids were developed only after 1960s.It took such a long journey to reach the state of device what is available now.The one available now is small, aesthetic, weight-less but expensive.
There has been a lot of study and development put into hearing-impaired people's assistive technology.The progress of technology has led to the production of a broad variety of tools that can enhance social interaction and communication for people with hearing loss.A hearing aid is one of the most well-liked pieces of assistive technology for those who are hard of hearing.
Cochlear implants are another type of assistive technology that can be very helpful for those with severe or profound hearing loss.Cochlear implants [2], which are electrical devices surgically placed in the inner ear, can directly stimulate the auditory nerve.People whose hearing loss is too severe to be helped by hearing aids frequently use them.A further category of assistive technology for those with hearing impairments is personal amplifiers, which are small, portable devices that may be worn in the ear or carried in the pocket that can amplify sound in noisy environments.Furthermore, FM systems, which transmit sound directly to hearing aids or cochlear implants, can help people understand speech better in noisy environments.
In addition to these assistive devices, there has been much research into other technologies that can help people with hearing loss communicate more effectively.For instance, speech recognition technology has advanced significantly in recent years, and there are now apps and software programs that can instantly convert spoken words into text, enabling persons with hearing loss [3] to participate fully in conversations.People with severe hearing loss may benefit from cochlear implants, which can restore their sense of hearing.As well as creating novel methods for implanting and programming these devices, researchers are striving to enhance the design and performance of these devices.For example, scientists are attempting to create new electrodes that can be implanted deeper into the cochlea, enabling more accurate stimulation of the auditory nerve and maybe higher sound quality.
The creation of new, more sophisticated, and efficient varieties of hearing aids is the subject of another field of study.This involves the creation of hearing aids that may adjust to a user's particular hearing requirements using machine learning algorithms.In order to optimize a hearing aid's amplification and noise reduction settings, machine learning algorithms may be employed to determine a person's unique hearing loss, listening environment, and listening preferences.Additionally, researchers are aiming to create hearing aids that can link directly to other devices, such as smartphones, enabling remote control, programming, and audio streaming to the hearing aid.
Hearing loss can be caused by several conditions, including predisposition to genetic diseases, trauma, loud noise exposure, and ageing.Some people are born with hearing disabilities, while others develop it later in life.Tinnitus can also be caused by medical diseases such as otitis media or Meniere's disease.Furthermore, chronic exposure to high-intensity noise [4], such as that found in industrial facilities or via the use of traditional high-frequency personal hearing aids, can damage the fragile tissues of the inner ear, eventually leading to hearing disabilities.With RESAAW, the aim is to provide a simple prototype device to assist the hearing challenged individuals.
Some of the previous work described [5] and proposed an IoT based communication assistant application for the hearing challenged individuals, by utilizing a microcontroller, pulse sensor, and interfacing electret microphone amplifier.Their web-based application can provide information about the strength of the sound produced and the text of the conversation of the interlocutor.In their research they have developed a sensor device consisting of a NodeMCU V3, a pulse sensor, and connecting electret microphone amplifiers, to create an Internet of Things (IoT)-based communication assistant application for the hearing impaired.To track the volume of sound people with hearing loss talk with while they speak, sensors regularly relay data to a computer.
According to Angdresey et.al.[6], the study examined the ways in which prolonged exposure to sound affects the brain.The study focused on hearing challenged individuals who had longterm usage of hearing devices despite being born with hearing disabilities.They matched their brain activity to that of individuals with normal hearing.While having the activity of their brains monitored, both groups experienced sensations on their hands.The area of the auditory cortex, which processes sound, was more active in the hearing challenged group.The reason, according to the researchers, is that the hearing challenged individuals' brains reacted to the vibrations differently from those of the hearing group since they were accustomed to experiencing vibrations from their hearing aids.
The researchers [7] proposed a wireless calling bell system for the hearing challenged individuals and mute called WiBeD2.It is a prototype that is worn on the wrist and is based on the idea of a wearable gadget.The system includes message acknowledgement switches, an LED indication, and a vibrator.The wireless device vibrates when the calling bell is pressed, and an LED indicator appears.The system makes use of wireless communication technologies and wearable gadgets to enable accessibility for the hearing challenged individuals and the mute.It incorporates switches for message acknowledgment, LED and LCD displays, and vibration.
Also, the purpose of the study was to evaluate how 15 preschool-aged severely retarded hearing and visually impaired children responded to aural stimulation [8].The difference in the children's operant choices between no sound and music performed at 80 dB SPL was assessed by the researchers.Those kids who had trouble answering were given vibratory help.It was discovered that the assistance was essential in enabling some of the kids to consistently respond to the auditory stimuli.The study's overall findings highlighted the significance of adopting a variety of tools and methods to support the listening and communication skills development of kids with hearing and vision impairments.
In the research [9], the paper describes a gadget that can help those who are visually, vocally and hearing challenged.It is based on Google API and Raspberry Pi technology.The gadget has the capacity to take pictures and use image-to-text conversion technologies to turn them into text.The captured text is then transformed into voice output so that visually impaired people can hear it.The gadget records spoken words with a microphone and translates them into text format, which is then shown on the screen for individuals who have trouble hearing.The device can help in translating documents, books, and other materials, which can enhance the quality of life for individuals with disabilities.
According to the study [10], the human nervous system may adjust to sensory deprivation by creating new methods of information processing.The auditory cortex, for instance, may begin to process tactile (touch) information in persons who are hard of hearing from birth.They discovered that the hearing challenged individuals had improved tactile sensitivity because they were better able to recognize changes in the vibratory stimuli.This increase in sensitivity might be result of more attention being paid to the stimuli or neuronal plasticity, where the brain creates new pathways to receive tactile information.
In the study [11], the proposed wearable gadget will alert hearing-impaired individuals when someone is attempting to grab their attention.The gadget vibrates to inform the user after identifying the user's name or common words like "hey" or "hello" using sound detection technology.The Voice Recognition Module (v3) sensor, which is used by the system, can identify the user's name when they are called by a third party.A vibration motor is then used to turn the sound signals into vibrations.
In the study [12], the created wearable device contains a number of LED outlets, vibration motors, and microphones.The sound data collected by the microphones is analyzed to determine the kind of sound.The device then employs particular vibration frequencies to let the user know where the sound is coming from.The most effective feature extraction and classification techniques were chosen using this procedure.The prototype gadget was then put to the test using these chosen techniques, with the KNN [13] classification and SVM [14] attaining 93% and 94% success rates, respectively.With SVM and two of the best feature approaches, the system had a 98% success rate up close.Overall, this system aims to help hearing-impaired individuals detect sounds and their direction through vibrations and LED indicators.

REALTIME SMART AUDITORY ASSISTIVE WEARABLE (RESAAW)
In this proposed research work, the focus is given more on using vibration as a signal for the hearing challenged individuals.According to medical research, vibration technology has proven to be a useful tool for individuals who are hard of hearing.Vibration is a powerful tool for individuals who are hard of hearing, as it allows them to perceive sound in a different way than those who can hear normally.Research has shown that hearing challenged individuals are able to perceive vibrations in a more sensitive and nuanced way than those who can hear.One of the main reasons that vibration is better perceived by hearing challenged individuals is that it bypasses the auditory system entirely.
When individuals hear sound, it is first processed by the ear and then transmitted to the brain for interpretation.However, for individuals who are hearing challenged, this process is not possible.Instead, vibrations are directly transmitted to the brain, bypassing the ear and allowing for a more direct and accurate perception of sound.Another reason that vibration is better perceived by hearing challenged individuals is that it can provide a more complete and holistic representation of sound.When individuals hear sound, they typically only hear a small portion of the sound waves.However, when vibrations are used, individuals can feel the entire sound wave, providing a more complete representation of the sound.In addition to these benefits, vibration technology can also improve the overall quality of life for hearing challenged individuals.For example, the use of vibrating alarms can help challenged individuals wake up on time, and vibrating doorbells can help them know when someone is at the door.This can greatly improve their independence and overall ability to navigate their daily lives.Using this as the basis of this proposed work, the ability of hearing challenged individuals to differentiate between vibrations of different intensity has been explored.Hearing challenged individuals have a unique ability to differentiate between vibrations of different intensity, which is better than normal people.
Research has shown that the brains of hearing challenged individuals have a heightened sensitivity to vibrations [15], which allows them to perceive a wider range of vibrations and differentiate between vibrations of different intensity more accurately than those who can hear normally.The brain can rewire itself to adapt to new experiences and environments.For hearing challenged individuals, the brain rewires itself to process vibrations differently, allowing for a more accurate perception of sound [16].This neuroplasticity allows the brain to process vibrations in a more refined and sensitive manner.Another reason that hearing challenged individuals are better at differentiating between vibrations of different intensity is because they rely more heavily on their sense of touch [17].
Research has shown that the neural pathways in the brains of hearing challenged individuals are more developed and specialized for processing vibrations than those of individuals who can hear normally.
When a noise is considered not part of the general white noise of the environment like a sudden noise or a progressively incremental noise, the RESAAW provides the user with vibrational feedback from the specific direction where this noise was picked from.

Architecture
The proposed system for RESAAW is a very simple one.It works using very simple techniques of digital signal processing and Machine Learning.The device itself is a very simple embedded system device using an Arduino Uno as its processor.In our proposed system, we are using omnidirectional microphones that would pick up noises in the surroundings and then use digital signal processing techniques to filter the sounds.When a sudden noise or a constant approaching noise is heard, the machine learning technique used in RESAAW will alert the user through vibrations of vibrational motors.The received noise is compared with historic data of noises and XGBoost [18] algorithm is used for prediction.It is an ensemble algorithm which does not depend upon single model's prediction.It ensembles the result of combined models.This algorithm is computationally effective for data received from such low level devices.The basic idea of the proposed system can be pictured through this flow chart in Figure 1.A condenser microphone that employs a permanently charged electret material as the diaphragm is known as an electret microphone.The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions.The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions.The other plate of the diaphragm is only a short distance from the other plate of the back plate.When the sound wave hits, the film will vibrate.When the diaphragm is vibrated, the separation of the two plates of the capacitor is different, and the tiny electrical signal will generate corresponding to the sound waves.

Figure 3. Electret microphone
A condenser microphone that employs a permanently charged electret material as the diaphragm is known as an electret microphone.The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions.The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions.The other plate of the diaphragm is only a short distance from the other plate of the back plate.When the sound wave hits, the film will vibrate.When the diaphragm is vibrated, the separation of the two plates of the capacitor is different, and the tiny electrical signal will generate corresponding to the sound waves.The vibrations that are sensed or heard are created by this oscillation.The motor's speed and direction control the vibrations' frequency and intensity.The frequency and power of the vibrations will grow as the motor speed increases.

RESULTS AND DISCUSSION
RESAAW provided promising results.Showing a clear pattern of how the 2 SMA [19] filters are working together and how the condition is being triggered to vibrate the microphones.The output is being visualized in the following screenshots of the Serial Plotter in Figure 6 and Figure 7.In Figure 6 and Figure 7, we can see that the blue line is a reference base line always fixed at '0'.The Yellow line is the line signifying the vibration of motor.
When the orange line which signifies a sudden loud noise is heard, we can see a vibration is heard.The green line signifies the varying baseline as per the sound levels in the environment of RESAAW.
In Figure 8 and Figure 9, the blue line represents the reference base line which is always fixed at '0'.The Yellow line is the line signifying the vibration of motor.When the orange line which signifies an approaching noise is heard, a continuous vibration can be observed.
The green line signifies the varying baseline as per the sound levels in the environment of RESAAW which is also increasing as the average sound levels of the surrounding are increasing.
From the figures it is clear that the RESAAW prototype is working as expected.Whenever a sudden noise is heard, the processed SensorValue shows a steep spike which implies the first SMA filter is working fine and there are steep changes in the value when a sudden noise is heard.Whenever a loud and approaching noise is heard, the processed SensorValue stays continuously above the baseline signifying the SMA filter is working for longer duration of high sound exposures.The vibrator is also working as expected.Whenever a steep sound is heard, a vibration for 1 second is produced but when a continuously high sound is heard, the vibrator keeps rumbling for a greater time period.Not only it worked for simple real time samples but also for the data used from CHiMe data set [20] containing variety of domestic sounds.The false positives were 3% and the true negatives were 23.33%.
Many research works are already going on in this assistive technology.Also many kinds of wearable assistive devices are in the market (Like smart cane and smart glasses) now.They use ML and are able to attain an accuracy between 90 to 95%.But most of them uses images for identifying the approaching objects and they are all computer vision based.In the work presented here, only from the digitized sounds, ML is used and alert is given to the user.It produces an accuracy of 82%.It can be customized as a smart cap in the future.

5.CONCLUSIONS
The RESAAW initiative aimed at aiding individuals with hearing impairments has proven to be a triumph.The RESAAW system has been developed with the aim of enhancing the awareness of individuals with hearing impairments.Empirical evidence from our testing indicates that the system is functioning as intended.RESAAW's innovative design facilitates ease of use and has the potential to enhance the quality of life for individuals experiencing hearing impairment.The potential advantages of the Resilience, Empowerment, and Advocacy in Action Workshop (RESAAW) are noteworthy, and it is our aspiration that it will be embraced by those who require it the most.In future, RESAAW can be designed as a wearable cap that is powered by batteries to make it easier to wear and carry.And it can be made IoT enabled.The vibration patterns can also be set for even more classification of noises.The design is made in such a way, that it can be easily integrated with home automation system using IoT with very little modification.

Figure 1 .
Figure 1.Architecture of RESAAW 3.1.1Components of REESAW Arduino Uno Microcontroller: In the Arduino family, the ATmega328P microcontroller chip is the basic building block of the open-source Arduino Uno microcontroller board.Due to its relatively low cost, straightforwardness, and versatility to construct numerous electrical designs, the Arduino Uno (Figure 2) is a widely recognized and frequently used development board.

Figure 2 .
Figure 2. Arduino Uno Electret Omnidirectional Microphones: A type of microphone that can equally capture sound from all directions is the omnidirectional electret Figure 3.They are often utilized in a wide range of settings, including recording studios,

Figure 4 .
Figure 4. Vibration motorVibration Motor: A motor that vibrates while it turns is known as a DC motor vibrator.It comprises of an eccentric weight, sometimes referred to as a vibrating mass, and a DC motor Figure4.The eccentric weight spins when the motor is turned on, which causes it to move in a circular manner.There are audible or sonic vibrations produced by the weight as it travels.The DC motor vibrator operates based on rotational motion and the mass's inertia.A force that oscillates back and forth is produced by the mass as it travels in a circular motion.

3. 1 . 2
Credo used in RESAAW • Step 1: Define the pins used for microphones, vibrator and the buffer sizes used for filtering and detection.2. • Step 2: Initialize variables and arrays for storing microphone readings and past sensor values.The analog signals are processed by the microprocessor and converted to digital data by the analog-to-digital converter.• Step 3: Set up the serial communication and the pins for the microphones and vibrator.• Step 4: Enter the main loop function.• Step 5: Obtain a clean sensor value by calling the smooth() function.Here is where the digital signal processing techniques • Step 6: Update the dynamic threshold by calling the updateSensorPastAverage() function.In this function, the XGBoost algorithm is applied for predicting the movement of objects closer to the user.• Step 7: Check if the current threshold has deviated from the past average value.If yes, update the current threshold.• Step 8: Print the current sensor value, past average value, and the motor vibration status to the serial monitor.• Step 9: Check if the motor is vibrating and if the vibration duration has exceeded the preset time limit.If yes, turn off the motor vibration.• Step 10: Check if the processed sensor value is greater than the current threshold + detection buffer.If yes, turn on the vibrator and set the vibration start time.• Step 11: Define the hasCurrentThresholdDeviated() function which checks if the current threshold has deviated.• Step 12: Define the updateSensorPastAverage() function which updates the past average of the sensor value.• Step 13: Define the smooth() function which obtains a clean sensor value by filtering the microphone readings and calculating the average.• Step 14: End of the main loop function.By using this algorithm, REESAW will update and store the current data about object movements and their predicted values.

Figure 6 .Figure 7 .
Figure 6.Vibrating when a loud sudden noise is heard

Figure 8 .Figure 9 .
Figure 8. Vibrating when an approaching noise is heard