Adedotun O. Adetunla* | Olanrewaju Kolade | Adeyinka M. Adeoye | Saheed Akande
© 2022 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Hearing loss is the inability to hear sounds ranging from 20 decibels or more in one or both ears. It can affect one or both ears and leads to difficulty in hearing speech or sounds in general. Single-sided deafness or unilateral hearing loss is a very widespread disability. However, most people only see hearing loss as being a binary problem assuming that you either have perfect hearing in both ears or are completely deaf in both ears, and dismiss the other types of hearing loss. Sensory substitution involves remapping the information gathered by one sensory receptor to another. Sensory receptors regardless of the signals they receive or capture, all encode the gathered information as electrochemical signals. This biological property of sensory receptors, coupled with the human brain’s neuroplasticity allows sensory receptors to be substituted, giving rise to new methods of sensory perception. This study aims to develop a sensory device known as a localizer. The localizer detects sound using numerous sound sensors, and feeds the input to the microcontrollers which then use the input to control the eccentric mass motor by implementing various motor drivers. The results gotten from this prototype device shows great improvement in the ability of a single-sided deaf person to localize sound.
binaural test, localizer, microcontroller, sensors, unilaterally deaf, white noise
Single sided deafness or unilateral hearing loss is a very widespread disability. However, most people only see hearing loss as being a binary problem assuming that you either have perfect hearing in both ears or are completely deaf in both ears and dismiss the other types of hearing loss. Single sided deafness implies that there is a significant decrease in hearing or complete deafness in only one ear. It could be temporary or permanent and it greatly affects how the sufferers perceive sound. There are plenty reasons for why single sided hearing loss occurs, for example it could be caused by trauma to the head, a very loud sound exceeding and sustaining 90 decibel and above. It could be caused by diseases or infections such as an Acoustic neuroma, Encephalitis, Meningitis, Lyme’s disease, etc. [1, 2]. The problems of the cochlea implant and other available hearing assistive devices like the Sonus Soundbite hearing aid underscore the need for a low-cost non-invasive solution, which is what this research aims to achieve through sensory substitution for developing nations.
Across the central nervous system in human beings, all the information captured by sensory receptors (the eyes, the ears) are all ultimately transmitted to the brain using electrical impulses, this, coupled with the brains’ ability to rewire itself (neuroplasticity), any model of sensory input can be given to the brain through the nervous system and the brain will learn to extract any information it needs from the signal or sensory input [3, 4].
The first sensory substitution study was performed in 1969 by Paul Bach-y-Rita. Paul Bach-y-Rita is known as the father of sensory substitution for his extensive work and research into the field of neuroplasticity. Bach-y-Rita first proposed the idea of sensory substitution as a way to treat patients with neurological conditions. The first application of his research was a chair that allowed blind people to gather visual information. The study involved passing visual information to blind people using tactile feedback [5]. A number of studies have looked into the effect of hearing loss on interaural time differences, and have found a clear trend for poor localization and lateralization in people with unilateral or asymmetrical hearing damage. This is due to the difference in performance between the two ears. However, they did not find significant localization problems in individuals with symmetrical hearing losses [6-9].
Regardless of the cause of the unilateral hearing loss, any individual with this disability will have to adapt to a different way of processing sound. The human brain processes information like distance, location, direction and height or depth and even velocity from sound by using the difference in the audio input gathered by both ears. Since the human ears are located at different parts of the head and are separated from one another, sounds reach the ears at different volumes and slightly different times allowing the brain to use the difference to calculate exactly where the sound is coming from. This phenomenon is called sound localization.
The problems of single sided deafness are the loss of sound localization, Understanding or following conversations in the presence of loud background noise and locating sound from a single source in an open or large environment. The aim of this study is to develop a device to solve these problems. The development of this non-invasive sound-to-touch sensory substitution device will be subjected to auditory experiments with single sided deaf adults to determine if there is an improvement in their ability to localize sound while using the device. The methodology and result obtained from this study are discussed extensively.
The components selected for the development of the sensory substitution device were defined based on the design requirements, capacity, size affordability and specifications. The following components were selected.
It can control both the speed and direction of two DC motors simultaneously. The board equipped with power LED indicators, on-board +5V regulator and protection diodes as shown in Figure 2a. Another device is a vibrator more which is a custom design Eccentric Rotating Mass (ERM) motors as shown in Figure 2b. Applying a voltage across the motor causes it to spin the weight, Increasing the voltage spins the weight faster, increasing the vibration frequency and force [10]. Other components include a micro-controller which serve as the brain behind the whole process, an Arduino Uno micro controller was choosen. A dev kit board which consists of an ESP32 module with integrated antenna and RF balun, power amplifier, low-noise amplifiers, filters, and power management module as shown in Figure 2c.
Figure 1. (a) KY-038 sound sensor module, (b) INMP41 omnidirectional sound sensor
Figure 2. (a) L298N Dual Bridge Motor Driver (b) Eccentric Rotating Mass vibration motor , (c) DOIT ESP32 Dev Kit V1
Figure 3. The localizer with the connections on a jacket(a) the inner (b) the outer appearance
Figure 4. Circuit Diagram of the sensory substitution device
For the build-up of the sensory device, wires were soldered to the 5v DC motors, and the motors were connected to the L298N Motor drivers. Two motors were connected per motor driver. The L298N motor drivers were connected to the microprocessors. Two L298N were connected to the Arduino Uno R3 and the last one to the ESP32. The sound sensors were connected to the microprocessors. The two KY-038 sound sensor were connected to the Arduino Uno R3 and the two INMP441 sound sensors were connected to the ESP32 as shown in Figure 3. The mode of operation of the prototype device consists of a sound sensor that detects sounds ranging from 10dB to 80dB, the Arduino Uno receives the signal from the KY-038, while the DOIT ESP32 Dev Kit V1 receives signals from the INMP441 sound sensor [11]. They both store the data and relay it to the corresponding motor driver. The L298N motor driver receives the signal from the Arduino Uno and The ESP32 powers the 5v DC motors accordingly. After the acoustic event has ended, the Arduino and ESP32 send signals to the L298N motor which powers down the DC motors. The Microcontrollers wait for another input signal within the sound sensor detection range then repeat the process. The circuit diagram is shown in Figure 4.
The Sensory Substitution device used for testing will be referred to from here on as The Localizer. The interface through which participants interacted with the localizer was run completely offline without reliance on a computer for power source. Three sets of custom experiments were conducted to test the efficacy of the localizer. The first set was the binaural sound localization test, the second set was Monoaural Left sound Localization test and the Third set was the Monoaural Right sound localization test.
4.1 The binaural testing method
Figure 5. The result of the binaural testing method for the single sided deaf adult
This testing method involved placing the subjects in the center of a room, six speakers were set up around the subjects and the subjects were blindfolded to eliminate audio-visual cues. Sound was played at random from each speaker at a time to emulate random acoustic events present in everyday life. The blindfolded subjects were then asked to point out which speakers produced the sound. The test was conducted with each subject, and testing involved the localizer being worn and without the localizer and the result is presented in Figure 5. The purpose of this experiment set was to obtain a baseline for sound localization in both the normal hearing subject and the unilaterally deaf adult.
The tests conducted with this method showed a high improvement in the ability of a single sided person to localize sound, surprisingly enough the Normal hearing adult saw improved scores in the sound localization tests from using the localizer, even though the normal hearing adult was not considered at any point during the design specifications and analysis.
4.2 The monoaural left testing method
This testing method involved placing the subject in the center of a room, four speakers were set up around the subject on their left side and the subject was then blindfolded and made to wear earphones in their left ear. Sound was played at random from each speaker to emulate random acoustic events present in everyday life. This test was conducted twenty times in total, five times with each speaker to obtain a reasonable accuracy average. The test was conducted with each subject, and testing involved the localizer being worn and without it being worn. Note that the unilaterally deaf subject is completely deaf in their left ear and was not required to wear the headphones during this testing phase.
Figure 6. The results of the monoaural left testing method for the normal hearing adult
The purpose of this experiment set was to determine how well the subjects could localize sounds coming from the direction their non-hearing ear. The accuracy of detection was based on five tests, which were scored correct or incorrect based on the response of the subject, where correct was 1 and incorrect was 0. The accuracy results were then calculated by using the percentage of the averages.
For example: Speaker FF $a+b+c+d+e=x \text { Accuracy }=x / 5 * 100$
where, a, b, c, d, e are different tests with each speaker, x is the sum total of the test results of each speaker.
FF = Front Forward, FR = Front Right, FL = Front Left, BL = Back Left, BR = Back Right, BB = Back Backward.
This method of testing shows a major improvement in the ability of the single sided deaf adult to localize sound coming from their non-hearing side as shown in Table 1. An interesting thing to note from this experiment set however is the results of the normal hearing adult who performed considerably poor, even more so than the single sided deaf adult in the tests conducted with and without the localizer, Showing the normal hearing adult’s extreme reliance on binaural hearing and further providing proof for the existence of neuroplasticity and the benefits it provides in improving the sound localization ability of unilaterally deaf people. The result is presented in Figure 6.
Table 1. Results of the monoaural left testing method
|
Subject |
Distance from subject |
Accuracy of detection |
Status |
|
Normal Hearing Adult |
5m (FF Speaker) |
80% |
With device |
|
|
5m (FL Speaker) |
40% |
|
|
|
5m (BL Speaker) |
40% |
|
|
|
5 m (BB Speaker) |
20% |
|
|
Normal Hearing Adult |
5m (FF Speaker) |
40% |
Without device |
|
|
5m (FL Speaker) |
20% |
|
|
|
5m (BL Speaker) |
0% |
|
|
|
5m (BB Speaker) |
20% |
|
|
Single sided deaf Adult |
5m (FF Speaker) |
80% |
With device |
|
|
5m (FL Speaker) |
60% |
|
|
|
5m (BL Speaker) |
40% |
|
|
|
5m (BB Speaker) |
60% |
|
|
Single sided deaf Adult |
5m (FF Speaker) |
60% |
Without device |
|
|
5m (FL Speaker) |
20% |
|
|
|
5m (BL Speaker) |
0% |
|
|
|
5m (BB Speaker) |
60% |
|
4.3 The monoaural right testing method
This testing method is very similar to the “Monoaural Left Testing Method” and it also involved similar testing conditions, The only variations being that the Speakers were set up on the right side of the subject and the subjects were made to wear headphones in their right ear, playing white noise, to simulate the experience of a single sided deaf person. Note that the unilaterally deaf subject was not made to wear the white noise headphones in their right ear, as that would essentially make them completely deaf. The purpose of this experiment set was to determine how well the subjects could localize sounds using only one ear and the result is presented in Figure 7.
Figure 7. The results of the monoaural left testing method for the single sided deaf adult
This method of testing like the monoaural left testing method shows an improvement in the ability of the single sided deaf adult to localize sound. The localization scores of the single sided deaf adult when tested without the localizer was on average higher than the results of the monoaural left testing method. This is due to the fact that the single sided deaf person has perfect hearing in their right ear and was not made to wear the white noise headphones, The finding of this study is in agreement with similar studies, especially for the back-backward (back center) position which the device couldnt locate the sound. The normal hearing adult without any hearing problem usually have difficulties in localizing sound at the back center position due to the damage of the pinched nerves as a result of the frequent exposure to loud noise overtime, usually with adults from 40 years and above [12, 13].
This study has explored the loss of sound localization sensory modalities and developed a prototype sensory substitution device to overcome the problem of sound localization often seen with the unilaterarily deaf people. The following conclusions can be drawn from our findings;
The authors wish to acknowledge the Automation and Robotics Laboratory in Afe Babalola University, Ado, Nigeria, where the tests were carried out.
[1] Turchetti, G., Bellelli, S., Palla, I., Berrettini, S. (2011). Systematic review of the scientific literature on the economic evaluation of cochlear implants in adult patients. Acta Otorhinolaryngologica Italica, 31(5): 19-327.
[2] Cunningham, L.L., Tucci, D.L. (2017). HHS Public Access. 377(25): 2465-2473. ttps://doi.org/10.1056/NEJMra1616601
[3] Akande, S., Adetunla, A., Olanrewaju, T., Adeoye, A. (2021). UAV and its approach in oil and gas pipeline leakage detection. Journal of Robotics, Article ID 1300740. https://doi.org/10.1155/2021/1300740
[4] Ige, E.O., Adetunla, A., Amudipe, S.O., Adeoye, A., Glucksberg, M. (2022). An archetypal model of a breathable air-circuit in an electro-pneumatic ventilator device. Heliyon, 8(5): 09378. http://dx.doi.org/10.1016/j.heliyon.2022.e09378
[5] Novich, S.D. (2015). Sound-to-touch sensory substitution and beyond (Doctoral dissertation, Rice University)., 1: 1-476.
[6] Oyler, R.F., Oyler, A.L., Matkin, N.D. (1988). Unilateral hearing loss: demographics and educational impact. Language, Speech, and Hearing Services in Schools, 19(2): 201-210. http://dx.doi.org/10.1044/0161-1461.1902.201
[7] Byrne, D., Noble, W. (1998). Optimizing sound localization with hearing aids. Trends in Amplification, 3(2): 51-73. http://dx.doi.org/10.1177/108471389800300202
[8] Firszt, J.B., Reeder, R.M., Holden, L.K. (2017). Unilateral hearing loss: understanding speech recognition and localization variability-implications for cochlear implant candidacy. Ear and Hearing, 38(2): 159-173. http://dx.doi.org/10.1097/AUD.0000000000000380
[9] Kumpik, D.P., King, A.J. (2019). A review of the effects of unilateral hearing loss on spatial hearing. Hearing Research, 372: 17-28. http://dx.doi.org/10.1016/j.heares.2018.08.003
[10] Istrate, D., Vacher, M., Serignat, J.F., Castelli, E. (2004). Multichannel smart sound sensor for perceptive spaces. In Complex Systems, Intelligence and Modern Technology Applications, 691-696.
[11] Vacher, M., Istrate, D., Besacier, L., Castelli, E., Serignat, J.F. (2003). Smart audio sensor for telemedicine. In Smart Object Conference, 222-225.
[12] Holman, J.A., Drummond, A., Hughes, S.E., Naylor, G. (2019). Hearing impairment and daily-life fatigue: A qualitative study. International Journal of Audiology, 58(7): 408-416. http://dx.doi.org/10.1080/14992027.2019.1597284
[13] Haile, L.M., Kamenov, K., Briant, P.S., Orji, A.U., Steinmetz, J.D., Abdoli, A., Rao, C.R. (2021). Hearing loss prevalence and years lived with disability, 1990–2019: findings from the Global Burden of Disease Study 2019. The Lancet, 397(10278): 996-1009. https://doi.org/10.1016/S0140-6736(21)00516-X
