Spontaneous Speech and Its Features Are Taken into Account When Creating Recognition Programs

The problem of this research work is the presence in the spontaneous speech of characteristic features associated with the company in it of emotions of varying severity, all kinds of noise pauses, and other artifacts that complicate the recognition process, require timely identification and must be taken into account when developing modern technologies for recognizing spontaneous speech. Speech, for their subsequent use in creating a multilingual corpus. Currently, the problem has been studied relatively poorly due to the relatively small number of available scientific publications that objectively reveal the stated topic. For this reason, new, modern studies of the features of spontaneous speech are of particular relevance to take them into account when designing speech recognition programs. In a joint scientific study, Leontyeva and Kipyatkova [1] considered the main features of spontaneous speech that are important in developing programs and systems for automatic speech recognition. According to the authors, when developing intuitive speech recognition models, it is advisable to use noise models and non-phonemic elements. Scientists note that in the future, the development of systems for automatic recognition of spontaneous speech, capable of taking into account its specific features, will eliminate some of the restrictions imposed on the dialogue with the user, which will ultimately make human-machine interaction more natural and productive.

This topic is being developed by Beskrovnykh and Markasova [2] in a scientific study of short tongue protrusion in spontaneous speech. The researchers concluded that short bows of the language in automatic speech make it difficult to recognize, being the equivalent of what was not expressed due to several internal and external reasons. The authors note that the ability of short protrusions of the language to connect opposing positions at the level of replicas of different participants, as well as to combine sound speech and silent reaction into a single text, allows us to consider it as a means of micromimic design of vertical connections in discourse.

At the same time, Sidorova [3], in a scientific study of the prospects for building an integrated approach to the study of the lexical characteristics of a text, draws attention to the fact that any text, as a source and method of transmitting information, must be comprehensively studied, both to form a correct assessment of the level of presentation of the material and to ensure high efficiency of its automatic processing and support of search language services. In his other scientific study, devoted to the study of several problematic aspects of the development of linguistic support for information systems based on ontological knowledge models, the author notes that the successful application in practice of technologies for developing the linguistic support of modern information systems makes it possible to create effective speech recognition programs that can take into account all its features and support at the proper level, the chain of creation and practical application of linguistic resources for the analysis of large text fragments in automatic mode. In addition, according to the scientist, such systems should include tools for creating a linguistic ontology, with the help of which direct knowledge holders can set up the content processing of documents – experts and linguists who do not have exceptional programming skills [4, 5].

The team of researchers represented by Prokofyeva et al. [6], in a joint scientific study, considered the problematic aspects of emotion recognition by speech features. The scientists concluded that the development of automated spontaneous speech recognition systems requires the combined efforts of specialists from various fields of knowledge, in particular, psychologists, psycholinguists, and computer technology specialists, to create effective strategies for their subsequent use in multiple areas of everyday life.

This research aims to study spontaneous speech's main features and their consideration when creating recognition programs. This is especially important given the widespread use of speech recognition programs in modern computer technology and the need to develop practical spontaneous speech recognition algorithms for their subsequent use in this kind of software.

To date, corpus linguistics is actively developing, which involves the development, formation, and search for opportunities for the practical application of multilingual text corpora. A multilingual text corpus is a carefully sorted and specially processed collection of texts in various languages by established rules. In the future, these texts will be used as a system base in linguistic research [7]. Such studies involve conducting a statistical analysis of the occurring speech turns and set expressions, testing the accepted statistical hypotheses in the field of corpus linguistics, and searching for evidence of the studied linguistic rules. The multilingual corpus is one of the study objects in the corpus linguistics section. The classification of multilingual corpora can be carried out by the basic principles, among which it should be highlighted: tasks that are supposed to be solved by creating a corpus; languages in which the texts are presented; genre focus; dynamism. It is customary to divide multilingual text corpora into two main types: parallel, including many texts in different languages, as well as their translations; pseudo-parallel, including exclusively original texts in many languages. To account for variations in spontaneous speech across different languages and cultures requires a cross-linguistic and cross-cultural approach that involves collecting and analyzing spoken language data from a variety of linguistic and cultural groups. This can help to develop speech recognition models that are more accurate and robust across different languages and cultures.

Constructing a multilingual corpus involves collecting a large amount of text or speech data in multiple languages and then organizing and annotating the data for use in natural language processing applications such as machine translation, speech recognition, and sentiment analysis. The general steps for constructing a multilingual corpus are: define the scope and goals of the corpus; collect the data; preprocess the data; annotate the data; align the data; evaluate the corpus; publish the corpus. Overall, constructing a multilingual corpus requires careful planning, data collection, and annotation, as well as rigorous evaluation and publication of the corpus to ensure its usefulness for natural language processing applications.

Spontaneous speech as an object of study in linguistics is considered in the context of the prospects for its subsequent processing in special recognition programs. Spontaneous, extemporaneous speech is delivered under conditions of constantly changing external circumstances and communicative conditions. Due to the above cases, automatic speech should be attributed to one of the critical indicators of the language competence of the person pronouncing it, as well as a manifestation of his general cultural level. Creating a program for automatic recognition of spontaneous speech to create a multilingual corpus involves using approaches that go beyond the trivial adaptation of known models of foreign languages. Automated text processing, both in any new, previously unknown language and in a familiar language, is associated with numerous difficulties that are caused by the characteristics of a particular language, as well as the characteristics of spontaneous speech in this language. There are several unrelated factors that can make spontaneous speech recognition difficult, including:

1. Spontaneous speech is often accompanied by environmental noise, such as traffic, machinery, or other people talking. This can interfere with speech recognition algorithms and make it harder to accurately transcribe speech.
2. People from different regions or with different cultural backgrounds often have unique accents and dialects, which can be challenging for speech recognition software to understand. These variations in pronunciation and intonation can make it difficult to accurately transcribe spoken language.
3. People may speak at different rates, use fillers such as "um" or "ah," or pause frequently during speech. These variations in speaking style can make it challenging for speech recognition algorithms to accurately transcribe speech.
4. Disfluencies such as stuttering, repetitions, or false starts can make it difficult for speech recognition software to accurately transcribe speech. These interruptions can cause the algorithm to misinterpret what is being said.
5. Speakers may have different voices, pitches, and speaking styles, which can make it difficult for speech recognition software to accurately transcribe speech. The algorithm must be trained to recognize and differentiate between different speakers.
6. Spontaneous speech may contain informal language, slang, and colloquialisms that may not be recognized by speech recognition algorithms. Additionally, grammar and syntax variations can further challenge recognition accuracy.

All of the above factors significantly complicate the work of the spontaneous speech recognition program and reduce its overall performance [1, 8]. This predetermines the need to develop unique algorithms for transcribing word forms contained in spontaneous speech for the purpose of its subsequent synthesis, as well as the assimilation of pronounced sounds. Recognition programs use several techniques to overcome difficulties with spontaneous speech, including:

1. Speech recognition programs use noise reduction algorithms to filter out background noise and isolate the speaker's voice.
2. Acoustic modeling techniques use statistical models to capture variations in pronunciation, speaking style, and dialects. This allows the algorithm to recognize speech patterns more accurately.
3. Language modeling techniques use statistical models to analyze the structure and context of spoken language. This allows the algorithm to recognize informal language, slang, and colloquialisms more accurately.
4. Speaker adaptation techniques allow speech recognition programs to adapt to a specific speaker's voice and speaking style. This can improve recognition accuracy and reduce errors.
5. Disfluency handling techniques enable speech recognition programs to recognize and handle interruptions, repetitions, and other disfluencies that are common in spontaneous speech.
6. Vocabulary and grammar adaptation techniques allow speech recognition programs to adapt to the specific vocabulary and grammar used by a particular speaker or in a particular context. Overall, recognition programs use a combination of these techniques to improve recognition accuracy and overcome the difficulties associated with spontaneous speech.

11.png

Figure 1. Algorithm for transcribing word forms, taking into account the provisions of sound reduction and assimilation

Source: Created by the authors based on Ref. [1]

Transcription algorithms for word forms contained in spontaneous speech are complex software tools designed to distinguish between the correct sounding of word forms and then determine their semantic meaning in the given context [9, 10]. Such algorithms include a given set of modules, the tasks of which are to accept a particular set of word forms, with their subsequent transformation into separate phonemes, and the implementation of the reduction principle. The end result, which is pursued at all stages of the process of functioning of a given algorithm, is to obtain a set of transcriptions that correctly describe the semantic essence of the spontaneous speech stream processed using the recognition program. Figure 1 shows a schematic representation of the word form transcription algorithm, considering sound reduction and assimilation provisions.

The word forms w entering the transcription module are converted into phonemes V1 and V2 by applying the introductory phonetic data processing provisions. Afterward, phonemes suitable for reduction are selected in the intraword and interword reduction modules. Then the selected phonemes are processed, selecting all possible combinations of their forms, which are automatically generated. After that, all generated combinations of phoneme forms are automatically processed separately. The result of the described process is obtaining a particular set of typical transcriptions (V1, V2, ...) of the original word forms (W1, W2, ...) when creating which all the features and options for reducing automatically generated phonemes were taken into account [1]. Spontaneous speech can rightly be classified as a unique linguistic phenomenon for which certain deviations from the norm are typical [11]. Among the characteristic features of spontaneous speech, which can be attributed to the category of deviations from the norm, expressed in a calm, smooth flow of speech flow, one should includeи:

– independent corrections by the speaker of the sounds, words, and phrases he utters;

– false starts, impulsive, thoughtless beginning of a sentence or a single phrase;

– failures in the pronunciation of some grammatical forms, as well as the established order of words and phrases;

– frequent, spontaneous breaking off of words and phrases without their expressed logical conclusion;

– constant reservations and problems with the pronunciation of individual sounds;

– speech repetitions;

– stammering without bringing the expressed thought or consideration to its logical conclusion.

When developing spontaneous speech recognition programs, one should consider the need to use elements that can detect deviations from the normal flow of smooth speech, as well as external interference and extraneous noise that interfere with the high-quality implementation of the automatic recognition [12-14]. To implement this task, it is advisable to introduce unique sound correction algorithms into the program that can provide a qualitative account of the characteristic features of the flow of spontaneous speech and refine the data at the program’s output. Correction of the sound stream of spontaneous speech through the use of this algorithm provides recognition of almost any fragment of spontaneous speech, achieves higher sound quality, and, consequently, improves the overall quality of spontaneous speech recognition [15]. In order to obtain experimental confirmation of the effectiveness of the audio stream correction algorithm, a test was conducted using the accumulated language base, which includes over 28,000 speech fragments (approximately 15 hours of the audio track). The results of the experiment are presented in Table 1.

Table 1. Comparison of the change in the accuracy of the spontaneous speech recognition program

As can be seen from the data presented in Table 1, when using each object of recognition, there is a decrease in the error when introducing the specified audio signal correction algorithm by an average of 19-65%. At the same time, the accuracy of recognition of the sound signal of spontaneous speech increases when using the algorithm for correcting the sound signal. The experiment results indicate the effectiveness of the practical implementation of algorithms of this type in speech recognition programs, which necessitates further research into the prospects for their development. The development of modern spontaneous speech recognition programs is associated with the need to consider its characteristic features, which are of fundamental importance in ensuring the final product's high quality. In this case, absolutely all features of the sound signal of spontaneous speech received at the input of the transcription module, as well as the nature of the transformation of word forms included in this sound signal into phonemes, should be taken into account [16]. With any changes like the incoming sound signal, the operator of the speech recognition program must make all the necessary adjustments to the program, taking into account the features of changes of this kind and the specified parameters for the clarity of recognition at the output. The creation of such a software system for the recognition of spontaneous speech implies the need to test this software, taking into account the above features and the ability of the developed software to account for and recognize them effectively. Figure 2 shows a diagram of a test software model for evaluating the effectiveness of the designed spontaneous speech recognition software.

The interaction of the software package with the operator is provided by the capabilities of the graphical user interface, which includes the means of interaction with the main module. The module for interaction with speech recognition systems is also associated with the main module. Its primary function is to ensure the transfer of recorded spontaneous speech directly to the system and to receive spontaneous speech recognition data. In addition, the module for interaction with speech recognition systems performs the functions of obtaining and saving information about the speed of the spontaneous speech recognition process and saving all the received statistical data. Sound recording and playback modules directly interact with the software audio system. Their essential functions are the recording of recognizable spontaneous speech, as well as its subsequent playback directly in the software system. The main module is the central part of the test model for evaluating the effectiveness of spontaneous speech recognition programs. Its primary function is to ensure smooth interaction between all nodes of the software complex and keep it in working condition at all stages of the above technological operations [17].

2.png

Figure 2. Model for evaluating the effectiveness of spontaneous speech recognition software

Source: Created by the authors based on Ref. [17]

Testing of spontaneous speech recognition programs should determine the recognition accuracy of both individual phrases, taking into account the sequential conversion of word forms into phonemes entering the transcription module and the entire text as a whole, taking into account the actual capabilities of the software interface itself. In this case, one should also consider the probability of errors when sending commands from the head module to applications and the possibility of inconsistent operation of individual parts of the test program as a whole. For a qualitative solution to such tasks, the operator must constantly monitor the progress of the operations performed and the state of the entire software system as a whole and its individual nodes in particular [18]. The main difficulty in using automated spontaneous speech recognition systems lies in the fact that such speech is characterized by the complete absence of a pre-prepared form. Therefore, it also does not contain any verbal messages or direct participation of persons directly conducting the dialogue. At the same time, such parameters as the pace of the speech flow, the manner in which the speaker pronounces individual phrases and phrases, as well as excessive emotionality against the background of an unpredictable large volume of phrases used, vary unpredictably [19, 20]. All this significantly complicates the development of spontaneous speech recognition programs and predetermines the need to take into account all these features both directly in the software development process and at all stages of its subsequent operation in order to obtain optimal results, expressed in the high final quality of spontaneous speech recognition by modern software, methods.

Improved spontaneous speech recognition technology has a wide range of practical applications across various industries. Spoken language recognition can be used to automate customer service interactions, such as call centers or virtual assistants, which can reduce wait times and improve customer satisfaction. Speech recognition technology can assist healthcare providers with dictation and medical transcription, which can improve the accuracy and efficiency of medical documentation and patient care. Speech recognition technology can be used to create automated language assessment tools that evaluate students' language proficiency, helping educators provide more personalized language learning experiences. Lawyers and legal professionals can use speech recognition technology to transcribe and document legal proceedings, such as depositions and court hearings, which can increase efficiency and accuracy. Police officers and first responders can use speech recognition technology to transcribe interviews and record incident reports, improving the accuracy and timeliness of incident documentation. Spoken language recognition can be used to control Internet of Things devices, such as smart speakers, thermostats, and lighting systems, making it easier for users to interact with their devices hands-free.

However, there are several potential limitations and challenges associated with implementing improved spontaneous speech recognition technology in real-world settings, including:

1. Collecting and processing large amounts of speech data for speech recognition applications raises concerns about data privacy and security, especially when dealing with sensitive personal or medical information.
2. While improved speech recognition technology can better handle variations in pronunciation and speaking styles, it may still struggle to recognize accents or dialects that are outside of the trained models.
3. Speech recognition technology may struggle to accurately transcribe spoken language that contains slang, colloquialisms, or domain-specific terminology.
4. Spontaneous speech recognition can be hindered by environmental noise, such as traffic or other people talking, which can interfere with speech recognition algorithms and reduce accuracy.
5. Implementing improved speech recognition technology can be costly, especially for small businesses or organizations with limited budgets.
6. Introducing new speech recognition technology can require changes to existing systems and processes, which can be disruptive and time-consuming.
7. Some users may be uncomfortable with using speech recognition technology or may prefer other methods of communication.