Building a Corpus for the Underexplored Moroccan Dialect (CFMD) Through Audio Segmentations

Artificial intelligence is becoming increasingly powerful these days. It is being integrated into our daily lives in many disciplines, including engineering, medicine, education and economics. There are many instances of conversational artificial intelligence in our daily life. Dialogue systems, conversational agents and personal assistants are systems designed to converse with humans using speech, gestures, graphics and other methods of interaction [1]. Furthermore, a chatbot is a computer programme that reacts like an intelligent entity when spoken to through text or voice, and which understands one or more human languages thanks to natural language processing (NLP) [2]. The chatbot is one of the most elementary and widespread forms of intelligent human-machine interaction. it plays a critical function in many real-life applications, like smart speakers, customer service systems, etc [3]. In general, Conversational AI is a constantly emerging discipline that has attracted the research efforts of natural language processors and companies such as Google, Amazon, Facebook, etc., who have developed speech and language technologies and are currently developing text and voice dialogue systems [4]. They have become more powerful nowadays thanks to developments in Natural Language Processing (NLP) as well as other fields of artificial intelligence. Not only are these developments gaining ground in Latin languages, but they have also extended to Arabic. However, Arabic dialects are not exploited as much due to the lack of corpora in these languages. This is why Arabic dialects Chatbots are not yet available.

Clearly, dealing with dialects is more complicated than Modern Standard Arabic (MSA) in Natural Language Processing for many reasons. First, Arabic dialects are spoken languages, so we do not need to write them down. Second, they differ from country to country. Third, each dialect is further divided into sub-dialects. Therefore, we need to pay careful attention when choosing a specific dialect to work on [5]. On the other hand, Arabic dialects have been little published, and suffer from a lack of NLP datasets. However, the increased use of the Internet and social media. The Moroccan dialect, for example, faces challenges due to the unavailability of corpora, which poses a great challenge in the use of this dialect [6].

Research into understudied dialects in developing countries is crucial to maintaining cultural and linguistic diversity and preventing the extinction of these languages. Whereas most research concentrates on the more formal, wealthier languages. The rarity of research into understudied dialects and the challenges they present are often due to the predominance of mainstream languages and the limited means to collect resources, even when they are available. Language is the main medium for preserving a country's heritage; it is the main reference for traditional knowledge, values, practices, and history over the centuries.

Indeed, there is current work on the Moroccan dialect. Firstly, AIOX LAB worked on an Open-Source Voice Dataset but only on single words [7]. Secondly, Issam and Mrini [8] used the model developed by AIOX Labs Darijabert and other models to summarize a long text. Then, the creation of corpora for sentiment analysis by exploiting social media comments [9]. Unfortunately, all this research is not suited to our research objective.

With a view to harnessing the power of the neural networks, we aim to generate firstly a Moroccan dialect dataset that could help a ChatBot learn Moroccan communication. To strengthen this content, we seek to create an open dataset, as a solution to increase the innovation and application of NLP for the Moroccan dialect. Written content in Moroccan dialect has grown significantly on the internet in recent years thanks to by social media. Therefore, social media has become the appropriate source for collecting data [10, 11]. While we were unable to collect audio data exclusively from the YouTube platform, we will have the opportunity to use this data in the future to train a chatbot capable of understanding speech.

To accomplish this objective, our study involved a series of methodical steps to transcribe the Moroccan dialect videos from the YouTube platform into text. Initially, we chose the appropriate videos with clear audio and general Moroccan dialect. Subsequently, we divided each recording into small recordings (30 seconds). Then, we wrote the content manually to make sure that our dataset included correct words in Moroccan dialect. our study distinguishes itself as the pioneering effort in creating a dataset that combines transcripts of text with corresponding audio recordings, using the YouTube platform to extract recordings and transcribe them manually with the assistance of different editors for quality assurance.

The rest of this article is organized as follows: Section 2 describes related works. Section 3 includes an overview of the Moroccan Dialect at the phonological, morphological, syntactic and lexical levels. Section 4 explains the main challenges concerning the Moroccan Dialect, such as morphological, semantic and syntactic ambiguities Section 5 outlines in detail the methodology of creating the corpus. Results and discussion are presented in Section 6. Finally, we provide a conclusion and lay the ground for future research in Section 7.

3.1 Overview

Arabic is one of the oldest Semitic languages in the world. The development of this language has made it possible to distinguish four different varieties. First, Old Arabic, which is no longer used today, is only present in earlier literature, more specifically in poetry. Secondly, Classical Arabic, also known as Literary Arabic, refers to the official language of Islam’s Holy Book. Third, Modern Standard Arabic (MSA), a more modern version of Classical Arabic and finally dialects. MSA is the official language of all countries in the Arab world. It is used in a range of contexts; namely, media, education, business, literature and in official or juridical written documents [15, 16].

Furthermore, we could categorize Arabic dialects regionally, such as: North African, Egyptian, Levantine, Yemeni, Gulf. Or, Sub-regionally: Moroccan, Tunisian, Lebanese, Jordanian, Kuwaiti, and Qatari [17].

Moroccan dialect poses many challenges, for several reasons:

1. The Moroccan constitution recognizes Arabic and Tamazight as the two official languages of Morocco. The Moroccan dialect is used among Arabic speakers, among Arabic and Amazigh speakers, and among Amazigh speakers of different Amazigh dialects. It is used in informal social settings.
2.  Moroccan dialect is the most used language in Morocco according to the official population census of 2014 (90% of Moroccans use the Moroccan dialect).
3. Only 27% of them could use at least one of the oral forms of Tamazight [14, 18, 19].
4. Moroccan Arabic diverges from MSA at the lexical and phonological levels due to various factors, including Morocco geographic location, colonial history and other considerations. Moroccan dialect has had some linguistic influences from the French and Spanish protectorates through loanwords. Alternatively, Arabic, especially at the lexical level, heavily influences Moroccan dialect as 81% of Moroccan words come from the Arabic language. Table 1 shows some Moroccan dialect loanwords from Arabic language.
5. Moroccan dialect is challenging to understand because of the great distance between Morocco and the Middle East. Furthermore, Moroccan culture is not as widespread as Egyptian or Gulf culture through television broadcasts.
6. Morocco has been colonized by different countries in the past, which has affected its dialect. The Moroccan dialect is influenced by foreign words from the French or Spanish language [5]. To a greater or a lesser degree 11% from French, and less than 1% from both Spanish and Tamazight languages [18, 12, 16].

Table 1. Moroccan dialect loanwords from Arabic language

Nowadays, Morocco has reached approximately 37 million people divided into 12 regions of the kingdom. Each region uses a specific dialect. Moroccan dialect is considered the most diverse dialect, containing urban, Amazigh and Sahraouian dialects. Along with each dialect, there are sub-dialects, for instance Tamazight contains Rifia (of the Rif region), Soussia (of the Middle and High Atlas region), among others [10].

Linguists provide different classifications of Moroccan dialect types. For one, Fatima Sadiqi (2002) distinguishes five varieties of Darija:

1. The Shamali variety in the north of Morocco.
2. The Fassi variety in the center.
3. The Rabat/Casablanca variety around these two cities.
4. The Marrakshi/Agadiri variety in the south.
5. The hassaniya variety in the Sahara.

Other researchers, such as Boukous and Amour, have divided Darija into four varieties only: The Mdini, which designates people living in the city, the Jebli, referring to people in the mountains, the Arubi, meaning the Bedouin population, and the Aribi for Hassani in southern Morocco [16].

As is the case with most Arabic dialects, Darija remains highly under-resourced from a computational point of view (NLP, machine translation, etc.) even though the written content of Moroccan dialect has developed rapidly on the Internet in recent years. Notably, through social media platforms, such as Facebook, Twitter, YouTube, etc [10]. This occurs through diverse forms such as written texts, audio and video documents, by using either the Arabic alphabet or a combination of the Latin alphabet and numbers [12]. Adoption of new modes of communication (SMS, Facebook, Twitter...), which are widespread in Arab countries, have strengthened dialect writing, particularly in Latin characters, the written content based on Latin Script is called " Arabizi " [19]. The next section reviews some of the features of Moroccan Darija.

3.2 Characteristics

3.2.1 Phonological level

The Moroccan Arabic (MA) consonantal system contains 28 consonant phonemes and four vowel phonemes [20] less vowels compared to Classical Arabic, but share the same features. MA uses non-Arabic phonemes /g/, /p/ and /v/, which are mainly used in words borrowed from foreign languages as French [21], for instance, the word Virage meaning a bend or turn, is used in Moroccan dialect; pronounced as /Virage/ but written as (ﭬيراج). The influence of Berber on MA is mainly seen at the phonological level. This influence is shown in many areas through the deletion or addition of a segment [21]. For example deleting Hamza (Arabic همزة : ء). The word الماء (Water) is pronounced in Classical Arabic /Almaa/, but in MA it is pronounced as /Lma/.

3.2.2 Morphological level

Moroccan Arabic is similar to MSA in a few ways. First, the order subject-verb object [22] dominates the composition of a sentence in Darija. Also, the original MSA words keep their morphology, or undergo some changes in their patterns or affixes [23]. For instance, the prefix غا or the word غادي in Darija, indicates the intention of speaker to do the action in the future. Moreover, writing the verb between the prefix ما and the letter ش designate negation (ما بغيتش) which means (I won’t). Furthermore, adding (كـ) to the verb refers to doing something at the same time of speaking (indicative present tense) like (كنرسم) which means I am drawing. Finally, we use ن as a prefix for present tense first person singular (ناكل) which means (I eat) [24].

3.2.3 Syntactic level

Moroccan dialect, like other Arabic dialects, has some syntactic specificities. Unlike English which uses Subject Verb Object (SVO) [25]. Moroccan dialect offers many possibilities for word order. The speaker will usually start the sentence with the element he or she wants to emphasize. In this sense, SVO and VSO can be used interchangeably. However, OSV and OVS are the most rarely applied [21]. Moreover, the disappearance of the feminine plural of the second person أنتن and the feminine plural of the third person هن. Beyond that, regional variations [25] in syntax play a key role in changes of Moroccan dialect. For example, the gender marker in the second person singular in Casablanca is different from Fes, Table 2 gives some Gender marker difference between cities.

1. Word order flexibility and Agreement: a sentence in Moroccan dialect has a subject and a predicate. It could have a simple SVO form, where the subject leads the verb and the complement of the verb. It can also have a VSO Structure, where the verb comes before the subject and the direct object [26]. Moroccan dialect demands complete agreement in person, number, and gender between the verb and the subject, independently of the word order.
2. Pro-drop nature: SA permits missing subjects in the subject position … of clauses [27], on the other hand, Moroccan dialect also drops the pronouns in the subject position, each the sentence below lacks subject yet remains grammatically correct. Table 3 gives an example of Moroccan dialect sentences without Personal Pronouns.

Table 2. Gender marker difference between cities

Table 3. Moroccan dialect sentences without Personal Pronouns

3.2.4 Lexical level

Moroccan dialect consists of Arabic with a number of borrowed words from many different languages. This was a direct influence of the colonization of Morocco in the past by different countries such as France, Spain [21] and Portugal, which affected daily communication. Moroccan dialect vocabulary is rich due to historical events.  Moroccan Arabic speakers use it and so do Amazigh, who represent about 50% of population in Morocco [19], as well as Amazigh speakers who do not understand each other; when the Amazigh variety of each person is not identical. Moroccan dialect includes verbs, nouns, and pronouns, like Arabic language.

3.3 Challenges

3.3.1 Phonological variety

Many phonological ambiguities make dealing with Moroccan dialect challenging due to the different varieties of this dialect:

1. The /q/ variant differ according to geographical regions, for example in Fes, Tetouan and Tangier the community use /q/. In the other cities most people use /g/ for example: /gal/ becomes /qal/, which translates in English as ‘he says’ in Fes variety.
2. Metathesis: the same word keeps meaning with a transposition of letters, for instance (زعما) /z3ma/ becomes (زمعا) /zm3a/
3. The double process of assimilation-deletion: words undergo some small transformation, but the meaning does not change. For purposes of fast connected speech, the word (بنتي) /bnty/ ‘my daughter’ becomes (بتي) /bty/ deleting “ن” but keeping the same meaning.
4. The use of /d/ could be different from one region to another. Some communities use it as a simple /d/ but others use it as a /t/ [25].
5. Intonation: this key point varies from language to language [20]. This is why we are under the obligation of keeping punctuation marks to know the difference between a declarative sentences, interrogative sentences, and imperative sentences.

3.3.2 Morphological richness

Diacritical marks in Arabic give different meanings to the same word, which is the case for Moroccan dialect. Deleting diacritical marks give rise to serious problems while determining the location of the vowels. Moreover, decomposing words occurs in two cases:

1. Decomposition at the level of words: verb, name …
2. Decomposition at the level of grammatical information: noun (singular), verb (1st person) [6].

Thanks to the richness of this dialect, it is common to create a sentence from one word that translates into a five-word English sentence “غيبيعوها” and ‘they will sell it’ [28]. The fact that there is a huge number of vocabulary items in Moroccan dialect compared to English represents a challenge for models based on Machine Learning [29].

3.3.3 Syntactic ambiguity

A sentence has only one meaning when we use diacritical marks, but without them the sentence can be interpreted in several ways, all of which are syntactically correct [6], which we could observe on Table 4.

Table 4. Meanings of sentence

This syntactic ambiguity exists because of the difficulty of adding diacritical marks in conversation.

3.3.4 Lexical ambiguity

Darija is a spoken dialect without rules, so everyone writes the same word in different ways. Many users on the web write Moroccan dialect with the Latin letters, and numbers to illustrate letters, which does not exist in Latin. To deal with this ambiguity, we need to transform all illustrated letters to Arabic letters. Table 5 gives examples of these:

Table 5. Illustration of Arab letters

Moroccan dialects suffer from many problems, including phonological variations, which depend on the geographical region, as well as metathesis, adding that we sometimes have words with the same meaning but with a small transformation, ultimately pronouncing the same letter differently, and intonation can change the meaning. In fact, the large number of vocabulary and diacritical marks in the Moroccan dialect represent a morphological richness and, at the same time, a challenge for models based on machine learning. To mention that the absence of diacritical marks represents a syntactic ambiguity This syntactic ambiguity exists because of the difficulty of adding diacritical marks in conversation. Then there are Arabizi letters, which require us to transform each of them into Arabic letters. Table 5 shows some illustration of Arab letters.

It should be noted that before creating a machine learning or deep learning model, we had to clean the dataset. The cleaning process is a crucial step in preparing our dataset to be understandable by a computer. Cleaning tools are quite poor when we are dealing with Moroccan dialect. Hence, we chose to do it manually. Figure 4 shows a screenshot after a manual cleaning of our dataset. At first, by not only deleting columns without useful information, but also empty rows, and assigning a suitable target for each row.

Data processing is time-consuming, but using the appropriate libraries to do it automatically helps us to reduce time. Although we do not have libraries devoted to the Moroccan dialect, we were able to use Natural Language Toolkit (NLTK), which is a platform for building Python programs to work with human language data. Text pre-processing techniques involves converting the raw data into an understandable structure, focusing on keywords that emphasise the context of the sentence or paragraph. The order in which the NLP pipeline is built has a considerable influence on the result. Tokenization, empty word removal, punctuation removal and lemmatisation are widely recognised and effective text pre-processing techniques [30]. We consider that the pre-processing steps have a significant impact on the accuracy of machine learning algorithms [31].

image006.png

Figure 4. Screenshot of dataset after manual cleaning

image007.png

Figure 5. Screenshot of dataset after automatic cleaning

After removal of punctuation and repetitive characters, we moved to the normalization of the dataset to make it standard and to reduce inflectional forms through stemming. Additionally, the segmentation of the dataset into sentences and words was done through tokenization, which helped us to get each word as a token, thus allowing us to remove stop words using the NLTK library. Figure 5 exposes the result of our automatic cleaning.

The analysis of our dataset allows us to obtain the number of words per theme, the frequency of words used, and some other information that we are going to show in Figure 6 and Table 8 below.

Table 8. Dataset in numbers

	Words Total	Vocabulary Size	Max Sentence Length
Social	1909	974	68
Cooking	10902	3701	62
History	17816	6951	170
Proverbs	8318	3587	71

image008.png

Figure 6. Number of words on each theme

The Figure 7 shows the word cloud of social context content, we observe word frequencies such as "هاد، دابا، الناس، الله، هادي، علاش". These words provide contextual understanding and meaning of the social theme. In another way, words like "شكرا، كامل، الخدمة، كاين" emphasize the importance of the general context, indicating its relevance to the social theme.

The Figure 8 represents the word cloud of cooking content, we see word frequencies such as "شويا، بزاف، ديال، باش، يعني". These words help to make sense of the context and contribute to the meaning of the cooking theme. On the other hand, such words as "الكمية، السكر، الضروري، كنزيدو، الطريقة، البصلة" highlight the importance of the general context, indicating its relevance to the cooking theme through the presence of words linked to this topic.

The Figure 9 exposes the word cloud of historical content, we see word occurrences such as 'ديال، هاد، المغرب، غادي، الناس'. These words contribute to understanding the context and help to convey the meaning of the story's theme. In contrast, words such as "الجيش، بزاف، المغاربة، كلشي، البلاد" emphasise the importance of the overall context, indicating its pertinence to the theme of the story through the presence of words related to this topic.

Overall, as we can see on the figures, words are different for each theme, which could give us a global view about the diversity of our dataset. This dataset contains written and vocal data with over 650 recordings. Each recording has a playback time of 30 seconds and is divided into four classes. In comparison, many of the older corpora do not have specific classes. The CFMD is accessible to the public, whereas the other corpora do not contain voice recordings. For all these reasons, we can see that the CFMD is created with a unique combination that does not exist in other corpora, which gives it added value. Table 9 below shows a comparison of all the corpora mentioned.

image009.png

Figure 7. Word cloud of top words in social

image010.png

Figure 8. Word cloud of top words in cooking

image011.png

Figure 9. Word cloud of top words in history

Table 9. Existing Moroccan dialect corpora