A Deep Learning and AutoML-Based Multimodal Text Extraction Framework for Detecting Online Gambling Advertisements in Indonesian Social Media

The rapid growth of social media platforms has fundamentally reshaped how people interact, exchange information, and access entertainment. While this digital transformation brings numerous benefits, it has also facilitated the circulation of illegal and harmful content, including online gambling advertisements. Globally, the online gambling industry reached an estimated market value of USD 63.53 billion in 2022 and is projected to grow further in the coming years. In Indonesia, where internet penetration exceeded 78% in 2023 and social media usage ranks among the highest in Southeast Asia [1], online gambling has become a serious societal and legal concern [2]. Although prohibited under national law, operators exploit the anonymity and vast reach of platforms such as Facebook, X (formerly Twitter), Instagram, and YouTube to promote gambling services to a wide audience, including vulnerable users such as teenagers.

The risks associated with online gambling advertisements extend beyond legal violations. They include social harm, such as fostering addictive behaviors, and economic harm, such as financial losses among players [3]. Moreover, gambling-related crimes and fraudulent schemes often follow, making detection and prevention a matter of public interest. Law enforcement agencies and platform moderators face substantial challenges in identifying such content [4], especially because promotional messages are often implicit, using coded words, local slang, or persuasive narratives to evade automated detection systems [5].

Detecting gambling-related content in text-based social media is inherently challenging due to the high volume, linguistic diversity, and noise in online posts. Messages often contain informal language [6], abbreviations, emojis, and code-switching between Indonesian and English. The morphological complexity of the Indonesian language, with its extensive affixation system, further complicates accurate classification. In addition, gambling promotions are not always embedded in plain text. They can appear in images, embedded within videos, or hidden in spoken audio. This limitation in prior research has left a detection gap, as most automated systems only process raw text fields and ignore content from other modalities.

Several approaches have been proposed to address harmful content detection, yet each carries notable limitations. Traditional machine learning methods, such as TF-IDF combined with Random Forest, have shown strong precision for gambling-related text in Indonesian Twitter data [2]. However, these models rely heavily on handcrafted features and often fail to generalize across diverse platforms. Convolutional Neural Networks (CNNs) are effective at capturing local n-gram patterns in text, yet they are limited in modeling long-range dependencies [7, 8]. Recurrent Neural Networks (RNN), including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), can better capture sequential dependencies and context but are slower to train and sensitive to noise in informal text [9, 10]. Transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) and its Indonesian variants (e.g., IndoBERT) offer state-of-the-art contextual representation [11], but they often underperform when applied to noisy, slang-heavy, and code-switched Indonesian data without extensive domain-specific pretraining. AutoML frameworks like AutoGluon have been increasingly adopted for text classification tasks due to their automated model selection and hyperparameter tuning capabilities [12]. However, their default pipelines may fail to fully capture linguistic nuances in morphologically rich and informal languages such as Indonesian. Prior multimodal studies also tend to focus on isolated modalities, with optical character recognition (OCR) struggling in low-resolution or stylized images [13], and automatic speech recognition (ASR) models suffering from transcription errors in noisy or dialect-rich audio [14].

Based on this review, two major research gaps are identified. The first is the absence of a comprehensive multimodal-to-text framework for Indonesian social media that integrates native text, OCR-extracted text from images, and ASR-transcribed speech into a unified representation. The second is the lack of systematic cross-model evaluations of CNN, RNN, Transformer-based models, and AutoML conducted under consistent preprocessing and hyperparameter settings.

To address these gaps, this study proposes a unified multimodal-to-text extraction framework that consolidates gambling-related content from native text, images, video frames, and audio transcripts into a standardized textual representation. On top of this dataset, a comparative cross-model analysis is conducted to evaluate CNN, LSTM, BiGRU, BERT, and AutoML (AutoGluon) under identical conditions, enabling a fair assessment of their strengths, weaknesses, and trade-offs in terms of accuracy, efficiency, and interpretability.

The contributions of this study are as follows: (1) it introduces a unified multimodal-to-text pipeline that integrates OCR and speech-to-text for detecting gambling-related content across text, images, video frames, and audio; (2) develops a domain-specific Indonesian dataset of 15,286 annotated samples from multiple platforms; (3) conducts a systematic comparison of CNN, LSTM, BiGRU, BERT, and AutoML under consistent settings; and (4) provides practical insights for real-world deployment that balance detection performance with operational efficiency.

This study follows a structured research pipeline consisting of five main stages: (1) data collection and annotation, (2) preprocessing, (3) feature representation, (4) model development, and (5) evaluation. The overall workflow is shown in Figure 1. The proposed methodology incorporates a multimodal-to-text extraction process to unify content from different data sources into a single text-based format before classification.

Figure 1. Research workflow for online gambling advertisement detection in social media

3.1 Data collection and annotation

Data were collected from four major social media platforms: Facebook, X (formerly Twitter), Instagram, and YouTube, following the sequential workflow illustrated in Figure 2. The process began with raw data acquisition, ensuring diversity across text posts, images, and videos. In the second stage, video content was processed by extracting keyframes at fixed intervals, after which OCR was applied to retrieve any embedded textual elements. Similarly, static images, including promotional banners and graphics, were processed through OCR to extract written content. In the third stage, the audio streams from videos were transcribed into text using an ASR system.

This multimodal extraction pipeline enabled the inclusion of gambling-related promotions that often appear in non-textual formats, a common evasion tactic employed by online gambling operators. The final stage integrated all outputs into a unified text corpus, consisting of three primary modalities: (1) native text posts directly obtained from captions, comments, and descriptions, (2) textual content extracted from images and video frames through OCR, and (3) speech-to-text transcriptions from video audio. This standardized representation served as the foundation for subsequent preprocessing and model training. To ensure completeness, video audio streams were also processed independently through ASR before being merged with OCR and native text, guaranteeing that no modality was overlooked.

After the collection and extraction process, the resulting corpus comprised 15,286 textual entries. Each entry was manually annotated by eight trained volunteers who had prior experience or exposure to online gambling activities. This background was considered beneficial for recognizing implicit gambling-related cues.

Figure 2. Data extraction workflow from multimodal social media content into unified text format

Table 2. Presents examples of annotated data for both classes

The annotators labeled the dataset into two categories: Positive, referring to text explicitly or implicitly promoting or discussing online gambling activities, and Negative, referring to text unrelated to gambling, including general conversation, unrelated promotions, or neutral content. Illustrative examples of each category are presented in Table 2, providing clarity on the annotation guidelines applied during dataset construction.

The distribution pattern is illustrated in Figure 3, which provides a visual overview of class balance across the dataset. The final label distribution after quality control was as follows: Positive contained 8,499 entries (55.6%), and Negative contained 6,787 entries (44.4%). This distribution indicates a slight dominance of positive samples, with around 11.2% more than negative samples, which is not considered severely imbalanced.

Figure 3. Label distribution in the dataset

The dataset was stratified and split into three subsets, consisting of 75 percent for training, 10 percent (from the training set) for validation, and 15 percent for testing, ensuring that the class distribution was preserved in all subsets.

3.2 Preprocessing

The preprocessing stage aimed to clean and normalize the collected textual data to ensure consistency and improve model performance. First, case folding was applied by converting all characters to lowercase, eliminating variations caused by capitalization and standardizing word forms. Second, noise removal was performed to remove non-alphanumeric characters, such as emojis, symbols, and URLs. This step also involved cleaning repetitive punctuation marks and excessive whitespace. Third, tokenization was carried out to split sentences into individual tokens based on whitespace, with additional adjustments for separating punctuation from words. Finally, stopword removal was conducted using the Sastrawi stopword list for Indonesian, removing frequently occurring but semantically uninformative words such as dan, yang, and atau. These operations reduced data sparsity and enhanced the signal-to-noise ratio, allowing models to focus on meaningful terms.

3.3 Feature representation

Two embedding strategies were employed: Word2Vec and IndoBERT.

• CNN and RNN (LSTM): Word2Vec embeddings (200-dimensional vectors pretrained on large Indonesian corpora) preserved semantic relationships, generating sentence-level sequences fed into neural architectures.
• BERT-based model: IndoBERT-base-p1 produced contextual embeddings through WordPiece tokenization, with sequences padded or truncated to length 128. Unlike static embeddings, IndoBERT generated dynamic context-sensitive vectors.
• AutoML (AutoGluon): handled feature representation automatically with built-in preprocessing (tokenization, BoW, TF-IDF, n-grams) and optionally pretrained embeddings in ensembles.

This multi-representation strategy ensured fairness by optimizing features for each model type, enabling a consistent basis for comparison.

3.4 Model development

This study implemented and compared four distinct modeling approaches for classifying online gambling advertisements in Indonesian-language social media content. Two approaches were manually designed deep learning models (CNN and RNN with LSTM), one was a transformer-based model (IndoBERT), and one was an automated machine learning approach (AutoGluon). All models were trained and evaluated using the same dataset and performance metrics to ensure a fair comparison All models were trained and evaluated using the same dataset and performance metrics to ensure a fair comparison, as shown in Table 3.

Table 3. Architectures and parameters of classifiers

3.4.1 CNN-based classifier

The CNN architecture was designed for binary text classification, using sequences of pretrained Word2Vec embeddings as input. Each token was mapped to a 200-dimensional dense vector. The architecture consisted of:

• An embedding layer for Word2Vec vector mapping
• A 1D convolutional layer with 128 filters and a kernel size of 5, followed by ReLU activation
• A max-pooling layer to reduce dimensionality and capture local patterns
• A fully connected dense layer with dropout regularization (dropout rate of 0.5)
• A final sigmoid output layer for binary classification

The model was trained using the binary cross-entropy loss function and optimized with Adam at a learning rate of 0.001. Training was conducted for 10 epochs with a batch size of 64, and early stopping was applied based on validation loss to prevent overfitting. The CNN was implemented using TensorFlow/Keras.

3.4.2 RNN-based classifier (LSTM)

The RNN model was implemented using a LSTM architecture, also taking sequences of 200-dimensional Word2Vec embeddings as input. The architecture included:

• An embedding layer for Word2Vec vector mapping
• An LSTM layer with 128 hidden units to capture sequential dependencies
• A dense layer with 64 units and ReLU activation
• Dropout regularization (dropout rate of 0.5) to mitigate overfitting
• A sigmoid output layer for binary classification

The model was trained with binary cross-entropy loss and the Adam optimizer (learning rate 0.001) for 10 epochs, batch size 64, with early stopping on validation loss. This architecture was chosen for its ability to capture long-range dependencies in sequential data.

3.4.3 BERT-based classifier (IndoBERT)

For transformer-based classification, the pretrained indobert-base-p1 model was fine-tuned on the gambling detection dataset. Input sentences were tokenized using the WordPiece tokenizer, padded or truncated to a maximum length of 128 tokens. The classification head consisted of:

• The IndoBERT encoder
• A dense layer with dropout (0.3) applied to the pooled output
• A final dense layer with a sigmoid activation for binary classification

The model was fine-tuned using binary cross-entropy loss and AdamW optimizer with a learning rate of 2e-5 for 4 epochs, batch size 32. The Hugging Face Transformers library was used for implementation.

3.4.4 AutoML (AutoGluon)

The AutoML approach utilized the AutoGluon framework, which provided automated preprocessing, model selection, and hyperparameter tuning. AutoGluon was configured with a maximum training time of 300 seconds per trial. Candidate models included Gradient Boosting Machines (GBM), Random Forests, Bagging Ensembles, and feed-forward neural networks. AutoGluon handled tokenization, feature extraction, and missing value imputation automatically, as well as generating stacked ensembles to improve performance.

By employing CNN, LSTM-based RNN, IndoBERT, and AutoML, this study enabled a direct comparison between manual deep learning architectures, transformer-based language models, and automated machine learning systems. All models were trained on the same preprocessed dataset and evaluated using accuracy, precision, recall, and F1-score to ensure consistency in performance assessment.

3.4.5 Evaluation

To evaluate the effectiveness of the proposed models in detecting online gambling advertisements, this study employed a set of standard classification metrics widely used in binary text classification tasks: accuracy, precision, recall, and F1-score. These metrics collectively provide a comprehensive performance assessment from different perspectives, capturing both correctness and robustness in classification. The evaluation metrics are calculated using Eqs. (1)-(4):

$F 1$-score $=2 * \frac{\text { Precision } * \text { Recall }}{\text { Precision }+ \text { Recall }}$     (1)

Accuracy $=\frac{T P+F N}{T P+F N+T N+F P}$     (2)

Recall $=\frac{T P}{T P+F N}$     (3)

$\text { Precision }=\frac{T P}{T P+F P}$     (4)

where, TP denotes true positives, TN true negatives, FP false positives, and FN false negatives.

In addition to the numerical evaluation, a confusion matrix was generated for each model to visualize classification performance and detect potential misclassification patterns between the positive and negative classes. This visual analysis is particularly important in the gambling advertisement detection task, as certain posts may contain implicit or coded promotional language that is harder to identify. Furthermore, error analysis was performed by manually reviewing a subset of misclassified instances, enabling the identification of recurring linguistic structures or contextual cues that contributed to incorrect predictions.

3.4.6 Hyperparameter tuning

To ensure transparency and reproducibility, a systematic hyperparameter tuning process was conducted for all models. A grid search approach was applied on the training and validation sets, and the optimal configurations were selected based on the highest validation F1-score while maintaining a balanced precision–recall trade-off.

In the case of CNN and RNN models, different batch sizes (32, 64, and 128), learning rates (0.001, 0.0005, and 0.0001), and training epochs (5, 10, and 15) were explored. The configuration with a batch size of 64, learning rate of 0.001, and 10 epochs yielded the most favorable balance between convergence speed and generalization.

For the BERT-based model, experiments were conducted with batch sizes of 16 and 32 combined with learning rates of 5e-5, 3e-5, and 2e-5. The best results were obtained using a batch size of 32 and a learning rate of 5e-5 for 5 epochs. This setup was chosen to address GPU memory constraints while minimizing the risk of overfitting.

With AutoML (AutoGluon), the maximum training time per trial was fixed at 300 seconds. This allowed adequate exploration of candidate models while keeping computational costs practical.

Across all models, a total of 54 hyperparameter combinations were evaluated. This systematic procedure and the rationale behind parameter selection ensured that the comparative evaluation was fair, transparent, and reproducible.

This study proposed a unified multimodal-to-text framework for detecting online gambling advertisements in Indonesian social media, integrating deep learning architectures (RNN, CNN, and BERT) with an AutoML approach. By incorporating textual content from original posts, OCR-extracted text from images, and speech-to-text transcriptions from videos, the framework effectively addresses the heterogeneous nature of gambling-related promotional content in real-world settings.

Experimental results demonstrate that the RNN model achieved the best overall performance, with an accuracy of 93.07% and strong generalization capability across diverse and noisy inputs. CNN also delivered competitive performance (92.80% accuracy), excelling in detecting keyword-rich promotional language. AutoML achieved 90.19% accuracy, showing the practicality of automated ensembling for rapid deployment in low-resource settings. BERT, while stable in convergence, achieved the lowest accuracy (68.12%), highlighting the challenges of applying transformer-based models without extensive domain-specific adaptation to slang, abbreviations, and noisy text.

Error analysis revealed three primary sources of misclassification: implicit or coded promotional language, noise from OCR and speech-to-text transcription, and contextual ambiguity in metaphorical or non-promotional uses of gambling-related terms. These findings underscore the importance of domain-specific vocabulary augmentation, noise-robust preprocessing, and hybrid modeling approaches for improving detection accuracy in real-world deployments.

Compared to prior studies, our framework not only surpasses traditional machine learning methods but also demonstrates competitive or superior performance relative to other deep learning-based approaches in similar detection domains. The integration of multimodal text extraction with optimized deep learning architectures provides a robust and adaptable solution for detecting illicit promotional content in noisy, mixed-language social media environments.

Beyond research contributions, the framework shows strong potential for real-world application in social media content moderation systems. By standardizing multimodal inputs into a text-based format, it can be embedded into automated pipelines to detect and filter gambling promotions across multiple platforms. Lightweight models such as RNN and CNN support scalable, real-time monitoring with low computational overhead, while hybrid or transformer-augmented designs can be deployed for high-accuracy offline analysis. This flexibility underscores the practical value of the framework for both social media providers and regulators seeking scalable solutions to mitigate the spread of online gambling advertisements.

For future work, this study suggests enhancing transformer-based models through domain-adaptive pretraining to better capture gambling-related slang and creative spelling variations, developing noise-resilient feature learning techniques that address OCR and ASR errors while improving multimodal alignment, and designing hybrid architectures that combine the sequential modeling strengths of RNN or CNN with the contextual representation power of transformers. Addressing these areas can further improve detection accuracy and robustness, enabling broader applications of the proposed framework in harmful content moderation, digital security, and social media monitoring.