Fake Reviews Detection in Tokopedia Using Fine-Tuned BERT-Indonesia

The development of information and communication technology has driven a major transformation in various sectors of life, including in the fields of trade and consumer services. One of the real impacts of this progress is the emergence of an e-marketplace platform that can allow the buying and selling process to be carried out online without the limitations of space and time. This convenience has made people increasingly accustomed to shopping online, especially through e-marketplaces that provide a variety of products from various sellers in one integrated platform [1-3], one of which is Tokopedia. Tokopedia is an e-marketplace that sells various products from household materials, beauty, cosmetics, tools, clothing, and others [4]. The high public interest in Tokopedia has led to an increasing number of people shopping without caution. This is because many sellers use a mode of asking third parties to write fake positive reviews to increase the popularity of their stores. Quoted from the Tirto Indonesia website with the news title "Beware of Fake Reviews in E-Commerce", that existing product reviews do not really match reality. A study by the research organization Mintel showed that approximately 57% of surveyed customers believe that companies or products only have positive reviews, with no negative ones. Furthermore, approximately 49% said that companies may incentivize those who post fake reviews online. Therefore, a system that can identify genuine and fake reviews is needed, which can then be implemented on a website interface so that users can use it to determine which products to purchase.

One of the technologies or approaches that can be used to create this system is natural language processing (NLP). The field of artificial intelligence known as NLP, focuses on teaching computers to comprehend, process, and produce language. Voice assistants, machine translation services, and search engines are all supported by this technology [5-7]. At the moment, NLP is extensively incorporated into daily life through virtual assistants like Google Home, Alexa, and Siri. NLP is also crucial for companies to obtain a competitive edge in the industrial sector. Applications of NLP can be useful in many aspects of life, particularly in the analysis and value extraction of unstructured data [8]. One of the most prominent models in NLP is BERT (Bidirectional Encoder Representations from Transformers). This model links every output token to all input tokens, with connection weights computed dynamically. Traditionally, language models process text in a single direction—either from left to right or from right to left—but not both at the same time. BERT differs from these approaches because it can analyze context from both directions simultaneously. Owing to this bidirectional capability, BERT is pre-trained for various NLP tasks, particularly masked language modeling and next sentence prediction [9]. Due to BERT's popularity, many people are trying to develop variants of this model for more specific tasks, such as BERT-Indonesian. BERT-Indonesian itself was created by Cahya Wirawan and can be accessed through the open-source website hugging face. This model is an extension of the BERT-Base model pre-trained using Indonesian Wikipedia data using the masked-language-model technique. In its description on hugging face it is stated that the model used is named BERT-Base-indonesian-522M [10].

In previous research conducted by Refaeli and Hajek, which focused on identifying and detecting fake reviews using Fine-Tuned BERT using two types of datasets, namely crowdsourced datasets such as review datasets on Amazon, hotels, and restaurants and Yelp datasets, namely datasets provided by Yelp itself. Yelp is an online review platform that allows users to provide ratings on local businesses based in the United States. The results of this study showed the best accuracy results were obtained on the crowdsourced dataset with an accuracy of 91% and on the Yelp dataset reaching 73% [11]. Furthermore, in research conducted by Alamsyah et al. [12] focused on detecting fake reviews from Tokopedia using the support vector machine (SVM) and Naive Bayes algorithms based on analyzed sentence patterns. In this study, the dataset was scraped from Tokopedia with a dataset of around 887 reviews of slimming products. The results obtained can be seen in the performance of the SVM algorithm which achieved 94% accuracy by classifying 29 new unlabeled reviews into 8 original reviews and 21 fake reviews (Computer Generated) [12]. It is hoped that the results of this study can help people in choosing good and good products when shopping online and can also provide a Fine-Tuned BERT-Indonesian performance that delivers optimal outcomes with a high level of accuracy.

3.1 Model development results

3.1.1 Dataset prepare

The dataset scraping from the official Tokopedia website uses the Python programming language. The tools and libraries used include Beautiful Soap (BS4) and Chrome Driver. The data collected comprises 1,787 items with three columns: user name, review, and star rating, taken from three stores: a clothing store, a sandal store, and a beverage store. The following is an example of the collected dataset. An example of the collected dataset is presented in Table 1. This data will be stored in Google Drive and processed using Google Colab.

Table 1. Dataset

3.1.2 Labeling data

At this stage, the process of assigning labels to data is carried out using the LDA algorithm, which is a topic modeling method widely used in text analysis [13, 14]. LDA works by identifying hidden patterns in a collection of documents and grouping them into a number of specific topics based on the distribution of words that appear. In the context of this research, LDA is used to group reviews obtained from Tokopedia into two main categories, namely real reviews and fake reviews. This process is carried out indirectly through probabilistic analysis, where each review is analyzed to determine its possible association with each topic. LDA calculates the probability distribution of each word in a review and associates the review with the topic that has the dominant probability. To ensure the validity of the dataset, a manual verification process was conducted on a sample of reviews from each generated topic. Based on this manual review, it was confirmed that Topic 0 is dominated by linguistic characteristics typical of genuine reviews (e.g., specific context, natural sentence structure). Consequently, Topic 0 is mapped to the “Real” label (encoded as 1). Conversely, Topic 1 primarily contains linguistic patterns indicating fake reviews (e.g., repetitive templates, overly general or excessive praise), and is mapped to the “Fake” label (encoded as 0). This mapping logic is implemented in the system by evaluating the dominant_topic condition. The use of LDA at this stage not only accelerates the semi-automated data labeling process but also helps identify linguistic patterns commonly found in both fake and genuine reviews, thereby providing a stronger foundation for training the classification model in the subsequent stage.

Figure 4. Labeled data

Based on the implementation of the LDA algorithm shown in Figure 4, the following is the result of the dataset labeled using LDA, as shown in Table 2.

Table 2. Labeled dataset

Figure 5. Comparison labeled dataset

Based on Figure 5, "Comparison Fake vs Real Review (Total: 1,787)," this graph presents a comparison between the number of real and fake reviews out of a total of 1,787 analyzed reviews. The real reviews are represented by the blue bar, while the fake reviews are shown by the red bar. Out of the total, 913 reviews, or 51.1%, were real, whereas 874 reviews, or 48.9%, were identified as fake. Overall, this comparison indicates that the number of real reviews is slightly greater than the number of fake reviews in the available data.

3.1.3 Prepocessing data

In this stage, the author uses three data preprocessing functions: stopwords, tokenization, and stemming, which will be applied to previously labeled data. Stopwords are used to remove meaningless data. Tokenization is used to break down sentences into phrases or words. Finally, stemming is used to remove affixes or suffixes from the broken down phrases. In this stage, the author applies three main techniques in the data preprocessing process: stopword removal, tokenization, and stemming, all of which are performed after the data labeling process is complete. These preprocessing stages aim to clean and prepare the text data to make it more structured and relevant before being used in the model training process. First, stopwords are removed, which are common words in Indonesian that frequently appear but have no significant meaning in the context of the analysis, such as "yang," "dan," "atau," and so on. Removing these words is important to reduce noise in the data and help the model focus on more meaningful words [15, 16]. Can see Figure 6 shows an example of the stopwords removal process.

Figure 6. Stopwords removal

For code implementation of stopwords removal can be seen in Figure 7.

Figure 7. Implementation code stopwords

Next, tokenization is performed, which is the process of breaking down sentences into smaller units, namely words or phrases [17, 18]. By breaking sentences into tokens, the system can analyze language structures and patterns more accurately. Figure 8 shows an example of the tokenize process.

Figure 8. Tokenize

For code implementation of tokenize can be seen in Figure 9.

Figure 9. Implementation code Tokenize

The final step is stemming, which is the process of converting a tokenized word into its root form by removing any affixes attached to the word, such as the prefixes "ber-", "me-", "di-", or the suffixes "-kan", "-an" [19, 20]. Stemming is important so that words with the same meaning but different forms can be recognized as a single entity by the model. Figure 8 shows an example of the stemming process. Figure 10 shows an example of stemming process.

Figure 10. Stemming

For code implementation of stemming can be seen in Figure 11.

Figure 11. Implementation code stemming

3.1.4 Training model

At this stage, the data that has been preprocessed will be split into a ratio of 8:2 for training and validation, with parameters such as Max Length 128, Batch Size 16, AdamW optimizer, Learning Rate 2e-5, and 3 epochs. The configuration and training process are illustrated in Figure 12 and Figure 13.

Figure 12. Set and config model

Figure 13. Training model

3.1.5 Model evaluation

Once the testing phase is conducted using evaluation data, the next step is to evaluate the model's performance. This evaluation is done by utilizing a confusion matrix, as shown in Figure 14. A confusion matrix is a tabular representation that shows the comparison between the model's predictions and the actual labels. From the results obtained, the model was able to correctly identify 304 real data as real and 44 fake data as fake. However, there was 1 real data point that was incorrectly predicted as fake and 9 fake data points that were incorrectly predicted as real. The misclassification of these 10 reviews may be attributed to ambiguous language usage, excessively short reviews lacking sufficient context, or the use of informal words that disrupted the model's semantic interpretation.

Figure 14. Results of confusion matrix

Next, the model evaluation results are continued using the classification report shown in Figure 15. This report presents key metrics such as precision, recall, and f1-score for each class, as well as overall accuracy. Based on the obtained results, the real class (0) achieved a precision of 0.99, a recall of 0.97, and an F1-score of 0.98 from 175 data samples. In comparison, the fake class (1) recorded a precision of 0.97, a recall of 0.99, and an F1-score of 0.98 across 183 data samples. Overall, the model reached an accuracy of 0.98, with both the macro average and weighted average also scoring 0.98. This shows that the model has excellent and balanced performance in detecting both classes, real and fake. This evaluation is based on the amount of evaluation data that has been divided into 358 data points.

Figure 15. Classification reports

3.1.6 Save model

At this stage, the model will be saved with a pkl extension, which will later be used on the web. This file will serve as a reference for detecting genuine and fake reviews on the web-based system. For more details, see Figure 16.

Figure 16. Saved model

3.2 Implementation result

The system implementation phase was conducted using the Python programming language and leveraged the Streamlit framework to build an interactive and easy-to-use web-based user interface. The entire system development process was conducted locally using Visual Studio Code (VSCode) software, which facilitates efficient code writing and live testing. The system implementation consists of two main code files: app.py, the core application component, and one or more supporting files that handle specific functions such as model processing and data management. During system development, several critical functions were developed to ensure seamless integration between the user interface and the fake review detection model. These functions include: a model load function, which initializes the trained BERT model; a function for directly retrieving review data, either through manual input or external sources; and a classification function to determine whether a review is genuine or fake. Furthermore, the system is equipped with a results visualization function, which presents the model's prediction output in an easy-to-understand graphical display; and a page navigation function to allow users to navigate between application sections, such as the main page, user guide, and detection results. The system interface is shown in Figures 17 and 18.

Figure 17. Detect view

Figure 18. User guide

3.2.1 Blackbox testing

At this stage, the system is tested to ensure all the functions created work as intended. The test Table 3 shows the system.

Table 3. Blackbox testing

3.2.2 Usability testing

To assess user satisfaction with the developed service or product, researchers developed a simple survey using the Google Forms platform. The survey instrument consisted of five questions designed to directly elicit user views and responses regarding the system used. This method was chosen because it efficiently collected feedback from a large number of respondents and presented the data in a structured and easily analyzed format. Based on the processed data from the survey responses, an interpretive score of over 87.4% was obtained, categorized as "Strongly Agree." These results were obtained from a questionnaire of 25 respondents, consisting of 11 women and 14 men. Data were collected using a Google Form, with the following percentages obtained.

Figure 19. Diagram question 1

Based on Figure 19, the statement "I feel that product reviews on Tokopedia greatly influence my decision to buy a product." received 25 responses, including 64% or 16 people who chose Strongly Agree, and 36% or 9 people who chose Agree.

Figure 20. Diagram question 2

Based on Figure 20, the statement "Fake reviews on Tokopedia can affect the quality of my shopping experience." received 25 responses, including 60% or 15 people who chose Strongly Agree, and 40% or 10 people who chose Agree.

Figure 21. Diagram question 3

Based on Figure 21, the statement "I will feel more confident buying a product if the product reviews have been checked using a fake review detection system" received 25 responses, including 40% or 10 people who chose Strongly Agree, 52% or 13 people who chose Agree, 4% or 1 person who chose Neutral, and 4% or 1 person who answered Disagree.

Figure 22. Diagram question 4

Based on Figure 22 above, the statement "I would buy products on Tokopedia more often if I knew that product reviews had been filtered to remove fake reviews," received 25 responses, including 28% (7 people) who strongly agreed, 48% (12 people) who agreed, and 24% (6 people) who answered neutral.

Figure 23. Diagram question 5

Figure 23 above, regarding the statement "I agree that technology like Fine-Tuned BERT-Indonesia can identify inauthentic reviews on e-marketplace platforms like Tokopedia," received 25 responses, of which 44% (11 people) strongly agreed, 48% (12 people) agreed, and 8% (2 people) answered neutral. Based on the questionnaire results above, the results were then calculated using a Likert scale (1-5) to measure respondents. The Likert scale score table can be seen in Table 4.

Table 4. Scala Likert

Based on the questionnaire score data in Table 4, the percentage of each answer can be calculated using the following formula.

$Y=\frac{x}{\text { skor ideal }} X$            (1)

Based on the formula above, we can calculate each question on the Google form as follows. The first question, I feel that product reviews on Tokopedia really influence my decision to buy a product, getting a percentage of 92%, the second question, fake reviews on Tokopedia can affect the quality of my shopping experience, I got a percentage of 92%, the third question, I feel more confident buying a product if the product reviews have been checked using a fake review detection system, got a percentage of 86%, the fourth question, I would buy products on Tokopedia more often if I knew the reason why products have been filtered to eliminate fake reviews getting a percentage of 80%, and last question, I agree that technology like fine-tuned BERT Indonesian can identify inauthentic reviews from e-commerce platforms like Tokopedia getting a percentage of 87%. if calculated it will produce a percentage result of 87.4%.

$Y=\frac{92+92+86+80+87}{5}=\frac{437}{5}=87.4$           (2)

This indicates that the majority of users were satisfied and responded positively to the fake review detection system. Therefore, it can be inferred that the developed system has met user expectations.