Performance Analysis of Regex-Based Processing for Dark Web Targeted Crawling

Security intelligence plays a pivotal role in safeguarding organizations and individuals from a rapidly evolving landscape of threats and vulnerabilities. By systematically gathering, analysing, and interpreting data from various sources, security intelligence provides a comprehensive understanding of potential risks and malicious activities. This proactive approach empowers decision-makers to make informed choices, allocate resources effectively, and implement targeted measures to mitigate potential breaches, cyberattacks, and physical threats.

As the dark web serves as a breeding ground for illicit activities, systematic data crawling becomes an essential tool for gaining insights into these hidden realms. By employing specialized algorithms and technologies, security experts can navigate this obscure landscape, collecting valuable data on emerging cyber threats from the extracted dark web content. This proactive approach enables organizations to stay ahead of malicious resources [1], identify vulnerabilities [2], and understand the tactics, techniques, and procedures they employ [3]. To harvest the dark web data, security intelligence developed a dark web crawler.

Focus crawling is a type of crawling that looks for specific subjects using keywords that most accurately describe the target topic. Previous researchers have investigated the focus crawling for surface web [4], hidden web [5], and dark networks [6]. In focused crawling, a regular expression (regex or regexp) is often used to define a pattern for locating specific text within a web page [7], exploring text semantics [8], and even DNA sequence mining [9]. By the use of regular expression, the surface crawlers were successfully developed with high accuracy and classified into certain classes [10, 11].

Over the past five years, there has been significant progress in the field of dark web crawlers, resulting in the development of crawlers that possess both speed and flexibility. The first research concern on dark web is about the content collection for specific topics. Most research in this area proposed a crawler that digs down the dark web to obtain a comprehensive database for topic profiling and classification. This kind of research can be seen in research of Pannu et al. [12] for creating a database of suspicious websites, Alkhatib et al. [6] for market structure and product summary, Yang et al. [13] for hidden threat intelligence, Lee at el. [14] to uncover types of cyber-criminal activities occurring in South Korea, Shiaeles et al. [15] for monitoring and analysis of attack trends in the IoT ecosystem, Shinde et al. [16] to uncover child and women abuse materials, Alharbi et al. [17] to analyze the Tor dark web graph's internal structure and connectivity. The second concern is about the technical aspect of the crawler to gain flexibility and overcome the website crawling limitations, such as crawler traps and captcha. In research of Shinde et al. [16], the relevance of the page and hyperlinks is calculated using natural language processing and ANN-based classifiers to produce a high harvest rate. David et al. [18] proposed the use of information theory to compare widely used distances and consider heterogeneous data to avoid crawler traps. The challenge of link harvesting was researched in the study of Dalvi et al. [19] by the use of a dedicated parser. The CAPTCHA breaking method was proposed in research of Zhang et al. [20] using the generative method. David et al. [18] uses information theory to compare widely used distances and considers heterogeneous data to produce more accurate distances for classification and data mining. Dalvi et al. [21] uses a pre-trained NLP model to determine the relevance score used in content classification and gain a high harvest rate.

From previous research, the regular expression is widely used in surface and dark web crawling methods.  Based on our latest research of Yunelfi et al. [22] the accuracy of Regex classified-based focus crawling can reach 20% higher than traditional filter crawling. However, for the advance of our knowledge, there still no research has analyzed the use of different Regex methods for the dark web crawler. Technically, the different Regex methods allow crawlers to have different speed, flexibility, and accuracy. This is essential to consistently uphold the principles of data integrity in alignment with forensic science standards, ensuring the potential admissibility of the acquired data as evidence in legal proceedings. This significance becomes particularly pronounced in cases involving the forensic acquisition of dark web content.

This research developed a Regex-based dark web-targeted crawler to elevate the accuracy and completeness without degrading the processing time. The crawling procedure was carried out by developing a system that details the dark web's crawling procedure from input to output in the form of URLs for content results categorized by a particular topic. The targeted crawling is developed based on the combination of focus and in-depth crawling. Regex Text, Regex Wildcards, and Regex Optional were used to process keywords in the focused crawling process to improve system accuracy and performance. The in-depth method can enhance the crawling depth by digging the dark web even deeper. White box validation was utilized to verify functions in this targeted crawling system, and the findings show that every function operates as it should. Additionally, in this study, accuracy and performance testing of the system findings were conducted by comparing the three Regex scenarios. Given the known that dark web URLs (Unified Resource Locator) are more complex than regular web URLs [23], it is important to develop a fast and accurate method to crawl the dark web.

This research contribution is stated as follows. First, the application of Regex-based targeted crawling has been successfully developed with more than 98% accuracy. Second, it was discovered that the Regex Optional processing was superior to the Regex Text and Regex Wildcard in terms of accuracy and performance. The combination of in-depth algorithm and focus crawling in the TOR network has been proven to provide high-accuracy crawling results.

In this report, the related issues are elaborated. In Section 2, the advanced research of dark web crawling was summarized. The novelty of this research is discussed in Section 3 by the targeted-crawling system design and implementation setup. The output of the crawling system is discussed and analyzed in section 4, in terms of time processing and crawled data accuracy. Based on the analysis from Section 4, the research conclusion is stated in Section 5.

Focus web crawlers are web crawlers that have been designed specifically for deep browsing [32]. By looking at similar terms, focus crawling searches online pages in a targeted manner [6]. There are three learning phases for the web crawling process of web exploration: content-based, link-based, and brother-based [33].

Focus crawlers discover web resources by navigating through link structures and selecting hyperlinks based on relevance. Focus crawling operates by precisely classifying information based on content relevance and preserving the maximum functionality of collected URLs and searchable forms [34]. This method of operation allows the use of focus crawling for platforms such as the World Wide Web [35], Twitter [36], and deep web data [34, 37]. This has led to the knowledge that focus crawling may be used by any search engine [38].

This research investigated sites on the dark web using a combination of in-depth algorithms and focus crawling techniques. A VPN (Virtual Private Network) and TOR browser are used as supporting software to directly check the website that is being browsed in addition to the targeted crawling technique. The focus crawling was combined with in-depth crawling to produce targeted crawling. The following block diagram shows the steps of the focus in-depth crawling method procedure. The basic model of our research is depicted as a level 0 data flow diagram in Figure 1. The diagram context shows a targeted crawler system that has three entities, namely the user, the TOR network, and the dark web site to be accessed.

1.png

Figure 1. The context diagram of the proposed dark web crawling system

The targeted crawler has two main processes according to focus and in-depth crawling method. The focus crawling handled the keyword processing and content filtering. The in-depth crawling was used to handle the URL processing in the focus crawling.

The system process begins with the user inputting keywords and the parent URL that the system will explore.

After processing the input, the system accesses the dark web with the assistance of the TOR network to reach the provided URL and further explore additional URLs based on the entered keywords. Following the exploration process, the discovered URLs, based on the input URL and keywords, are stored in a dataset under the name of the dark web input. The process is depicted in Figure 2.

2.png

Figure 2. Data flow diagram level 1

The URL processing is then divided into three processes, namely the input verification process, the excluding process, and the fixing process. At DFD (Data Flow Diagram) level 2, there are already 2 data stores, namely the data store for raw URLs and the result URL data store. The raw URL data store contains a list of URL lists placed in the “a href” and “area href” tags. As the URLs contained in this tag are not completely perfect, it then enters the URL fixing process and then stored in a database.

The keyword processing and content filtering in the targeted crawling system is carried out using three processes: Regex Text, Regex Wildcard, and Regex Optional. Regex Text involves exploration based on the text of the keywords entered by the user such in RegexText pseudocode. Regex Wildcard explores keywords with the limitation symbols of a period (.) and/or a plus sign (+) to represent multiple letters in the keyword such as Regex Wildcard pseudocode. Meanwhile, the Regex Optional processing explores keywords with a question mark (?) symbol to indicate that the letters before the symbol are optional such as in Regex Optional pseudocode. The crawled data was then stored in a result database. Figure 3 shows the DFD level 2 of the URL processing system.

Program RegexText (keyword, inputkw, url)

FOR url contains keyword F(i,j) à string inputkw[0…i] matches with string keyword[0…j] IF (inputkw[i]==keyword[j])     PRINT (URL. Keyword)     return TRUE   ELSE     Return FALSE End FOR

Program RegexWildcard (keyword, inputkw, url)

FOR url contains keyword  F(i,j) à string inputkw[0…i] matches with string keyword[0…j]   IF (i<0 && j<0)         return TRUE   END IF   IF (i<0 && j>=0)         return FALSE   END IF          IF (i>=0 && j<0)         return isAllStarts(keyword,j)   End if END FOR

Program RegexOptional (keyword, inputkw, url)

FOR url contains keyword F(i,j) à string inputkw[0…i] matches with string keyword[0…j] IF (inputkw[i]&&keyword[j] not in konten)         return FALSE     ELSE         PRINT (URL. Keyword)         Return TRUE     END IF END FOR

3_2.png

Figure 3. Data flow diagram level 2

The system was tested across various scenarios, including functional testing, accuracy testing, and performance testing. Functional testing involves assessing the system's functionality, with a focus on white-box validation to gauge its compliance with system requirements. During white-box validation, each process within the system was scrutinized individually to determine if it functioned correctly and as intended. This validation process was conducted for all code files, with the expectation of achieving a 100% success rate, thereby confirming the program's full functionality.

The testing and analysis were conducted in dark web TOR network. Due to the large scale of the dark web network, this research only crawls the specific URL which contains the specific keywords. First, the TOR network is activated, and the crawling parameter is set. The TOR connection was done on TOR network through TOR software which integrated with Virtual Private Network. In the experiment, the crawling parameters are keywords, in-depth value, pause time, and URLs. And then, the crawler crawls the dark web and keywords processing.

This research has successfully developed an application that utilizes a targeted crawler. Following white-box validation and testing, the crawler aligns with the system requirements. During performance testing, Regex Optional outperformed both Regex Wildcard and Regex Text, boasting an average crawling time that was over a second faster than Regex Wildcard and 2 seconds than Regex Text. Regex Optional proves superior to Regex Text and Regex Wildcard due to the swift access attempts on the initial access, thus reducing crawl duration.

In terms of accuracy, the accuracy of Regex Wildcard matches that of Regex Optional, standing at 99.14%, which is 4.83% higher than the accuracy achieved with Regex Text. This outcome arose from the observation that registering on Regex Wildcard and Regex Optional is a swifter process compared to Regex Text. Moreover, when using Wildcard Regex and Optional Regex, there have been no instances of access failures during dark web crawling, influencing the quantity of obtained URL results. The best keyword processing method in targeted crawling is Regex Optional, with an accuracy rate of over 99%.

However, the Regex-based processing may suffer a slower process with the rising number and complexity of characters and wildcards. The objective of future research is to improve this drawback by the use of better data processing. The use of an HTML parser for HTML files may increase the crawling time. The use of automata and formal language may improve the accuracy to gain better content analysis of the dark web.