Harnessing the Power of Cloud-Based Big Data Analytics for E-Government Advancement in Morocco: A Catalyst for Development

The burgeoning field of big data is emerging as a potent avenue for governmental investment. It is postulated that E-governments can harness big data to discern patterns and trends in societal behavior gleaned from social networking sites, thereby refining services and enhancing efficiency. The exploitation of big data is posited to engender novel opportunities for value creation and decision support within the realm of e-government activities.

By employing mobile and social networking data, such as browsing histories, purchasing records, and booking details, governments are equipped to gain insights into the habits and preferences of their citizenry. This, in turn, facilitates the prediction of citizen demands and the tailoring of advertising and programs to meet these needs. Furthermore, big data is instrumental in the creation of smarter, more efficient services for citizens, thereby fostering greater speed, transparency, and efficiency in public sector operations.

Despite these benefits, it is crucial to acknowledge that for e-government to deliver substantial value and instigate a revolution in ICT agencies, the right form of big data is necessary. E-government endeavors must acknowledge the pivotal role of effective big data management, comprehending not only its inherent advantages but also the analytical potential it brings along with the technologies intertwined. This acknowledgment serves as the linchpin in ushering in a new era of possibilities, fostering innovation, elevating productivity, enhancing competitiveness, and elevating the overall quality of services. The fundamental essence of big data revolves around the extraction of predictive insights, the refinement of operational capabilities, and the formulation of data-driven rules that contribute significantly to fortifying data governance. Therefore, the integration of big data into the realm of e-government necessitates meticulous contemplation and strategic embedding within the broader framework.

2.png

Figure 2. A Comprehensive look at the framework for big data analytics in digital government

To accommodate different computing needs of government organizations, the framework (Figure 2) offers flexibility in infrastructure solutions, from on-premises data centers to public cloud providers or a hybrid solution managed by the government.

The government infrastructure architecture has four interconnected components: agency cloud, ministry cloud, data linkage and exchange services, and government cloud.

The government infrastructure architecture has four interconnected components.

Agency cloud: This framework provides elasticity to agencies in executing their infrastructure. The agency can choose a cloud, a data center, or any compound solution to process their data. So if an agency chooses clouds for its data processing, they will be entirely managed by that agency itself. So they can independently manage their storage requirements and computing requests.

Ministry cloud: These clouds are managed by the ministries of government. It is own choice of the ministry to make its own data center or to rent a public cloud. If they build their own physical data center, they have to manage the security and availability of resources on their own. Ministries have to help out their junior agencies.  If the ministries choose to use a public cloud, then the data center of that public cloud service provider should locate in the same country for security reasons. A ministry can provide cloud services to junior agencies by using the following ways:

Colocation: In this structure, the agencies have to use their hardware to get high security.

Infrastructure-as-a-Service: The agencies will get the hardware from ministries.

Software-as-a-Service: The ministry will provide the central hardware and software. And all junior agencies will share them.

Data linkage and Exchange services: In this component, a data center is used for data sharing. All government organizations exchange their data through this data center. Data is shared between the users through a proper system. It supports public and private channels and provides user access rights. It also keeps track of all transactions.

Government Cloud: It is the biggest component that provides cloud services to all agencies and ministries on a national level. This will minimize the cost for all agencies and ministries because they don’t have to build their data centers. And the government cloud will also provide high-level security and availability through a Service Level Agreement (SLA). The government can provide different network connection choices for example VPN, intranet, and a general network. The government can also have two data centers so that the second data center can be used if the first gets some problem.

3.1 Human resource development

Human resources growth, specifically in the fields of data science, engineering, and analysis, is just as important as infrastructure development. The government should work to get expert manpower in these areas to support the shift towards data-driven decision-making and organization management.

Our proposed solution involves developing human resources in three domains: business, analytics, and infrastructure. To achieve this, we suggest a three-part solution that can be applied simultaneously:

Short-term project-based training: Standardized syllabus can be designed if government works by joining hands with academia. This syllabus should emphasize hands-on project-based learning for each of the three domains.

Government consulting agency: The government can form an agile consulting agency comprised of experienced data analytics professionals to train government officials and help them implement data-driven technologies and policies.

3.png

Figure 3. Data analytics learning in short-term training for human resource development

Open government data platform: An open platform that gives knowledge, data resources, and best practices to all collaborators, including citizens can be designed. This platform can increase continuous economic growth through public awareness and usage of big data analytics (Figure 3).

3.2 Government data governance

A huge amount of data needs to be processed during digital transformation. This data can be in physical form or digital form. During this procedure, we can achieve high data quality and many other benefits by using data governance protocols. The use of these protocols will provide regulation on data control and data sharing. Figure 4 is depicting the data governance model having four major components.

• Data Governance Committee Building: The committee is made up of project managers, executives, users, custodians, and stewards. The people in this committee make and analyze new policies that can clarify data ownership in departments and procedures of data management. They can get IT systems for data management and storage.

• Data Management Policies: These policies should be made by taking into account four things: Data life cycle, data privacy and security, data quality assurance, and data exchange. Following is the detail on these aspects.
• Data life cycle: Institutes can minimize costs and obey data privacy rules by knowing how data is collected, stored, used, and destroyed in its life cycle. To make the policies, the data must be organized in data sets with additional data (metadata), and saved in an institute-level data catalog for users.
• Data privacy and Security: There should be proper categorization of important data. And only authorized users should have access to this data with limited operations to ensure data integrity. A backup copy of data should always be available to stop the loss of data.
• Data quality assurance: Data set quality can be calculated by using some metrics such as completeness, accuracy, and consistency. Every involved user in the organization should play his/her role to maintain the quality of data.
• Data Exchange: The conditions to use data should be properly defined by the committee. These conditions can include laws, encryption keys responsibility, and encryption standards.

4.png

Figure 4. Data governance model

• Auditing: Through external and internal evaluations, compliance with risk management, laws, and quality assurance protocols should be observed.
• Cultivating Awareness and knowledge-sharing culture: Data governance success is highly dependent on effective communication: So, the agency should organize workshops to raise awareness about the advantages of data governance.

3.3 Government data catalog

With the advancement in technology, a huge amount of data is produced by several sources in different formats. So, it has become difficult to find the needed datasets. For the provision of datasets, the government should make a data catalog. This catalog should contain all the data sets with their additional data (metadata), so users can get desired data directly. This will increase the continuous growth of the economy. Also, government institutes will be able to get standardized data-sharing protocols. As depicted in Figure 5, the data catalog contains three parts; a metadata database, a data directory service portal, and a data linkage system.

5.png

Figure 5. Data catalog architecture

The front of the data catalog is the data directory service portal. This portal allows the user to search the metadata of datasets. Users can apply filters to get their desired data such as tags, data owner names, business segments, etc. The Portal should also secure the sensitive data through an authentication layer. Metadata of the dataset is required for the catalog so that it can work properly; it doesn’t need to store actual data. It stores data about tags, owners, collection methods, access request methods, attribute names, data types, and other descriptions.

6.png

Figure 6. User journey of a data catalog platform

The intermediary component between the database and portal is the data linkage system. Because of this component, users can access actual data by sending API requests. But they should have access rights to send these requests. The system should be linked with the actual data sources. It should have the ability to preprocess data in API requests, and to provide data management tools to data owners so they can categorize their datasets. Figure 6 shows the complete data linkage process of searching, identifying, and requesting datasets.

3.4 Government data exchange

Government-level data need to be shared between data owners and data users. The data exchange platform works behind the data catalog. This platform should be available for both public and private channels. It should only grant access rights when needed. It should be able to perform user authentication, data sharing between institutes, and save records of all transactions. It should be machine-readable without any additional software, and also machine independent. The appropriate formats for data exchange are XML, JSON, and RDF.

For data exchange, there should be assured data quality assurance by the data governance committee. Data quality assurance, authentication features, and data exchange formats are universally accepted, but still, government need to implement rules and opportunities for data exchange with different classification levels.

Figure 7 shows the data exchange architecture. Two types of data exchange platforms are proposed; Government data exchange (GDX) service, and open data sandbox. The first one is for non-public data and the second one is for public data. Both of these platforms coordinate with the government data catalog to provide services to users.

The sharing of government data requires data security such as transaction logging, access control, and user authentication. The dataset with its metadata is required in the data exchange workflow in the GDX platform. Users search datasets and send access requests through the government data catalog. GDX checks access rights and notifies the user who has requested the owner of that dataset. Data transfer options can be SPTF, email, API, and encrypted data in removable storage.

The sharing of public data operates like Kaggle and Sandbox. Where the public not only does data sharing, but also does data analysis, research, and collaboration. Open data sandbox benefits not only citizens but also government officials by increasing access to big data analytics to educational resources. It also increases data literacy.

7.png

Figure 7. Data exchange architecture

3.5 Smart and open government

The complex problems of data analytics can be solved when the government sector (state enterprises, agencies, and ministries) realizes the full capability of storing, analyzing, and managing the data systematically. It can be possible when government provides services to the public, solve their issues, and gives competition to the private sector. The smart and open government also needs to be cost-effective and transparent.

E-government has maximized its initiatives by investing in ICT and engaging with external and internal stakeholders. Investment in ICT helps to make the services better for people and these projects increase collaboration, efficiency, transparency, and e-participation [6]. E-government services are finding the potential of big data to enhance value, efficiency, and effectiveness. Big data contains people's information and e-government uses this data to make services for the betterment of citizen, E-government can provide certain services at less cost and better quality. Traditional government can be transformed into smart government with improved internal business decisions with the power of big data [2, 7, 8].

Big data benefits in e-government are given below [9, 10]:

• Merging efficient big data resource;
• Including important data in e-government for decision-making;
• Faster data generation;
• Bigger income;
• Increased storage volume;
• Enhanced quality life;
• Managed use of e-government resources;
• Improved transaction processing;
• Better transparency levels.

To achieve the above benefits, high-level resources, tools, and people engagement is needed. And this requires efficient use of big data, effective development, better technology, and effort. Policies should be developed for the sake of accuracy, and data security. For policy employment, big data analytics contains tools and applications for e-governments.

Big data can be a particular asset for e-government that can collect valuable information for citizens, businesses, and governments. In a report proposed by Riedad and Hawkins [5], the role of big data in boosting profit, efficiency, quality, and competition are elaborated.

Big data have improved the services and outcomes of many e-governments. These governments employed big data to improve their services and got remarkable results. U.S. government employed real-time analysis systems from big data to get real-time data from thousands of sources. Then, they employed data.gov for the accountability and clarity of government. The government of Michigan developed a warehouse to provide a single source of information for their citizens [11, 12].

In 2012, European Union used big data to know about the economic potential of the public. U.K government was the earliest to employ big data in improving their services. They created a public website (http://data.gov.uk) in 2009 by getting data from seven departments of government. South Korea used big data to join the public and private sectors to serve their citizens in a better way. The Australian government provided access to government data to the general public through a website (http://data.gov.au/) to save their time and resources by providing automated tools [13-15].

In the current era of big data and cloud computing, the amount of energy consumed by big data in E-government's data centers is significant. Unfortunately, data loss often occurs due to equipment failures, power outages, and other unstable factors, leading to information gaps and damage, resulting in losses. Traditionally, this problem is controlled by using the method of rough set theory; this theory is boring in process and only can control little amounts of data [4].

To address this challenge, this paper proposes a complete compatibility theory, which extends the compatibility relations theory [27]. According to Figure 11, the management architecture contains three main things: a cluster monitoring module, sensors, and a data center. Cluster monitoring is used to get the raw data set. And then for data categorization and data filling, this raw dataset gives data. The data centers of E-government face the problem of missing data. To solve this problem, an algorithm is proposed in this study. This algorithm is based on the theory of compatibility and completeness.

11.png

Figure 11. E-government big data system management architecture

The main steps of this algorithm are:

(1) The attribute values of data are discretized.

(2) Break the flow of this data.

(3) Selection of missing data details so that they can be separated.

(4) Sorting of attribute values for further processing.

(5) Put on inverted indexing processing.

(6) Distinguish whether missing data is perfectly compatible or not.

(7) If the data is perfectly compatible, we applied the rule of minimum value.

(8) The resulting attribute is then used to fill in the missing data.

(9) In case the missing data is not perfectly compatible, then the missing attribute is filled with the biggest frequency of the attribute.

In the above process, the Double clustering method is used to break down the data set. And it is classified into small parts according to the divergence of data, and each part contains data with different attributes. Due to the use of the clustering method, cluster data have more similarity between them if there is a smaller and average residual in it. The minimum and average residuals in each cluster are then changed into quadratic shapes. And then the quadratic minima are used for solving missing data values.

The specific algorithm is given below:

The data set is donated by B, and its related set of expression attributes is donated by C. The subgroups of B and C are I and J respectively. And A_ij is the data elements in matrix D. And the average residual of I are calculated as given below:

$Z(I,J)=\frac{1}{|I||J|}\sum\limits_{i\in I,j\in J}{{{a}_{ij}}}\left( {{a}_{ij}}-{{a}_{Ij}}-{{a}_{iJ}}+{{a}_{IJ}} \right)$

where, a_i and a_j are the averages of row i and column j, respectively, and a is the overall average.

Consider a given m x n matrix A and let δ be a fitted value. Let Aij be a submatrix of A, where i and j are subsets of the set of row and column indices, respectively. Let a_ij be the mean of row i of the submatrix, a_iJ be the mean of column j of the submatrix, and a_ij be the mean of the submatrix. If the submatrix Aij satisfies Z(I, J) ≤ δ and 0 ≤ δ, where Z(I, J) is the absolute deviation of a_ij and a_iJ from a_ij, The data will be more similar in the related submatrix if there is the smaller value of δ.

Let’s use a bicluster matrix S that has only one missing data value donated by X. And the total number of columns and rows in S are n and m. And the columns of missing data are donated by q and rows of missing data are donated by p. SUM is used to donate the sum of all values in S excluding the missing value. (1, 2, . . ., p − 1, p + 1, . . ., n) are the values of P, and (1, 2, . . ., q − 1, q + 1, . . ., n) are values of q. And the average residual is calculated as given below:

$Z(m,n)=\frac{1}{mn}\sum\limits_{i=1}^{m}{\sum\limits_{j=1}^{n}{{{Z}_{ij}}}}$

where,

${{Z}_{ij}}={{\left( {{a}_{ij}}-{{a}_{Ij}}-{{a}_{iJ}}+{{a}_{IJ}} \right)}^{2}},{{a}_{ij}}=\frac{1}{mn}\sum\limits_{i=1}^{m}{\sum\limits_{j=1}^{n}{{{a}_{ij}}}}=\frac{1}{mn}(x+SUM)$

The average of data in S is represented by A_p and B_p, where B_p is for column j and A_p is for row i. And the average residual of data clusters is calculated as follows:

$\left\{ \begin{array}{*{35}{l}}   {{A}_{p}}=\overline{{{A}_{p}}}+\frac{x}{n}  \\   {{B}_{p}}=\overline{{{B}_{p}}}+\frac{x}{m}  \\\end{array} \right.$

And then we got the following expression by using the above equation where j represents q and i represent p.

$\left. {{Z}_{ij}}=\left( x-\overline{{{A}_{p}}}-\frac{x}{n}-\overline{{{B}_{p}}}-\frac{x}{m} \right)+\frac{{{(x+SUM)}^{2}}}{mn} \right)$

The quadratic function for the missing data x can be calculated using the following formula:

${{Z}_{ij}}={{c}_{ij2}}{{x}^{2}}+{{c}_{ij1}}x+{{c}_{ij0}}$

Here, cij2, cij1, and cij0 are constants. Now we need to calculate the minimum value. And it is calculated by examining the attributes of minimum value and the characteristics of the quadratic function. A submatrix has more similarity between the data in it if the value of Z(m, n) is smaller. If Z(m, n) is minimum, the formula to find missing data is as follows:

$x=\frac{1}{(m-1)(n-1)}\sum\limits_{i=U}{\sum\limits_{j=V}{{{a}_{ij}}}}$

By using the above-described method, Missing data can be filled very efficiently. The efficiency of this algorithm can be further improved by the optimization of this algorithm.

In the realm of data analysis and handling, various conventional techniques have historically been employed to address the issue of missing values within datasets. Among these methods, one can mention the utilization of rough set theory, the Mean method, the FE method (which involves Discrete Random Forest), and the ERS method (an acronym for Weakly Correlated Random Forest method). These approaches have served as valuable tools in the quest to fill in the gaps in datasets.

However, in the context of this particular study, a pioneering solution emerges in the form of a cloud computing-based system known as CLPD, specifically designed to address the challenges presented by big data in the domain of E-government. This innovative system represents a significant departure from traditional methods and offers a fresh perspective on handling large-scale data.

To gauge the effectiveness and performance of the proposed CLPD algorithm, a comprehensive comparison was conducted with the aforementioned conventional techniques, as documented in Figure 13. The results of this comparative analysis reveal a compelling narrative. Notably, the proposed CLPD algorithm outshines its predecessors, boasting an impressive accuracy rate of 96%. This outcome is a testament to the algorithm's ability to leverage the complete information within the dataset, resulting in a substantial enhancement of accuracy.

13.png

Figure 13. Comparison of the accuracy between different methods for filling in missing data sets

In essence, this study demonstrates the pivotal role that CLPD, the cloud computing-based system, can play in advancing data analysis within the E-government sector. It not only surpasses traditional methodologies in terms of accuracy but also highlights the potential for innovative solutions to revolutionize the way we approach and address data challenges in the era of big data.

The proposed algorithm is far better in accuracy and reasonableness than the rough set theory mean method, ERS method, and FE method. It is because the predicted algorithm has taken into account the wide amount of information, and data dimension, and an efficient decision strategy is used in processing. Also, the proposed algorithm has more speed and quality as compared to previous methods.