Data Deduplication and Integrity Check Scheme Based on Unique Tags in Edge Cloud Computing

Multi-user cloud storage solutions with shared networks are gaining popularity due to the constant increase in data creation on a global scale. However, many users still need to be more hesitant to move data to remote storage due to concerns about data security. Encrypting data before it leaves the owner's property is a standard procedure. While this method is beneficial for safety, it hinders storage efficiency features such as compression and deduplication. These features allow storage providers to utilize their back-ends more effectively and serve more customers with the same infrastructure.

However, because they no longer own this data locally, service providers find it challenging to maintain complete control over their data housed on dispersed edge servers. In the highly distributed edge computing environment, edge servers should not be taken for granted since hardware and software exceptions might occur accidentally or maliciously. When it comes to edge computing, the issue intensifies considerably. The unique characteristics of edge computing allow us to maintain traditional methods of maintaining data integrity with edge data from cloud service providers.

First, edge servers typically have low processing power, whereas most cloud data integrity techniques assume and leverage the near-limitless computing power of cloud servers. Then, to serve regional customers, service providers typically cache their viral data on edge servers in specific locations. The service provider cannot look at each edge data point separately to find the problematic ones because of excessive processing and bandwidth consumption. One technique now being used to guarantee cloud data security is proven data possession (PDP) and its variations. They are intended to enable users of thin clients to seek data integrity challenges from major cloud storage providers such as Google or Amazon.

Data deduplication is when a storage provider keeps just one duplicate of a file (or portion of a file) that several users hold. Four distinct deduplication methods differ based on whether deduplication happens at the file or block level and whether it happens on the client or server side. Because client-side data deduplication ensures that many uploads of the same content only consume a single upload's network bandwidth and storage space, it is preferable to server-side data deduplication.

Although security was not included in the initial deduplication designs, it rapidly became necessary when users sought data protection. Deduplication seems possible with convergent encryption, which uses plaintext as the encryption key. This technology is secure and easy to use. Regretfully, it was shown to be susceptible [1]. Moreover, a general impossibility conclusion contends that basic convergent encryption techniques cannot provide traditional semantic security.

Figure 1 describes deduplication in edge cloud using the chuck files to store the cloud, first segmenting the file in a cluster set to reduce the similarity hash contents. Reduce the number of computation operations for each hash that computes a part of the hash and minimize duplication. This further accelerates the decoupling process while achieving the same decoupling result. On the other hand, tags assume a user-based response mechanism instead of duplicating files, so cloud users cannot cheat users using duplicate check results when files are uploaded. Complete, and the same checks will be ignored. Therefore, deduplication is assumed to exist.

1.png

Figure 1. The basic flow of deduplication in the cloud

A lot of storage space is one of the many resources a cloud service provider offers its clients. Managing the constantly expanding volumes of data in the cloud is one of the main problems. In cloud edge computing, the DD approach improves data management's scalability. However, the primary issue with data deduplication is security. To overcome this issue, this study provides a safe data deduplication system via an integrated cloud-edge environment using convergent and Unique Tag Deduplication Integrity Check (UTDIC) methods. This algorithm classifies people according to their traits. Depending on whether the number of owners of a collection of data matches the degree of popularity, the data set may be considered duplicate.

2.png

Figure 2. Proposed system

3.png

Figure 3. Diagram for edge could deduplication

Figure 2 provides a proposed system for Deduplication using in the cloud and then generating the chunk file to reduce the duplicates, initially preprocessing using Threshold Similarity Measures (TSM) to estimate the text and copies are removed, then segmenting the cloud folder using CISWH for storing the file in cluster format and ranking the order based on the cloud for hashing identification. Checksum Selective Indexing Chunking (CSIC) is finding the selective index range for chunking content using Verify the integrity of deduplicated data by integrating a checksum or error-checking appliance. Prefetching the local index means allocating the store's frequently accessed data blocks to the local cache. This reduces redundant operations on processed chunks and improves overall deduplication speed. Finally, the Unique Tag Deduplication Integrity Check (UTDIC) is a validation mechanism to verify the uniqueness of the tag when creating or assigning a new tag. This includes checking the repository to make sure the suggested tag is correct.

3.1 Edge cloud deduplication model

Only on centralised cloud servers does edge computing allow data to be processed closer to the point of generation. The process of eliminating multiple copies of repeated content is known as Deduplication. It is frequently used in storage systems to boost data speed and reduce the amount of storage space needed.

Figure 3 shows a balanced design for end users and end service providers. This edge computing solution consists of user, storage, and management nodes for error file reduction and Deduplication.

3.2 Preprocessing data using Threshold Similarity Measures (TSM)

Preprocessing uses a word-based similarity metric to discover inputs that surpass a predefined similarity threshold. Sets the point at which inputs are deemed identical. Using a similarity metric based on threshold points can help to decrease duplicate files in cloud storage. To avoid duplicate data sets, use thresholding similarity algorithms to locate related texts and delete unnecessary information.

${T h e r}(t e x t)=1-\frac{\sum_{x=1}^N S c_x(t x t)}{N}$           (1)

where, Thre(text)  is the threshold of a particular text, and the N number of records in cloud data, index (x) SCx (text) is a similarity finding, and SCx (text) =1.

If the x^th values of a particular similarity score is between 0 and 1, cloud records X1 and X2 of records calculating similarity score values of texts. If the similarity score range between two values of a field is greater than or equal to that field's range, then the Similarity Score (SS) is defined as:

$\begin{gathered}{S c}(x 1, x 2)=t_1 * V_1(x 1, x 2)+t_2 * V_2(x 1, x 2) \\ \text { Similarity }=0 ; \\ \text { For } \mathrm{n}=1 \text { to X-1 do } \\ \quad {Thre}(n)=V(n)-V(n+1) ; \\ \quad \text { While } {Max}(S c)>0 \\ \mathrm{X}=\text { finsTh.sc }(\max ) \\ \\ \quad \text { Swap }(V(n)-V(n+1)) ; \\ \quad \text { Remove similarity; } \\ \text { End while }\end{gathered}$

X1 and X2 are input strings or records to be compared field by field. T1 and T2 are weighted by the similarity and string similarity algorithms, respectively, and are equal (t1=t2=0.5). V1 represents the similarity score, and V2 represents the string similarity score. This code shows the algorithm for obtaining the similarity class. It is an iterative process that extracts the outermost node from the entire set.

3.3 Cluster Index-Based Sliding Window Hashing (CISWH)

Compared to reduced window or landmark models, very few research has concentrated on clustering algorithms that use sliding windows. A CISWH algorithm that is effective. The technique seeks to deliver quick, excellent clustering outcomes. Compared to previous methods, our method's unique data structure and process can perform full-range clustering on tuples and reduce the computational cost of operations such as insert, delete, and search. Sliding window aggregation divides the window into blocks and aggregates the summaries of these blocks to create a window.

Our method uses hashing, whereas the other uses a tree structure to find the nearest CF. The ' current state is S1=c1, c2, S2=c3, S3=x4, and the input variables are c1, c2, ... (ci+1 is fresher than ci). The current loads are combined when a new tuple, c5, appears, and a new bucket with the new tuple is created, with S1=c1, c2, S2=c3, c4, and S3=c5. A sliding window advances and deletes loads that have expired timestamps. Nonetheless, the timestamps within a container could differ. If the sliding window in the example has a size of 4, it should remove the tuple x1.

3.3.1 Sliding window

Only arrays with timestamps from the window's start to the current timestamp are included in sliding windows. In particular, the timestamp of the tuple $x_i$ and the current timestamp t can define the window as a weight function.

$\boldsymbol{s w}\left(x-x_i\right)=\left\{\begin{array}{l}1, { if }\, x-x_i \leq V \\ 0, { if } \, x-x_i>V\end{array}\right.$          (2)

where, V is the sliding window's temporal range, the window removes the Cluster's loads based on the vectors. When another 100 tuples come, the previous 100 are removed from the window, and the new 100 are attached. The terms "a window slides" and "a window moves" indicate that when set SLIDE to V, the window's oldest V tuples are eliminated, and fresh V tuples are inserted.

First, all possible starts of the range are repeated. Each range's elements are iterated from V to V+n-1, and their sum is calculated.

Depending on the sliding condition and time unit, sliding windows can be classified as either time-based or triple-based. An index-based window for clarity, while a time-based window may also be examined using the same techniques.

3.3.2 Cluster index

A clustered index is constructed using a column or group of columns that uniquely identify each entry. This makes index maintenance easier.

Assuming the incoming data streams $x=$ $h x 1, t 1 i, h x 2, t 2 i, \ldots$, I cannot fulfill that request for the sliding interval $L=3$, with 3 clusters ($c=3$) and a window size $\mathrm{R}=9.\,|C|=5$.

The number of tuple structures, their square sum, and their linear sum are all maintained by the Clustering Feature (CF). If the radius of a new tuple is less than a predetermined threshold, the nearest CF absorbs it.

$\mathrm{I} \leq C^I$ is the clustering index range to be calculated based on the data distance evaluation using weights $w_1, w_2$,

$\left(w_1, w_2\right)=\left\|w_1-w_2\right\|^2=\sqrt{\sum_{x-1}^w\left(\left|w_1 x-w_2 x\right|^2\right)}$          (3)

${Const}(w, v)=\sum_{x \in V} {range}^2(w, v)$         (4)

${Const}$ $_w(w, v)=\sum_{x \in S} w(x) \cdot$ ${weight}$ $^2(w, v)$                 (5)

Aggregation aims to reduce the total squared distances between each tuple in S and the nearest range. The clusters have a weight function value with a sliding window. Figure 4 illustrates the cluster indexing grouping.

${Const}$ $(w, v)=\sum_{x \in V} {range}^2(w, v) *\left(t-t_w\right)$                 (6)

${Cont}_{O p t}^n(w)=\min _{C \in { range }^2,||C||=n} {const}(w, v)$           (7)

4.png

Figure 4. Cluster index grouping

The clustering sliding window is processed in the cluster center, and the indexing range follows each Cluster; the quality of the clusters depends on the index cluster centers.

3.3.3 Cluster index hashing with Sliding window

$x_1=0, x_2=0 ;$ cont $=0$         (8)

Sliding window cluster index (data)

While x₂  is less, the counting of the clusters

$x_2=x+1$          (9)

Error rate Index. Cluster (Data, x₁, x₂)

If index err is generating the Hash

In addition, blocks often contain features much deeper than the average sort depth, which increases the RAM required to store hash tables in a particular location. Instead of a hash table with conflict resolution, use a hash table H with fixed size h without conflict resolution. Differently named alignments are assigned the same hash value.

3.4 Checksum Selective Indexing Chunking (CSIC)

A checksum is a value calculated by summing the bits in a data collection. It is used to identify mistakes in data transfer or storage. Selecting specific data or material for inclusion in an index is called selective indexing. In databases or search engines, it entails choosing which qualities or fields to include in an index to permit faster and more efficient search operations. Indexing Chunking is the technique of dividing vast information into smaller, more manageable "chunks." It is frequently used for memory and learning.

Input: Default values, Length of sliding window, W

Output: Index selection

Function Checksum Chunking (File, values)

5.png

Figure 5. Chunking process

Figure 5 shows a comparison of the Chunking Method. (A) While fixed-size chunking is fast, it has a boundary-shift problem. (a) The issue can be resolved using variable-size chunking, but each byte requires some processing. (c) This problem can be improved with the time complexity of O(1) by using the constant-time chunking that this study suggests; variable-size and reverse chunking, on the other hand, only identify the first and final chunks, leaving a sizeable middle chunk c.

File types and sizes are represented as an array of bytes in the input.

Different-sized File Chunks, Max Th and Min Th, are generated.

Step 1: Configure the File Length (f).

Find the Length of the L.

Set the Endpoint to 0 and the Startpoint to 0.

Step 2: Though Not the End of the DocumentIf Len is equal to zero, then move on to Step 5. Should Len match Min Th

Proceed to Step 3 after setting the File Chunk Bytes from Startpoint to (Startpoint+Len).

Else If (Min Th.+L)=Len, Then

Proceed to Step 3 after setting the File Chunk Bytes from Startpoint to (Startpoint+Len+L).

Endpoint

Bytes of File from Endpoint+L should be selected as the window size.

Step 3: End

Additionally, chunking files with minimum and maximum values are sometimes used as a hard barrier to prevent massive storage of chunk size variation factors.

3.5 Prefetching Local Index Caching (PLIC)

Prefetching Local Index Caching is an approach used in computing to increase system speed by retrieving data or instructions on a local system or device before generating an index rather than depending on a centralized index. Reducing the need to query a central index can improve access times for frequently used data. Caching is keeping copies of frequently requested data at a location that enables quicker retrieval. Local index caching may refer to retaining a cache of commonly used index data on a local system to speed up search or retrieval processes.

$C_n=\left(f_i v: v \in W, w \in|w|=t, x \in|N|\right)$              （10）

$c_d=\left(\oplus \in W ; \frac{f_i v}{\lceil s\rceil}: S \subseteq|N|,|s|=t+1\right)$            （11）

Which requires broadcasting $v \in W: S$ bits. Note that user $\mathrm{k} \in \mathrm{S}$, for S ), wants $\frac{f_i v}{[s]}: S$ and has cached $t \in W$ for all $\mathrm{s} \in \mathrm{S}: \mathrm{s}$ $6=\mathrm{k}$ so that it can recover $\frac{f_i v}{[s]}: S$ rom Xt and the cache content.

$w_{x, y} {Max}(t)=\frac{\binom{N}{T+1}-\binom{N-\min (n, m)}{t+1}}{\binom{N}{t}} \geq V^x$           (12)

${ }_{T+1}^N$ With $t \in v$ the sub set t - an integer, one takes the lower convex envelope of set of points $(\mathrm{x}, \mathrm{v})=^{N-\min (n, m)}$ for $\mathrm{t} \in[0$ : $\mathrm{n}]$.

$\epsilon_n:=\max _{x \in[n]} {Pre}\left\{w_x\left(x^n,(N: x \in i)\right) \neq D_v\right\}$          (13)

$R \in \bigcap_{x \in|n| X v \prime C i \in V} \bigcup R\left(\frac{N}{P}+C i\right)$           (14)

$\bigcup_{X v^{\prime} C i \in V} R\left(\left.\frac{N}{P} \right\rvert\, C i\right)=R(P+1)$          (15)

$\sum_{X: X \in|N|} U x<1$           (16)

The attainable step technique for composite (index) coding is based on prefetching decoding. If the cache placement phase is coded in caching, the delivery phase is a well-specified message and demand set.

3.5 Unique Tag Deduplication Integrity Check (UTDIC)

Users can use UTDIC to create unique validation tags to ensure data integrity across CSPs and enable Deduplication. At the same time, to prevent the results of CSP duplicate checking during the file upload process from misleading users, UTDIC is based on the private set intersection, which allows users to determine whether files are duplicates before CSP.

UTDIC requires challenger A to reply with (R) valid note sets to pass the integrity check.

$\begin{aligned} \delta=\sum_{i=1}^\gamma P_{( {Sucess },\, i)}^A= & \left(1-\rho_{a d v}\right)^\gamma +\underbrace{\frac{\gamma \rho_{a d v}\left(1-\rho_{a d v}\right)^{\gamma-1}}{2^{n s}}+o\left(\frac{1}{2^{n s}}\right)}_{\xi}\end{aligned}$       (17)

UTDIC can recover rD ¼ d if there are more than d 2 corrupted errors in a block, our protocol cannot recover the blocks.

$P_{(F a i l, i)}^\sigma \leq {exp} \left(-\frac{\rho_{a d v} D}{3}\left(1-\frac{\rho}{\rho_{a d v}}\right)^z\right)$         (18)

$\left(1-\frac{\rho}{\rho_{n \epsilon g}}\right)^2 \rho_{n \epsilon g}=\frac{3 \ln (2) \tau}{D}$ and $\rho_{n \in g}<\rho$           (19)

Next, choose a threshold (T) for the query time $g$ so our protocol will discover a chunk reduction. A with an overwhelming likelihood if it corrupts more than $\gamma$ neg proportion of the blocks. if g$>=$neg and radv $>$ $\eta n e g$.Next

$\begin{aligned} { Resp } * b^{-1}\, {modm} & =(v \times D) * b^{-1} \,{modm} \\ & =(b e \times D) * b^{-1} \,{modm} \\ & =e \times D \, { modm }\end{aligned}$         (20)

$D=\left(\begin{array}{ccc}d_{11} & \cdots & d_{1 y} \\ \vdots & \ddots & \vdots \\ d_{x 1} & \cdots & d_{x y}\end{array}\right)$           (21)

$\begin{aligned} e \times D \,modm & =\left(\sum_{i=1}^x e_i d_{i 1}, \sum_{i=1}^x e_i d_{i 2}, \ldots, \sum_{i=1}^x e_i d_{i y}\right) \,modm \\ & =\left(\sum_{i=1}^x e_i d_{i 1}, \sum_{i=1}^x e_i d_{i 2}, \ldots, \sum_{i=1}^x e_i d_{i y}\right)\end{aligned}$       (22)

Demonstrate the efficacy of utilizing the UTDIC approach to retrieve the query column containing the notes, enabling many users to generate their verification tags while maintaining support for tag deduplication at the CSP. Additionally, try your hardest to guarantee that the data duplication check is accurate.

3.5.1 Deduplication file

The proposed approach of deduplication functions at the file level. Only text files (.txt) are allowed. The user and his cloud environment are the two stakeholders. Google Drive is the cloud environment in this case. The files to be uploaded are selected by the user using this method. The UTDIC technique is used to produce the file's hash. The hash value of the cloud and uploaded files are compared throughout the upload process. Uploading of the file will be blocked if a match is discovered. The chosen file will be uploaded successfully, even without a support file.

The new file's bin information and the segment numbers that were generated for it are added to the file-to-segment metadata. The segments that are duplicates have the same segment number. The quantity of file-generated unique Tag segments is updated in the bin structure configuration file. In essence, the number of segments beneath each bin is counted in this file. This module uses gzip to compress and decompress files, saving additional storage space.

A list search has already produced the computed hash value. The hash () function generates this list lookup. Specific segments are not preserved if the hash value is present. If not, the section is stored as indicated and then downloaded afterward.