Skip to main content

Data DeDuping!

Data deduplication, often called "intelligent compression" or "single-instance storage", is a process that uses matching logic to eliminate file records that are duplicates (dupes). It is a method of reducing storage needs by eliminating redundant data and replacing it with a pointer to the unique data copy

Data deduplication offers other benefits. Lower storage space requirements will save money on disk expenditures. The more efficient use of disk space also allows for longer disk retention periods, which provides better recovery time objectives (RTO) for a longer time and reduces the need for tape backups. Data deduplication also reduces the data that must be sent across a WAN for remote backups, replication, and disaster recovery.

Deduping is a 3 step process
Step 1: Move the non duplicates (unique tuples) into a temporary table
SELECT * into new_table
FROM old_table
WHERE 1
GROUP BY [column to remove duplicates by];

Step 2: Delete the old table. We no longer need the table with all the duplicate entries, so drop it!
DROP TABLE old_table;

Step 3: Rename the new_table to the name of the old_table
RENAME TABLE new_table TO old_table;

The above set of code works fine if you have to remove duplicate code. What if you just have to mark the duplicate records, but not delete the records

Step 1: Move the non duplicates (unique tuples) into a temporary table
SELECT * into #new_table
FROM old_table
WHERE 1
GROUP BY [column to remove duplicates by];

Step 2: Update the column of the old table to mark it as duplicate
UPDATE old_table
SET old_table.column3 = 1
FROM old_table INNER JOIN #new_table
ON old_table.column2 = #new_table.column2
   
Step 3: Delete the new table
DROP TABLE #new_table

Labels:

Comments

Post a Comment

Popular posts from this blog

Insufficient access rights to perform the operation. (Exception from HRESULT: 0x80072098)

While accessing the active directory (AD) and authorization manager (AZMAN) , If you get “   Insufficient access rights to perform the operation. (Exception from HRESULT: 0x80072098)  “ message check the    account that is being used to get the LDAP query from AD .  ERROR DETAILS Exception Details:  System.Runtime.InteropServices.COMException: Insufficient access rights to perform the operation. (Exception from HRESULT: 0x80072098) Source Error: Line 154:    'Session("FullName") = System.Security.Principal.WindowsIdentity.GetCurrent.Name.ToString() Line 155: Line 156:    If Not User.IsInRole("Role1") Then Line 157:          Response.Redirect("./Login.aspx") Line 158:    End If  Stack Trace : .... SOLVE IT Steps to do check the app pool rights: Click on the website name that you are having problem with in IIS  In the right panel you will se...
Post a Comment
Read more

JavaScript Interview Questions

This is a compilations of all the interview questions related to Javascript that i have encountered.  Q: Difference between window.onload and onDocumentReady? A: The onload event does not fire until every last piece of the page is loaded, this includes css and images, which means there’s a huge delay before any code is executed. That isnt what we want. We just want to wait until the DOM is loaded and is able to be manipulated. onDocumentReady allows the programmer to do that. Q:  What is the difference between == and === ? A: The == checks for value equality, but === checks for both type and value. Few examples: "1" == 1; // value evaluation only, yields true "1" === 1; // value and type evaluation, yields false "1" == true; // "1" as boolean is true, value evaluation only, yields true "1" === false; // value and type evaluation, yields false Q: What does “1″+2+5 evaluate to? What about 5 + 2 +...
1 comment
Read more

Do's and Don't SQL

Do's: Writing comments whenever something is not very obvious, as it won’t impact the performance.  (--) for single line  (/*…*/) to mark a section Use proper indentation Use Upper Case for all SQL keywords. SELECT, UPDATE, INSERT, WHERE, INNER JOIN, AND, OR, LIKE. Use BEGIN... END block for multiple statements in conditional code  Use Declare and Set in beginning of Stored procedure Create objects in same database where its relevant table exists otherwise it will reduce network performance. Use PRIMARY key in WHERE condition of UPDATE or DELETE statements as this will avoid error possibilities. If User table references Employee table than the column name used in reference should be UserID where User is table name and ID primary column of User table and UserID is reference column of Employee table. Use select column name instead of select * Use CTE (Common Table Expression); its scope is limited to the next statement in SQL query, instead of...
Post a Comment
Read more