Skip to main content

Data DeDuping!

Data deduplication, often called "intelligent compression" or "single-instance storage", is a process that uses matching logic to eliminate file records that are duplicates (dupes). It is a method of reducing storage needs by eliminating redundant data and replacing it with a pointer to the unique data copy

Data deduplication offers other benefits. Lower storage space requirements will save money on disk expenditures. The more efficient use of disk space also allows for longer disk retention periods, which provides better recovery time objectives (RTO) for a longer time and reduces the need for tape backups. Data deduplication also reduces the data that must be sent across a WAN for remote backups, replication, and disaster recovery.

Deduping is a 3 step process
Step 1: Move the non duplicates (unique tuples) into a temporary table
SELECT * into new_table
FROM old_table
WHERE 1
GROUP BY [column to remove duplicates by];

Step 2: Delete the old table. We no longer need the table with all the duplicate entries, so drop it!
DROP TABLE old_table;

Step 3: Rename the new_table to the name of the old_table
RENAME TABLE new_table TO old_table;

The above set of code works fine if you have to remove duplicate code. What if you just have to mark the duplicate records, but not delete the records

Step 1: Move the non duplicates (unique tuples) into a temporary table
SELECT * into #new_table
FROM old_table
WHERE 1
GROUP BY [column to remove duplicates by];

Step 2: Update the column of the old table to mark it as duplicate
UPDATE old_table
SET old_table.column3 = 1
FROM old_table INNER JOIN #new_table
ON old_table.column2 = #new_table.column2
   
Step 3: Delete the new table
DROP TABLE #new_table

Labels:

Comments

Post a Comment

Popular posts from this blog

Insufficient access rights to perform the operation. (Exception from HRESULT: 0x80072098)

While accessing the active directory (AD) and authorization manager (AZMAN) , If you get “   Insufficient access rights to perform the operation. (Exception from HRESULT: 0x80072098)  “ message check the    account that is being used to get the LDAP query from AD .  ERROR DETAILS Exception Details:  System.Runtime.InteropServices.COMException: Insufficient access rights to perform the operation. (Exception from HRESULT: 0x80072098) Source Error: Line 154:    'Session("FullName") = System.Security.Principal.WindowsIdentity.GetCurrent.Name.ToString() Line 155: Line 156:    If Not User.IsInRole("Role1") Then Line 157:          Response.Redirect("./Login.aspx") Line 158:    End If  Stack Trace : .... SOLVE IT Steps to do check the app pool rights: Click on the website name that you are having problem with in IIS  In the right panel you will se...
Post a Comment
Read more

Do's and Don't SQL

Do's: Writing comments whenever something is not very obvious, as it won’t impact the performance.  (--) for single line  (/*…*/) to mark a section Use proper indentation Use Upper Case for all SQL keywords. SELECT, UPDATE, INSERT, WHERE, INNER JOIN, AND, OR, LIKE. Use BEGIN... END block for multiple statements in conditional code  Use Declare and Set in beginning of Stored procedure Create objects in same database where its relevant table exists otherwise it will reduce network performance. Use PRIMARY key in WHERE condition of UPDATE or DELETE statements as this will avoid error possibilities. If User table references Employee table than the column name used in reference should be UserID where User is table name and ID primary column of User table and UserID is reference column of Employee table. Use select column name instead of select * Use CTE (Common Table Expression); its scope is limited to the next statement in SQL query, instead of...
Post a Comment
Read more

SQL Server 2008 - Inline Variable Assignment

Microsoft SQL Server 2008 brings in a new feature of 'Declaring and Assigning' a variable all in a single line. Earlier, while using SQL 2005 and earlier versions, we had to declare a variable before assigning it. This feature is explained in detail below. In SQL 2005 and earlier versions we had to write: Declare @age int Declare @name nvarchar(25) Declare @date date Set @age = 25 Set @name = "Garry" Set @date = GETDATE() With SQL 2008 we can combine both these statements to write Declare @age int = 25 Declare @name nvarchar(25) Declare @date @date = GETDATE() Advantage : This helps us to write less code, yet get the same functionality. Even though this feature has been available it is not being used in many projects because of backward compatibility issue. If the same code is being used on both 2005 and 2008 server we would prefer writing in a format compatible to both. Do share with us :  Would you like to use this new method? If not, why?
Post a Comment
Read more