Windows Server 2016 Data Deduplication Scales and Performs Better

I’ve been leveraging Windows Server Data Deduplication since it became available with great results.

Embedded image permalink

One of the enhanced features in Windows Server 2016 is Data Deduplication and it’s one I welcome very much. The improvements we’re getting mostly have to do with scale and performance. I’m quite pleased that Microsoft listened to our previous feedback on this.


You cannot imagine how much money on backup target storage we have saved by using this. So we’re very happy that Windows Server 2016 Data Deduplication scales and performs better. The fact that we can no get even better scale and performance is music to our ears. The Backup target servers are the first in line for an upgrade, that’s for sure! That’s the reason I mentioned it as a subject to look into in the Hyper-V amigos interview at Ignite!

Scale Improvement of the supported LUN sizes, up to 64TB

Actually I was already pushing this to 50TB Embarrassed smile in some cases for testing but over all I used 6 to 10 TB volumes. But the support for bigger volumes is very welcome. Now, please not that you should NOT go any higher than 64TB (I actually stay below that) otherwise deduplication doesn’t work due to it’s dependency on VSS. Please read my blog

Windows 2012 R2 Data Deduplication Leverages Shadow Copies: “LastOptimizationResultMessage : A volume shadow copy could not be created or was unexpectedly deleted” on this subject.

In Windows 2012 R2 we were limited because data deduplication used a single-threaded job and I/O queue for each volume. That makes it wiser to have 10 target LUNS of 60TB than one huge 60TB LUN. The big issue otherwise is that large volumes could lead to the dedup processing keeping up with the rate of data changes (“churn”).  Now your milage would very depending on the type of data and the delta. More info on this in the blog post:Sizing Volumes for Data Deduplication in Windows Server. It will help you size the volumes but note that in Windows Server 2016 the rules have changed Smile

The dedup optimization processing now runs multiple threads in parallel using multiple I/O queues on a single volume which gives you better performance and doesn’t incur the overhead of having to use more smaller LUNs.

File sizes up to 1TB are good for dedup

Windows Server 2012 R2 Data Deduplication supports the use of file sizes up to 1TB, but they are considered as “not good candidates” for dedup.  So that DPM workaround of backing up to a truckload of virtual machines with 1TB virtual disks that are deduplicated is borderline. You can see one improvement in CPS v2 coming already (also see the next header). 1TB is now fully supported and a good candidate. I’ll be pushing it higher … in my opinion this is were the most work will need to be done for future improvements. It would allow for more scenarios (I have VMs that hold VHDX virtual disks of  2TB or more). Scale it something that helps keep things simple. Simple avoid costs & issue with complexity. That’s always a good thing if possible.

In Windows Server 2012 the algorithms can’t scale as well and performance suffers due to things like scanning for and inserting changes can slow down as the total data set increases. These processes have been redesigned in Windows Server 2016. It now uses new stream map structures and improved partial file optimization. As a result 1TB file sizes have become good candidates.

Virtualized backup is a new usage type

DPM is already leveraging deduplication of virtual machines (CPS drove that I think, see Deduplicating DPM Storage).


In Windows Server 2016 all the dedup configuration settings have been combined into a new usage type called “Backup”. This simplifies the deployment and helps “future proof” your setup as future changes can automatically be applied true this usage type.

Nano Server support

Data deduplication is (or will be) fully supported in Nano Server (new in TPv3). It’s not completely done yet so deduplication support in Nano Server still has a few restrictions:

  • Support has only been validated in non-clustered configurations
  • Deduplication job cancellation must be done manually (using the Stop-DedupJob PowerShell command)

Microsoft welcomes any feedback on the deduplication feature via an email sent to For me the standing order is to break through that 1TB barrier!

My take & Magic Ball

In combination with the right backup product it saves a ton of money. I have leveraged VEEAM and in the past Windows Backup (inbox) with great results. The benefit of these two is that you can backup to physical storage and leverage deduplication. Virtualized backup as a new usage type and makes live easier for the supported “workaround” around the limitations of DPM where normally they only support VDI for  with deduplication.  What I’m really curious about is another possible future usage type: “Virtual Servers” … I guess for that one deduplication support for the OS disk would be very beneficial for “cloud” providers. We’ll see

I’m a Veeam Vanguard 2015

Veeam has announced it’s Veeam Vanguard program last month while I was on vacation. I am honored to have been nominated as 1 of 31 professionals world wide. Veeam states the following, which I consider to be a great compliment:

These individuals have been nominated as Veeam Vanguards for 2015. A Veeam Vanguard represents our brand to the highest level in many of the different technology communities in which we engage. These individuals are chosen for their acumen, engagement and style in their activities on and offline.


Rick Vanover is spearheading this program together with the Veeam Product Strategy Team and the entire company is behind this initiative as you can read here What is the Veeam Vanguard Program?

Veeam now has a program like the VMware vExpert, Cisco Champion and Microsoft MVP programs. I’m honored to be nominated and I’m sure Veeam will execute this well as I have one very consistent experience with both Veeam employees and products: quality and dedication to deliver the best possible solutions for their customers. The fact that I’ve been nominated makes me feel appreciated by people whom I respect for their professionalism and skills. As I’m confortable acting as the tip of the spear implementing technologies at the organizations I support I kind of feel that being a Veeam Vanguard is a great fit Smile

I have shared insights, ideas and feedback with VEEAM before and I’m sure we’ll get plenty of opportunities to do even more of that in the future.

Using VEEAM FastSCP for Microsoft Azure to help protect my blog

My buddies in IT know about some of my mantras. The fact that I like “* in depth”. Backup in depth for example. Which is just my variant on the 3-2-1 rule in backups. Things go wrong and relying on one way to recover is risky. “One is none, two is one” is just one of the mantras I live by in IT. Or at least try to, I’m not perfect.

So besides backups in Azure I also copy the backup files I make for my blog outside of the VM, out of Azure. That means the BackWPup files and the MySQL dumps I create regularly via a scheduled job.

That copy is not made manually but is automated with VEEAM FastSCP for Microsoft Azure. It’s easy, free and it works.  I’ve blogged about it before but that blog might have been lost in the huge onslaught of Microsoft Ignite 2015 announcements.

It’s all quite simple. First of all you need to create a data dump location for the backups we do on our blog server. That’s copied out by but VEEAM FastSCP for Microsoft Azure ensures I have an extra copy do those which doesn’t rely on Azure



Add your VM in Azure to VEEAM FastSCP for Microsoft Azure


It’s easy, specify the information you can find about your VM on the Azure management portal. Optionally you can skip the SSL requirement and certificate verifications. Do note you need to use the correct PowerShell port (end point) for that particular VM in your Azure subscription.


When successful you can browse the file system of your Azure VM.


Create one or more jobs (depending on what & how you’re organizing your backups)


Give the job a descriptive name


Select what folders on the Azure VM you want to backup by simply browsing to it.


Select the target folder on the system where VEEAM FastSCP for Microsoft Azure is running by, again, simply browsing to it.


Set a schedule according to your needs


If you need to run some PowerShell before or after a download here’s the place to do so.


Click finish and hit Start Job to lick it of and test it. Here’s the WordPress Blog backup download job running.


By using VEEAM FastSCP for You can download folders and files to your system at home, to a virtual machine, whether this is on premise or also in the cloud. Perhaps even in AWS (IAAS) if you’re really paranoid. By doing a simple restore of your blog and changing your DNS entry you can even get it up and running if Azure would ever be the target of a major outage causing attack. You could even keep blogging about it Smile.

So do yourself a favor. Check it out!

WorkingHardInIT Blog Maintenance Window & Tools Used

As you might have noticed my blog was down last night for about 1 hour and 45 minutes between 22:20 and 00:10. A bit longer than I wanted but I needed more time do deal with the upgrade of MySQL as part of the routine maintenance I do on my WordPress blog server.

In the environments under my care I take care to take the time to do routine maintenance to avoid falling behind to much in firmware, drivers, patches, etc. This takes some effort but as it helps prevent bigger issues in the long run it’s worth while to do so. I take the same approach with my blog as much as possible. Most of this maintenance goes by without you ever noticing. The windows updates reboots being the exception. WordPress upgrades, plugin upgrades, PHP upgrades, etc. … all go swiftly usually which means I’m pretty well covered there, frequently.

Upgrading MySQL however is always a bit of a time consuming effort and depending on what version you’re upgrading from and to witch one it can actually mean multiple sequential upgrades (5.1 to 5.5.44 to 5.6.25).image

I practiced this upgrade on a copy of the VM in azure to make sure I could handle whatever came up and still I had to deal with some challenges I did not encounter in the test environment. That show that I’m not a full time hard core MySQL guru I guess.

Anyway after getting to MySQL 5.6.25 from 5.5.44 and fixing some issues with TIMESTAMP with implicit DEFAULT value is deprecated (easy fix) and dealing with the error in MySQL Workbench:

An unhandled exception occurred (Error executing ‘SELECT t.PROCESSLIST_ID,
t.INSTRUMENTED,t.PROCESSLIST_INFO,a.ATTR_VALUE FROM performance_schema.threads t 
LEFT OUTER JOIN performance_schema.session_connect_attrs a ON t.processlist_id = a.processlist_id AND (
a.attr_name IS NULL OR a.attr_name = ‘program_name’) WHERE t.TYPE <> ‘BACKGROUND”
Native table ‘performance_schema’.’threads’ has the wrong structure.
SQL Error: 1682). Please refer to the log files for details.

which I fixed by running run mysqld –performance_schema I’m rocking everything up to date once more.


Always have good backups, make exports of your database schema, data and structures in MySQL and have multiple ways out when things go south. In Azure I’m relying on Backup Vault where I protect my virtual machine with schedules backup jobs. I also backup my WordPress with the data via a plug in and export the database via MySQL Workbench.


Those dumps are copied out of the VM to where ever I want (Azure, One Drive, home PC, a VM running in AWS …) to make sure I have multiple options to recover.


VEEAM FastSCP for Microsoft Azure comes in very handy for this by the way. You might want to check it out if you’re in need of an automated and secure way to get data out of a VM running in Microsoft Azure!