Testing Compellent Replay Manager 7.8

Testing Compellent Replay Manager 7.8

So today I found the Replay Manager 7.8 bits to download.image

As is was awaiting this eagerly (see Off Host Backup Jobs with Veeam and Replay Manager 7.8). So naturally, I set of my day by testing Compellent Replay Manager 7.8. I deployed in on a 2 node DELL PowerEdge Cluster with FC access to a secondary DELL Compellent running SC 6.7.30 (you need to be on 6.7).


The first thing I noticed is the new icon.


That test cluster is running Windows Server 2016 Datacenter edition and is fully patched. The functionality is much the same as it was. There is one difference and that if you launch the back upset manually of a local volume for a CSV and that CSV is not owned y the Node in which you launch it the backup is blocked.


This did not use to be the case. With scheduled backup sets this is not an issue, it detects the owner of the CSV and uses that.


Just remember when running a backup manually you nee to launch it from the CSV owner node in Replay Manager and all is fine.


Other than that testing has been smooth and naturally we’ll be leveraging RM 7.8 with transportable snapshots with Veeam B&R 9.5 as well.

Things to note

Replay Manager 7.8 is not backward compatible with 7.7.1 or lower so you have to have the same version on your Replay Manager management server as on the hosts you want to protect. You also have to be running SC 6.7 or higher.

Wish list

I’d love to see Replay manager become more intelligent and handle VM Mobility better. The fact that VMs are tied to the node on which the backup set is create is really not compatible with the mobility of VMs (maintenance, dynamic optimization, CSV balancing, …). A little time and effort here would go a long way.

Second. Live Volumes has gotten a lot better but we still need to choose between Replay Manager  snapshots & Live Volumes. In an ideal world that would not be the case and Replay manager would have the ability to handle this dynamically. A big ask perhaps, but it would be swell.

I just keep giving the feedback as I’m convinced this is a great SAN for Hyper-V environments and they could beat anyone by make a few more improvements.

Full or Thick Provisioned Volume on Compellent


There are pundits out there that claim that you cannot create a fully provisioned LUN on a Compellent SAN.  Now that what I call unsubstantiated rumors, better know as bull shit.

Sure the magic sauce of many modern storage array lies in thin provisioning. Let there be no mistake about that. But there are scenarios where you might want to leverage a fully provisioned volume. This is also know a s thick provisioned LUN. You can read about one such a scenario where they make perfect sense in this blog post Mind the UNMAP Impact On Performance In Certain Scenarios

Create a  Full or Thick Provisioned Volume on Compellent

First of all you create brand new volume in the Storage Center System Explorer. That’s a standard as it gets.

You then map this volume to a server

At that moment, before you even mount that volume on your server let alone do anything else such a bringing it on line or formatting it you’ll “Preallocate Storage” for that volume in Storage Center.


You’ll get a warning as this is not a default action and you should only do so when the conditions of the IO warrant this.


When you continue you’ll get some feedback. This can take quite some time depending on the size of the volume.


When it’s done peek at the statistics of that full or this provisioned volume on the Compellent.This is what it looks like when you look at the statistics for that volume after is was done. So before we even formatted the volume on a server and wrote data to it. It’s using all the space on the SAN for the start.


Due to data protection it’s even more. It’s clear form the image above that a 500GB disk in RAID 10 fully provisioned is using 1TB of space as its all still in RAID 10 (no tiering down has occurred yet). Raid 10 has an overhead factor 2. The volume is for a large part in Tier 2 because my Tier 1 is full, so writing spilled over into Tier 2.

Now compare this to a thinly provisioned volume that we just created and again we haven’t even touched it in any other way.


Yup, until we actually write data to the volume it’s highly space efficient, there is absolutely no spaces use and we’ll see only a little when we mount, initialize the disk in Windows, create a simple volume and format it.image

This is completely in Tier 2 and my tier 1 is full. I accept donations of SANs and SSD’s for my lab it this bothers you Winking smile. When we write data to it you’ll see this rise and over time you’ll see it tier down and up as well.

DELL Compellent Storage Center 7.1 Certified for Windows Server 2016

When it comes to selecting storage, especially when it comes to a “traditional” SAN, you all know that price performance wise I’ve been using the DELL Compellent series with great success for many years now. It’s a very capable solution that also has some other benefits when it comes to Windows Server and Hyper-V. It has one of the better hardware VSS providers, way better than average support for ODX  and UNMAP etc. but it’s also very good at delivering fast support for new versions of Windows Server. This has allowed us to move from Windows Server 2008 R2 to 2012 and from there to Windows Server 2012 R2 very fast.

In that regards I’m very happy to see that Storage Center 7.1 is already in the catalog as certified for Windows Server 2016.


Customers that have up to date hardware and want to move fast to benefit from and leverage the new and improved capabilities in Windows Server 2016 Hyper-V, Clustering, Networking, …are ready to do so. Nice Smile.

Dell Compellent SCOS 6.7 ODX Bug Heads Up

UPDATE 3: Bad and disappointing news. After update 2 we’ve seen DELL change the CSTA (CoPilot Services Technical Alert)  on the customer website to “’will be fixed” in a future version. No according to the latest comment on this blog post that would be In Q1 2017. Basically this is unacceptable and it’s a shame to see a SAN that was one of the best when in comes to Hyper-V Support in Windows Server 2012 / 2012 R2 decline in this way. If  7.x is required for Windows Server 2016 Support this is pretty bad as it means early adopters are stuck or we’ll have to find an recommend another solution. This is not a good day for Dell storage.

UPDATE 2: As you can read in the comments below people are still having issues. Do NOT just update without checking everything.

UPDATE: This issue has been resolved in Storage Center 6.7.10 and 7.Ximage

If you have 6.7.x below 6.7.10 it’s time to think about moving to 6.7.10!

No vendor is exempt form errors, issues, mistakes and trouble with advances features and unfortunately Dell Compellent has issues with Windows Server 2012 (R2) ODX in the current release of SCOS 6.7. Bar a performance issue in a 6.4 version they had very good track record in regards to ODX, UNMAP, … so far. But no matter how good your are, bad things can happen.


I’ve had to people who were bitten by it contact me. The issue is described below.

In SCOS 6.7 an issue has been determined when the ODX driver in Windows Server 2012 requests an Extended Copy between a source volume which is unknown to the Storage Center and a volume which is presented from the Storage Center. When this occurs the Storage Center does not respond with the correct ODX failure code. This results in the Windows Server 2012 not correctly recognizing that the source volume is unknown to the Storage Center. Without the failure code Windows will continually retry the same request which will fail. Due to the large number of failed requests, MPIO will mark the path as down. Performing ODX operations between Storage Center volumes will work and is not exposed to this issue.

You might think that this is not a problem as you might only use Compellent storage but think again. Local disks on the hosts where data is stored temporarily and external storage you use to transport data in and out of your datacenter, or copy backups to are all use cases we can encounter.  When ODX is enabled, it is by default on Windows 2012(R2), the file system will try to use it and when that fails use normal (non ODX) operations. All of this is transparent to the users. Now MPIO will mark the Compellent path as down. Ouch. I will not risk that. Any IO between an non Compellent LUN and a Compellent LUN might cause this to happen.

The only workaround for now is to disable ODX on all your hosts. To me that’s unacceptable and I will not be upgrading to 6.7 for now. We rely on ODX to gain performance benefits at both the physical and virtual layer. We even have our SMB 3 capable clients in the branch offices leverage ODX to avoid costly data copies to our clustered Transparent Failover File Servers.

When a version arrives that fix the issue I’Il testing even more elaborate than before. We’ve come to pay attention to performance issues or data corruption with many vendors, models and releases but this MPIO issue is a new one for me.