Warning on Windows Server 2016 Deduplication Corruption

UPDATE 2 – 2017/02/06

DO NOT INSTALL KB3216755 if you don’t need it.  Huge memory leak reported to associated with this. If you need it I’d consider all my options.

UPDATE – GET KB3216755

As you can read it the comments, Microsoft reached out and confirms the issues are fixed as part of KB3216755 => https://support.microsoft.com/en-us/help/4011347/windows-10-update-kb3216755 . I commend them for responding so quickly and getting it sorted. Do not that at the time of writing this (late on January 30th CET) the Windows Sever 2016 update isn’t in the Windows Catalog yet, only the Windows 10 ones. But Microsoft confirms you should install the update  on their blog

Windows Server 2016 Data Deduplication users: please install KB3216755!

The issue

Good morning. A quick blog post to give a heads up to my readers who might not be subscribed to Anton Gostev (Veeam) his “The Word Form Gostev”. It concerns a warning on Windows Server 2016 Deduplication corruption.

Warning on Windows Server 2016 Deduplication Corruption

There are multiple reports of data corruption with Windows Server 2016 deduplication. One is related to file sizes over 2TB. The other with the loss of checksum values. Microsoft is aware these issues and a fix is coming for these issues.

I quote Gostev

I’ve already received the official confirmation from Microsoft that this is the know issue (ID 10165851) which is scheduled to be addressed in the next Windows Server 2016 servicing update. There are actually two separate issues, both leading to file corruption when using deduplication on very large files. One issue occurs when files grow to 2.2TB or larger, and another one causes loss of checksums for files with “smaller sizes” – this is the actual wording of the official note, so I have no idea how small

What to do?

If you use Windows Server 2016 deduplication for backups, create new full backups regularly. Also make sure you do backup integrity testing and restore tests. Follow up on the update when it arrives.

If you use the for production data make sure you have frequent and validated backups! Design & operate under the mantra of “Trust but verify”.

Also, we’ve heard reports and noticed that Windows Server 2016 Deduplication resource configuration isn’t always respected. I.e. it can take all resources away despite limitations being set. We hope a fix for this is also under way.

Being a Microsoft MVP and “The Big Ask”

Being a Microsoft MVP and “The Big Ask’’

I’m proud to be recognized once more as a Microsoft Most Valuable Professional (MVP) in 2017 for the Cloud and Datacenter Management expertise. It’s fantastic to be part of that community and I enjoy the feedback, discussion between my fellow MVPs and Microsoft. So what’s all this about Being a Microsoft MVP and “The Big Ask’’ you might wonder?

Being a Microsoft MVP and “The Big Ask’'

Many readers have reached out for help lately.  I hope that, at least, I have guided you towards the information and knowledge to find a solution. Please understand that I cannot help out everybody out there individually. With a job to do, community efforts, a life and only 24 hours in a day, there is a limit.. So yes, it’s a big ask, but I also enjoy the learning and the interaction. So don’t get me wrong on this. I love doing it.

I hope I can keep learning, growing and sharing in this industry for many years to come. I’m looking forward to some “out of the box” solutions based on Microsoft technologies in 2017-2018. The aim is  to offer the best possible solutions fast, good & affordable. These will be shared via this blog and the community for the benefit of you all!

Thank you reading!

Import of RD Gateway configuration file with policies referencing local resources wipes all policies clean!

Introduction

When you have Windows Server 2016 RD Gateway server and you expect to be able to import a configuration XML file you’ll might find yourself in a pickle when you are also using local resources. Because the import of RD Gateway configuration file with policies referencing local resources wipes all policies clean! With local resources I mean local user accounts and groups. These are leveraged more than I imagined at first.

When does it happen?

In the past I have blogged about migrating RD Gateway servers that contain policies referencing local resources here: Fixing Event ID 2002 “The policy and configuration settings could not be imported to the RD Gateway server “%1” because they are associated with local computer groups on another RD Gateway server”.

We used to be able to use the trick of making sure the local resources exist on the new server (either by recreating them there via the server migration wizard or manually) and changing the server name in the exported configuration XML file  to successfully import the configuration. That no longer works. You get an error.Import of RD Gateway configuration file with policies referencing local resources wipes all policies clean!

As far as migrations go from older versions, they work fins as long as you don’t have policies with local resources. Otherwise you’d better do an in place upgrade or recreate the resources & policies on the new servers. The method described in my blog is not working any more. That’s to bad. But it gets worse.

Import of RD Gateway configuration file with policies referencing local resources wipes all policies clean!

As said,it doesn’t end there. The issue is there even when you try to import the configuration on to the same server you exported it from.That’s really bad as it a quick way to protect against any mistakes you might make, and allows to get back to the original configuration.

What’s even worse, when the import fails it wipes ALL the policies in the RD Gateway Server => dangerous! So yes, the import of RD Gateway configuration file with policies referencing local resources wipes all policies clean!

Precautions

Only a backup or a checkpoint can save your then (or recreate the all manually)! Again this is only when the exported configuration file references local resources! The fasted way to clean out an RD Gateway configuration on Windows Server 2016 is actually importing a configuration export which contains a policy referring to local resource. Ouch! I’m not aware of a fix up to this date.

For now you only protection is a checkpoint or a backup. Depending on where and how you source your virtual machines you might not have access to a checkpoint.

You have been warned, be careful.

How tech debt happens

Introduction

While I plan our actions to improve or add new functionality I always refer back to my “map” on where we are, where we need to go. It helps me see the terrain, the problems, challenges, obstacles, opportunities, possibilities in the grand scheme of what I call in big words my strategy & doctrine. Sometimes it’s big in time, complexity or budget and sometimes it’s a small target of opportunity that has both immediate & long term benefits. On of those targets of opportunity is to make sure that every upgrade/migration of Hyper-V workloads lands on a generation 2 virtual machines running on Windows Server  2016 Hyper-V. Seemingly unimportant, yet … when you understand how tech debt happens you’ll see it does make a difference when done with a plan.

How tech debt happens

I have witnessed tens of millions wasted over the years by organizations that fell into every tech debt trap there is. Never forget that it’s not just bad because of failed goals and added costs but also due to being stuck and missing out on opportunities. It’s astonishing to see how bad it can become. Even at organizations that like to act and profile themselves at being modern, agile and in full digital transformation mode. Look, buzz words & glossy brochure like pictures on an “infomercial” website don’t make you the real deal. Yet while I spend many words on illustrating how tech debt happens, it’s easy to sum up.

How tech debt happens

Technical debt exists because people don’t realize what it is, how it materializes and how badly it effects the organization. And for the record, not every legacy is debt just like not all redundancy is bad.

Avoiding Tech Debt

So how do you avoid getting tech debt? Well by understanding you get  into it and doing something about it! The lack of process or understanding (even worse, I rather have no process but with understanding) to the real nature, causes and effects of technical debt. This leads to decisions void of any consideration of the implications. There’s a ton of individual reasons all across the board that will create tech debt.

Business pressure

One of them is “business” pressure, the need to be seen as in charge and get things done. This “can do” attitude is a killer of improvement and competence. It leads to an environment where what sucks can’t be pin pointed because people focus on showing how good they are instead of finding and fixing what’s broken. They have to or they cant constantly answer “Can do” to every request. In short a 100% can do culture will make you fail.

The results are many. let’s look at some of them. Having ITIL and a change board that is so ridiculously heavy in overhead because it’s the blind leading the blind and serves only to have the checkbox ticked. After some time the process is simplified and automated (send in docs that no one reads but it does lead to an approval e-mail) and over a longer period of time it’s ignored. This leads to a constant barrage of last minute specification changes without a clue as to their impact.

Scope creep & scope dumping

Scope creep: non-managed changes or perhaps better worded and more realistically, changes by people with no technical clue but a lot of pressure and a desire to please other people of server other needs. There’s also the opposite, scope dumping: Ill considered scope reduction by people who need to make deadlines or avoid complexity they don’t want to deal with, often under business pressure.

Integration is still important

Lack of integration is another. Yes we all know we need to reduce plumbing in IT but the reality is that good, necessary plumbing avoids a truck load of problems. If you want to avoid spending too much time keeping the lights on you’ll need to do good plumbing to avoid flooding.

Handing over the key to the kingdom

Relying too much on consulting, external advisors. Always ask your self who they work for and why. Follow the money. The money will lead you to the one paying and than you’ll find out their politics, plans and ambitions. If these are not yours, alarm bells should
Disdain for learning, testing, hands on work. Knowledge comes from understanding and that requires doing. Doing with understanding grows skills, insight an knowledge to be effective when needed. If I want to destroy a company I send in consultants and I make sure they hire the wrong employees. It’s a long term play against my enemies where they fall for the perceived short term benefits for them. When I see a CEO that makes tens of millions and year after year you see the company do down the drain you just know they are not worth the money and their presence makes no difference what so ever. You could have gotten those results way cheaper.

The right stuff

Lack of any real functional documentation and insights is also an issue. You need to have the elements to find out how things are done quickly. That doesn’t mean 100% perfect descriptions to the smalls detail. That’s just windows dressing. Quality trumps quantity. This goes for anything, processes, documentation and people. Beckwith was right when he said “I’d rather go down the river with 7 man than with a 100 shitheads”

Incompetence floats to the top

The lack of a good command & control structure due to not having competent management is also problematic along with a lack of skills, talent & knowledge. Yup, too many Peter principle people looking out for number one will catalyze all bad things. Put the right people in the right place. Hard to do when you think you can but actually can’t … or when power play and politics are way more important than effective results. This leads to lack of collaboration in an environment where everybody survives in isolation instead of thriving  by working together. The results are a lack of maintenance, updates, migrations, rebuilds to improve current and support future needs.

Own the problems

Which brings us to the lack of ownership. Here you’ll often see that a PO or PM only is responsible on paper and doesn’t have a clue about the service. They don’t care about it, let alone about the effects on some other business unit. The deficiency in business and technological leadership, leads to avoiding responsibility. Problems are just thrown over the wall and any issues that appear are just assigned to the “incompetence” of the staff.

Conclusion

Tech Debt has many reasons and that’s why it’s hard to avoid and fix. But it can be done. But you cannot buy your way out or outsource solving the problem.