On route to Bellevue/Redmond

On route to Bellevue/Redmond

I’m writing this form the lounge in LHR before I make my way to the gate to board BA049 that will take me to SEATAC. Yes, it is that time of year again, I am on route to Bellevue/Redmond. Many other MVPs and I are heading to Bellevue/Redmond for the Microsoft MVP Global Summit 2019

On route to Bellevue/Redmond – 747

Good times ahead

There are indeed good times. The accumulated skills, experiences and knowledge of so many intelligent people working at the Microsoft Redmond campus get an influx or real-world experiences from all over the globe. Meanwhile, the rest of the globe experiences a small brain drain in the IT sector. I actually run into quite a few of them as we all make our way over via different routes crossing paths.

The summit is what you make of it. That’s why I arrive on time and don’t leave early. I save up my pennies to attend and I will make the most of it. It is one the highlights every year in my technology existence. I’m learning continously in a world that spans the cloud & datacenter and combines on-premises, private, hybrid and public cloud to support both proven and emerging technologies. This is the age of combing skills that cover a very broad spectrum of technologies and require a deep understanding and common sense to deliver future proof designs and solutions. They have to keep the business running while enabling the future and avoiding dead ends. Quite a challenge, but I feel privileged to have the opportunity to work and learn with so many smart people in Redmond that have put enough trust in me to award me my MVP title.

The summit is an opportunity to learn, network and socialize with the MVP community members we often only see at conferences and events. The digital age enables us to keep in touch, collaborate and share year round. But, the summit is something special. Since my very 1st summit (an extra special experience) many moons ago I have not missed one. Microsoft has always made me feel very welcome, appreciated and respected.

Seattle Skyline

I also make time to meet up with friends who live around Puget Sound. Some natives, some moved there for work, love or both. That’s always a fun and good use of time. We’ll be raising our glasses and making a toast to absent friends. Those who could not make it, they are genuinely missed!

I’m happy to be able to attend the Global Microsoft MVP Summit

I am truly happy to be hopping over the big pond once more on route to Bellevue/Redmond for the Microsoft MVP Global Summit 2019. We get to immerse ourselves in technology, give feedback and learn about future plans. It makes me grow as a person, it helps me recharge my battery. These are the good days. The days that compensate for those times no one wants to know about. When the chips are down, the budgets tight, the services down – mostly not by your own doing – and you give it all you have to make it work like clockwork again. Or in 99% of times when things run along smoothly and no one knows or notices. Proactive work and interventions are not seen or recognized.

It takes a lot of effort and dedication to be really good at what one does. And you can’t administer your way of out problems. High value, effective and efficient high availability, high performance that is affordable don’t allow for wishful engineering. Something that many managers and ITIL aficionados forget or like to pretend that they don’t exist. Those times when those hard-won skills, your knowledge, and experience are brought in to make the world right again. The times when owning the full stack also puts the full responsibility on your shoulders. You are taken for granted when it all works, it’s forgotten when you fixed it when the universe went “boink” and you are reminded of it when the downtime needs to be explained. Those lonely, dark and cold moments in the glorious life of technical experts no one mentions. But the coming week, I spend in the halls of technical Walhalla where the brave and best IT professionals sit side by side. This investment in my professional education is a gift to myself.

It takes a village (community) effort to raise kids and enable lifelong learning, But you also have to get out of that village and into the big world (another community) to grow, learn and share. My fellow MVPs, I am on my way!

Force Mellanox ConnectX-4 Lx 25Gbps to 10Gbps speed

Introduction

As you might remember I wrote a blog post about SFP+ and SFP28 compatibility. In this i discuss future proofing your network investments and not having to upgrade everything all at once. One example is that buying 25Gbps NICs when your main network infrastructure is still on 10Gbps is not an issue. 25Gbps normally handles 10Gbps well so you don’t have do replace all parts in the fabric at the same time but you can start with either the network fabric or the server NICs. It’s a way of future proofing the investments you make today.

When installing Mellanox ConnectX-4 Lx 25Gbps NICs in a bunch of servers we hit an issue when connected them to the DELLEMC N4000 10Gbps switches. The intent is to replace these with 25/50/100Gbps in the future.

The links did not come up.

The links did not come up. The switch ports are normally forced 10 Gbps in our setups so we check that. The speed was indeed set fix to 10Gbps. When changing that to auto-negotiate the link would come up at 1Gbps.

Naturally you check everything from cabling to used transceivers (BCM84754 on the switches) but that all checked out. We also check the firmware on the switches to determine if they were up to date and perhaps a new version fixed a known issue related to this. But no hardware wise everything was up to date on the switches and on the NICs.

Note that these links worked fine when used with 10 Gbps cards like the ConnectX-3 Pro. The DELL branded transceivers on the switches were BCM84754 (Broadcom)

The fix: Force Mellanox ConnectX-4 Lx 25Gbps to 10Gbps speed

I do not need to tell you that when you want 10Gbps getting 1Gbps doesn’t fly well. The fix was easy enough. We put the switch ports back to 10Gbps fixed speed. Auto-negotiate doesn’t deliver. No worries we fix the ports anyway. We then used mlc5cmd.exe Mellanox tool to change the NIC ports from auto-negotiate to fixed.

On hosts with Mellanox Connect-X4 NICs you open an elevated command prompt.

Navigate to C:\Program Files\Mellanox\MLNX_WinOF2\Management Tools. Run the below command to check the current link speed.

mlx5cmd.exe -LinkSpeed -Name “MyNicName ” -Query

Note 10 and 25 Gbps are supported, so it’s autonegotiate.

We force the link speed to 10Gbps:

mlx5cmd.exe -LinkSpeed -Name “MyNicName ” -Set 10

Link speed is forced to 10Gbps

The link comes up at 10Gbps

Likewise you can force the link to 25Gbps. If you want to change it back to the default you can force the link speed to auto-negotiate again.

mlx5cmd.exe -LinkSpeed -Name “MyNicName” -Set 0

See https://community.mellanox.com/s/article/mlx5cmd-exe for more information on this tool.

Do note that the switch port also needs to be set to 10Gbps fixed. As you can see below the command will notify you when those are still on auto.

The change was done but still no uplink when the switch port isn’t fixed to 10Gbps.

Conclusion

So my statement hold true the path to 25/50/100Gbps is one you can do step by step with future proofing. You might run into some issues but these are normally fixable. I have shared with you how to fix failing or wrong speed negotiations on 25 Gbps RDMA NICs (Mellanox ConnectX-4 Lx) when connecting to 10Gbps ports. I’m pretty sure the same holds true for other models. I have also had cards where things work out of the box but don’t give up when you hit an issue. I hope this helps some of you out there.

Invest in attending VeeamON 2019

Protecting your data is paramount

It is often stated about storage engineers that they have only one rule and that is to never lose data. This is true and it pretty much determines all of their actions. But in reality, this is a top priority for all of us. It has always been like this, but in a full stack world, it is truer than ever. The prime directive of anyone working in ICT, in any role, from helpdesk to CIO, from IT Pro to developer, is to keep services up and running. You have to do this in order to be able to deliver value to the customers. That is how you make a living.

And guess what? In order for services to be up and running, you also need to have any data required protected, available and usable. It’s no good to have developer corrupt the data logically or physically, have a CISO force encryption but lose the keys to the kingdom, or have ransomware render your backups useless. The list of bad things that can and do happen is long and sobering.

It’s no good to have data available you cannot use or read. This is true for all data no matter its nature or where it resides. While funny to a point to the people who are not involved or affected, stating that technically the data is not lost as you know where it is but you just can never access it again is no good at all and far from a joke.

The reality is that bad things do happen no matter how well you try to prevent it. So, you should never bank on solitary solutions. Always give yourself multiple options. From the good old 3-2-1 rule and to having air-gap studies; to not banking on one type of storage, technology, location, process or person, it is all part of adequate to excellent protection. If the business truly and honestly states its data is the gold that makes them thrive, they must act to protect it accordingly.

Are you going then?

No, unfortunately, I’m am otherwise engaged on the days of the conference. Didier, how can you say that to invest in attending VeeamON 2019 when you are not even going yourself? 1st of all, I’d go if I did not have prior engagements – I have done so in the past. I learned a lot and I’m still in touch with customers and Veeam employees I met there. Secondly, an investment does not always involve my physical person being on the scene. It is about getting the right knowledge and insights to the right people in the right places. That means sending colleagues, employees, customers to conferences and training and not always yourself. You develop the skills, knowledge, and capabilities of your organization. Data availability is a primary concern and a goal that I can only achieve when it does not depend solely on me. That too is investing. Helping others learn, grow and succeed help everyone out. Needless to say, I wish I was able to go.

Attend VeeamON 2019

Given the important task of keeping data available and the responsibility to protect and having the ability to recover it when things go wrong, you have a business case to attend VeeamON 2019. In today’s diverse, complex, and fragmented IT ecosystem that data is spread across more places, in different layers and in more elaborate solutions than ever before. This makes data protection and availability a challenge.

When business happens at the speed of light you also have to make data available at that pace when required in different locations for various use cases. Data mobility is a reality and this also has to happen in a secure and workable manner.

We can use all the help we can get to make this happen. The good news is you do not have to go it alone. Attend VeeamON 2019 to give yourself a head start. Invest in your business and your staff. Help them succeed. Register!

You can tap into the collective brain of all the attendees. Talk to knowledgeable and experienced practitioners. Learn from your peers and Veeam experts. Gain insights from others on how to approach the challenges you face. Have that discussion to gain a better understanding of what you can do and how. Get your compass aligned so you can confirm you are right on track to deliver what is asked for you. After all, it is not a small or benign job, to have to ensure the safety of your organizations 21st-century gold, its data. Discover what Veeam is doing in R&D to help you meet future requirements where ever your date lives.

Don’t forget that others can learn from you as well. Consider responding to the request for proposals. Share what you know and have learned. Contribute and learn whilst teaching. Invest in attending VeeamON 2019.

Windows Server 2019 SMB Direct Best Practices

Windows Server 2019 SMB Direct Best Practices

The Hyper-V amigos, @Hypervserver and working@workinghardinithardinit recently did a webinar together about Windows Server 2019 SMB Direct Best Practices. We also discuss some trends and put some things into perspective. Rachfahl IT Solutions does more of these cool webinars for you to check out (see the info at the end of the video). You can watch the webinar below on Vimeo. It’s quite an honor to be invited to talk on this subject as Carsten is one the worlds most experienced S2D practitioners.

https://vimeo.com/319818020
Windows Server 2019 SMB Direct Best Pratices Webinar

Need to know more?

I hope this get’s you started and updated. Need to know more? Want more details, advice and a deeper and more elaborate discussion. I will be talking about this on various occasions this year. One of them is the Cloud & Datacenter Conference Germany 2019 in Hanau. Register to secure your spot. It is a great conference with a lot of hands on, real life knowledge being shard. I will be around for the Hyper-V pre day and during the entire conference. This means you can find me to talk shop. Be warned, I can go one about the subject or a while