Force Mellanox ConnectX-4 Lx 25Gbps to 10Gbps speed

Introduction

As you might remember I wrote a blog post about SFP+ and SFP28 compatibility. In this i discuss future proofing your network investments and not having to upgrade everything all at once. One example is that buying 25Gbps NICs when your main network infrastructure is still on 10Gbps is not an issue. 25Gbps normally handles 10Gbps well so you don’t have do replace all parts in the fabric at the same time but you can start with either the network fabric or the server NICs. It’s a way of future proofing the investments you make today.

When installing Mellanox ConnectX-4 Lx 25Gbps NICs in a bunch of servers we hit an issue when connected them to the DELLEMC N4000 10Gbps switches. The intent is to replace these with 25/50/100Gbps in the future.

The links did not come up.

The links did not come up. The switch ports are normally forced 10 Gbps in our setups so we check that. The speed was indeed set fix to 10Gbps. When changing that to auto-negotiate the link would come up at 1Gbps.

Naturally you check everything from cabling to used transceivers (BCM84754 on the switches) but that all checked out. We also check the firmware on the switches to determine if they were up to date and perhaps a new version fixed a known issue related to this. But no hardware wise everything was up to date on the switches and on the NICs.

Note that these links worked fine when used with 10 Gbps cards like the ConnectX-3 Pro. The DELL branded transceivers on the switches were BCM84754 (Broadcom)

The fix: Force Mellanox ConnectX-4 Lx 25Gbps to 10Gbps speed

I do not need to tell you that when you want 10Gbps getting 1Gbps doesn’t fly well. The fix was easy enough. We put the switch ports back to 10Gbps fixed speed. Auto-negotiate doesn’t deliver. No worries we fix the ports anyway. We then used mlc5cmd.exe Mellanox tool to change the NIC ports from auto-negotiate to fixed.

On hosts with Mellanox Connect-X4 NICs you open an elevated command prompt.

Navigate to C:\Program Files\Mellanox\MLNX_WinOF2\Management Tools. Run the below command to check the current link speed.

mlx5cmd.exe -LinkSpeed -Name “MyNicName ” -Query

Note 10 and 25 Gbps are supported, so it’s autonegotiate.

We force the link speed to 10Gbps:

mlx5cmd.exe -LinkSpeed -Name “MyNicName ” -Set 10

Link speed is forced to 10Gbps

The link comes up at 10Gbps

Likewise you can force the link to 25Gbps. If you want to change it back to the default you can force the link speed to auto-negotiate again.

mlx5cmd.exe -LinkSpeed -Name “MyNicName” -Set 0

See https://community.mellanox.com/s/article/mlx5cmd-exe for more information on this tool.

Do note that the switch port also needs to be set to 10Gbps fixed. As you can see below the command will notify you when those are still on auto.

The change was done but still no uplink when the switch port isn’t fixed to 10Gbps.

Conclusion

So my statement hold true the path to 25/50/100Gbps is one you can do step by step with future proofing. You might run into some issues but these are normally fixable. I have shared with you how to fix failing or wrong speed negotiations on 25 Gbps RDMA NICs (Mellanox ConnectX-4 Lx) when connecting to 10Gbps ports. I’m pretty sure the same holds true for other models. I have also had cards where things work out of the box but don’t give up when you hit an issue. I hope this helps some of you out there.

Invest in attending VeeamON 2019

Protecting your data is paramount

It is often stated about storage engineers that they have only one rule and that is to never lose data. This is true and it pretty much determines all of their actions. But in reality, this is a top priority for all of us. It has always been like this, but in a full stack world, it is truer than ever. The prime directive of anyone working in ICT, in any role, from helpdesk to CIO, from IT Pro to developer, is to keep services up and running. You have to do this in order to be able to deliver value to the customers. That is how you make a living.

And guess what? In order for services to be up and running, you also need to have any data required protected, available and usable. It’s no good to have developer corrupt the data logically or physically, have a CISO force encryption but lose the keys to the kingdom, or have ransomware render your backups useless. The list of bad things that can and do happen is long and sobering.

It’s no good to have data available you cannot use or read. This is true for all data no matter its nature or where it resides. While funny to a point to the people who are not involved or affected, stating that technically the data is not lost as you know where it is but you just can never access it again is no good at all and far from a joke.

The reality is that bad things do happen no matter how well you try to prevent it. So, you should never bank on solitary solutions. Always give yourself multiple options. From the good old 3-2-1 rule and to having air-gap studies; to not banking on one type of storage, technology, location, process or person, it is all part of adequate to excellent protection. If the business truly and honestly states its data is the gold that makes them thrive, they must act to protect it accordingly.

Are you going then?

No, unfortunately, I’m am otherwise engaged on the days of the conference. Didier, how can you say that to invest in attending VeeamON 2019 when you are not even going yourself? 1st of all, I’d go if I did not have prior engagements – I have done so in the past. I learned a lot and I’m still in touch with customers and Veeam employees I met there. Secondly, an investment does not always involve my physical person being on the scene. It is about getting the right knowledge and insights to the right people in the right places. That means sending colleagues, employees, customers to conferences and training and not always yourself. You develop the skills, knowledge, and capabilities of your organization. Data availability is a primary concern and a goal that I can only achieve when it does not depend solely on me. That too is investing. Helping others learn, grow and succeed help everyone out. Needless to say, I wish I was able to go.

Attend VeeamON 2019

Given the important task of keeping data available and the responsibility to protect and having the ability to recover it when things go wrong, you have a business case to attend VeeamON 2019. In today’s diverse, complex, and fragmented IT ecosystem that data is spread across more places, in different layers and in more elaborate solutions than ever before. This makes data protection and availability a challenge.

When business happens at the speed of light you also have to make data available at that pace when required in different locations for various use cases. Data mobility is a reality and this also has to happen in a secure and workable manner.

We can use all the help we can get to make this happen. The good news is you do not have to go it alone. Attend VeeamON 2019 to give yourself a head start. Invest in your business and your staff. Help them succeed. Register!

You can tap into the collective brain of all the attendees. Talk to knowledgeable and experienced practitioners. Learn from your peers and Veeam experts. Gain insights from others on how to approach the challenges you face. Have that discussion to gain a better understanding of what you can do and how. Get your compass aligned so you can confirm you are right on track to deliver what is asked for you. After all, it is not a small or benign job, to have to ensure the safety of your organizations 21st-century gold, its data. Discover what Veeam is doing in R&D to help you meet future requirements where ever your date lives.

Don’t forget that others can learn from you as well. Consider responding to the request for proposals. Share what you know and have learned. Contribute and learn whilst teaching. Invest in attending VeeamON 2019.

I am speaking at Experts Live Netherlands 2019

I am speaking at Experts Live Netherlands 2019

I have the distinguished honor of announcing that I am speaking at the Experts Live  Netherlands 2019 conference.

This conference is a high quality event for ICT professionals in any role in any place. Infrastructure, DevOps, DevSecOps, 0365, automation as well as data & service availability and protection in any cloud (private, hybrid and pubic).  Over the years it has proven to be an excellent conference that has grown fat beyond the Dutch borders. It’ both rewarding and humbling to be able to speak there.

RDMA & SMB 3: State of the union

I will be giving a state of the union on RDMA & SMB Direct. We’ll look at best practices, guidance, trends and current developments that are interesting to decision makers, analysts, architects and hands on professionals that work with SMB Direct in ever more use cases.

As always I’ll be around the entire day to learn and educate myself, networking and I am available to talk shop on any subject within my area of expertise.

Call to Action

You need to do one simple thing. Register here to secure your spot and come learn with and from your peers.

The content is excellent, the speakers are passionate technologists, the attendees smart and experienced practitioners. This means it is an excellent opportunity to network, share ideas, experience and gain new and different insight. All this helps you set a course for your work and career. Make the most of it and have fun while doing so!

Windows Server 2019 SMB Direct Best Practices

Windows Server 2019 SMB Direct Best Practices

The Hyper-V amigos, @Hypervserver and working@workinghardinithardinit recently did a webinar together about Windows Server 2019 SMB Direct Best Practices. We also discuss some trends and put some things into perspective. Rachfahl IT Solutions does more of these cool webinars for you to check out (see the info at the end of the video). You can watch the webinar below on Vimeo. It’s quite an honor to be invited to talk on this subject as Carsten is one the worlds most experienced S2D practitioners.

https://vimeo.com/319818020
Windows Server 2019 SMB Direct Best Pratices Webinar

Need to know more?

I hope this get’s you started and updated. Need to know more? Want more details, advice and a deeper and more elaborate discussion. I will be talking about this on various occasions this year. One of them is the Cloud & Datacenter Conference Germany 2019 in Hanau. Register to secure your spot. It is a great conference with a lot of hands on, real life knowledge being shard. I will be around for the Hyper-V pre day and during the entire conference. This means you can find me to talk shop. Be warned, I can go one about the subject or a while