Stacking connection failed and no redundancy

[tristar]

Today, replaced with the VC Flexfabric v4.31 in Bay 2.

From Checkpoint Pending to VC Domain Checkpointed took about an hour, I supposed synchronisation had taken place.

Followed up with vcsu on healthcheck, other than the stacking connection failed and stacking no redundancy, the rest were good, particulary both the Domain Configuration and Module Configuration were in sync for primary and backup.

With that, ran vcsu update with health option using v4.31 firmware to uplevel Bay 1, ended in fatal error. Reason being the VC Domain Configuration out of sync..

Rerun the steps an hour later, the result ended the same.

We followed the procedure as documented.

Anyone encounter this problem.

[Vinky_99]

The fact that the VC Domain Configuration is reported as out of sync during the firmware update process is concerning.

Here are some steps and considerations you can take to troubleshoot and address the issue:

>> Verify in the Virtual Connect Manager (VCM) interface that both Domain Configuration and Module Configuration are indeed in sync for both the primary and backup modules.

>> Ensure that the stacking cables are properly connected between the primary and backup modules (Bay 1 and Bay 2). Physical issues with stacking connections can lead to synchronization problems.

>> Check the error logs in the Virtual Connect Manager interface for any specific error messages related to the synchronization issue. This can provide more details about what aspect of the configuration is causing the problem.

>> Confirm that the firmware version you are attempting to install (v4.31) is compatible with your existing configuration and that it aligns with any specific requirements outlined in the release notes.

>> Instead of jumping directly from v4.30 to v4.31, consider whether there are any intermediate firmware versions that need to be applied first. Sometimes, certain versions may have dependencies on prior updates.

>> Double-check the HPE Virtual Connect documentation and release notes for any special instructions or considerations when upgrading firmware in a stacked configuration.

>> Before attempting the firmware update, run the health check using VCSU to ensure that everything is in sync. Address any issues reported by the health check before proceeding with the firmware update.

>> Before making any significant changes, ensure that you have a backup of the current configuration. This allows you to roll back to a known good state if needed.

>>Plan for a maintenance window to minimize the impact on operations during the troubleshooting and update process.

>>If the issue persists and you cannot find a resolution, it's advisable to contact HPE support.

[tristar]

Here are some steps and considerations you can take to troubleshoot and address the issue:

>> Verify in the Virtual Connect Manager (VCM) interface that both Domain Configuration and Module Configuration are indeed in sync for both the primary and backup modules.

Yes,  I run the vcsu healthcheck, return results show both Domain Configuration and Module Configuration are in sync. for primary (Bay 1) and backup (Bay 2).

>> Ensure that the stacking cables are properly connected between the primary and backup modules (Bay 1 and Bay 2). Physical issues with stacking connections can lead to synchronization problems.

Stacking use internal port s x9 and x10, there is no stacking cables used.

>> Check the error logs in the Virtual Connect Manager interface for any specific error messages related to the synchronization issue. This can provide more details about what aspect of the configuration is causing the problem.

VCM log indicated Bay 1 is degraded,. The root cause is the VC FLexFabric firmware mismatch. (Bay 1 is v4.30, Bay 2 is v4.31)

>> Confirm that the firmware version you are attempting to install (v4.31) is compatible with your existing configuration and that it aligns with any specific requirements outlined in the release notes.

The firmware version supports VC FlexFabric 20/40  F8 module and  VC 8GB 24-port FC module

>> Instead of jumping directly from v4.30 to v4.31, consider whether there are any intermediate firmware versions that need to be applied first. Sometimes, certain versions may have dependencies on prior updates.

There is no intermediate version, v4.30 was removed and replaced with v4.31.

>> Double-check the HPE Virtual Connect documentation and release notes for any special instructions or considerations when upgrading firmware in a stacked configuration.

There is no special instruction., to update the firmware using force health option in this situation.

>> Before attempting the firmware update, run the health check using VCSU to ensure that everything is in sync. Address any issues reported by the health check before proceeding with the firmware update.

VCSU version 1.15 is used.
Run vcsu healthcheck, both primary and backup Domain configuration and module configuration are in sync

Run vcsu update with force health ended in fatal error, stating that the VCM domain configuration is Out of Sync.

I had repeated the process a day after but still face the same error.

>> Before making any significant changes, ensure that you have a backup of the current configuration. This allows you to roll back to a known good state if needed.

I did the backup of current configuration (v4.30), but there is no v4.30 available for download, in case I need to fall back.

>>Plan for a maintenance window to minimize the impact on operations during the troubleshooting and update process.

>>If the issue persists and you cannot find a resolution, it's advisable to contact HPE support.

I already contacted HPE,  he presented readme of the version history update from v4.30 to v4.31,  with vcsu v1.10.1 at the time of release. He also said that vcsu v 1.15.0 should be able to do the job which currently I'm using.

I'm not sure should I use vcsu v 1.10.1  instead to see if it can solve the issue.

The next thing I try is to update Bay 1  alone  from v 4.30 to v 4.31 (with Bay 2 module removed) , to make it on par with Bay 2 version, before I put back Bay 2 module. In doing so, I'm not sure the VCM domain configuration would remain the same.

[tristar]

Hello HPE community

I'm now in deadlock situation, unable to update VC FlexFabric firmware to higher versions.

I tried to raise the Bay 1 firmware to v 4.31 first before upgrade both to higher versions, but without success.

I run vcsu v 1.10.1 update with health option, the process terminated with error VCM domain configuration Out of Sync. I also used the latest vcsu version 1.15.0 still encounter the same error. Did I miss out any step before running vcsu? Any clue to this situation?

[simoncorner]

Hi,

I had a similar experience a while ago and the solution was to reboot the module with the older firmware. This will make the newer one active and then you should be able to update the older one. to 4.31.

You will probably still need to use the "health" option to upgrade the older one.

Once that's complete, you should be able to get them up to newer versions.

Obviously there is a chance this will cause an outage so take a backup of your VC config and make the change outside of busy hours.

Simon

[tristar]

@simoncorner Thank you, Simon, for your valuable feedback.

A second thought.

Since Bay 1  VC Flexfabric is active with VCM domain configuration.

When reboot Bay 1 Flexfabric module, the Bay 2 Flexfabric would take over active role with the VCM domain configuration?

It is a production system, I don't have the privilege to work on test system.

Cheers.

[simoncorner]

Yes, the idea is that as long as the module in bay 2 has the config, as soon as you reboot bay 1, bay 2 will take over and then you should be able to update bay 1 to 4.45.
When I did this, I was confident that the standby module had the full vc config on.
The other option, as previously stated is to remove bay 2. Then upgrade bay 1 to 4.45 but that will incur an outage as when bay 1 reboots you won't have another module in bay 2 to switchover to.

[tristar]

Hi Simon

We are closer to overcome the issue. I share the vcsu healthcheck snippet. The report says stacking connection failed and no redundancy. The primary module and backup module domain configuration and module configuration are in sync. Can I safely say that the backup module also has the  same full VC configuration as the primary? As you mentioned to reboot Bay 1 and let Bay 2 to takeover as active. From here, I can update with health option Bay 1 firmware to the same as Bay 2, and subsequently, once both are at the same version, I can proceed on the update path to higher versions.

When I observed in the VC manager, I saw Bay 1 as degraded, and Bay 2 is incompatible. This is where my concern is.

Thank you for your sharing, very appreciative.

[simoncorner]

It certainly looks like the config is ok on both. I'm not sure what impact the stacking issues could have though.
I think it's worth trying but obviously it's not my system. Also, if you're running an air traffic control system or similar, more care may be required!

[tristar]

@simoncorner Totally agreed, different systems have their own behaviour, depending on the setup.

Your sharing method of reboot the active module of lower firmware, to make the passive one of higher firmware to takeover the active role. It worked in your case. I give it a shot.

I came across a forum being shared, the active module was of higher firmware version, and could update the passive module to the same version as active one.

So to say of encountering my issue,  it seems like if the active module is of lower firmware, when vcsu update with health option, the process ended with fatal error stating VCM domain out of sync.

Actually, my intention is either place the Bay 2 module in unused Bay or remove from the Bay as you also mentioned as another option. In doing the firmware update process takes about 40 minutes, and the systems are inaccessible during this period.

My only primary concern is to protect the VCM domain configuration, which I had backed up for VC v 4.30, but not sure is compatible with v 4.31 in case need to roll back, because v 4.30 no longer available. I also make a copy of vcshowall.txt.

Nevertheless, if I don't try, I never know the answer.

Cheers!