HPE Virtual Connect SE 100Gb lost IPv4 Address

[CSKA]

Hello,

Recently one of our VC modules lost IPv4 address quite suddenly without any management operations over the configuration, we just received incident from our ping up/down monitoring agent. The IPv4 addresses are configured from the LE configuration, we used reserved IPs and I'm curious why this is happened? I’ve created official support case, but HPE lines L1/L2/L3 are not able to get any feedback for more than 2 weeks. The case ID is 5380239271.

My I asked @ChrisLynch  and his team to step in and look, seems like you are to only persons who knows what is happaning on OneView/Synergy behind the GUI.

OneView suggest reapplying configuration on the VC, but I would like to avoid a full config reset of the module, just for the IPv4 address lost. It seems like the IPv6 Link-Local Default IP is still in place, and OneView and FLMs can communicate with the module. The traffic on UP/Down ports is also still not affected, so we have something on the management level. I don’t understand why everything on these modules is locked and for each issue no matter the impact we have to reapply config. In our particular case, I’m not sure if that related, but this enclosure was patched a day ago to OneView 8.60.01 (from 8.30.01) and SPP SY-2023.10, but there is a window of 15hours where IP was reachable, and then accidently lost. In our environment we have more than 30 frames which are patched to the same FW level without any issues, but I still thought I should mention it as this is the last management operation.

[DanCernese]

Hello, sorry for the delay.  I don't have all your answers but I can explain a few things.  First, your case.. ..internal analysis in the support team found (a week ago), that a firmware update was performed and that's when the IPv4 address was lost (after the module rebooted).  This detected by OneView and the alert raised on 2024-02-27 10:18.  However, OneView may have told the module not to configure its IPv4 and that requires deeper analysis and an elevation was created.  That was worked on and moved to the next level just yesterday (12-March).  It is making progress and I apologize that nobody has updated you.

On the basics.. ..IPv4 is used only for external monitoring.  Synergy internal to the frames does not require it-- so the FLM & module and production traffic are completely unaffected.    Allowing direct modification of the module (for example to attempt to fix this IPv4 issue) would obviously put the module out of sync with the config that OneView is tracking; it would immediately become an unmanageable nightmere.

[CSKA]

Hello Dan,

Thank you very much for your explanation, it is really appreciated. Just to clarify, some important points from time perspective:

1. The update was completed on 26-02-2024 at 3.03PM CET  (26-02-2024 at 2.03PM GMT)
2. Ping lost started on 27-02.2024 at 10.36PM CET (27-02.2024 at 9.36PM GMT)
3. Alarm generated in OneView on 27-02-2024 at 11:18 CET (27-02-2024 at 10:18 GMT) is triggered by support dump which I've created. Prior dump creation the OneView was able to recognize the IP is missing (if you go to IC module and click on General Tab), but alarm was generated only after dump was created. The dump creation on LE also triggered 2 IC ADD activities which was marked to my username, but for sure, this was not initiated by me. 

So, there are around 20h between the firmware upgrade and IP lost. What is your proposal, to re-apply configuration as usual? This way we will lost redundancy and capacity during operation, and not sure that trigger the refresh on IC6 will not trigger automatic refresh on IC3, which is the pair in that LIG.

Regarding the ticket, I’m sending e-mails without any responses, no comments also on the case ☹

Below you can find screenshot for the activities mentioned

[DanCernese]

i do not want to set expectations inappropriately.  I cannot forward this easily to our support team nor can I speak to what a proper plan of action should be without working the entire process.  Please send this to your support case and ask that the elevation team receive your information.  They have the logs; so they already know what it looks like from a time sequence at the lowest level; but they may not understand what your observations were.

The generation of a support dump temporarily puts the module into 'maintenance' mode just to pull logs off it.   When it finishes; it goes through a normal 'add' lifecycle that should be entirely passive but perhaps you stumbled into a condition where it lost something.

[CSKA]

Thanks, I've forwarded the information to our account manager and will wait for official statement. 

[DanCernese]

Honestly, in the end the answer is almost certainly going to be to follow the resolution and reapply the configuration.  The question remains unanswered why this occurred.  The only way to fix/adjust VC module config or settings is always to push that config from OneView.  Despite the warning that there will be an interruption, there might not be (we can't guarantee it because if there's something off or out of sync in the module OneView is going to correct it). ..depending on the number of servers/frames, it would be done in 1-5 minutes; or in a worst case I believe near 6-8 (guessing!).

[CSKA]

Today I’ve did the following.

1. Reset the affected IC module.
2. Reapply configuration only on the affected module, from Interconnects menu.
3. Reapply configuration from Logical Interconnects view.

None of this helped me. New support dump was uploaded. After a while I understand from assigned L2 engineer, that I must reapply the whole logical enclosure. I’m not sure how to express my filings. As I already send my feedback to them, the procedure is like some advice to repaint a whole vehicle, when you have just a small scratch on the door. What will happen if reapply in LE doesn’t fix the issue? Should I expect, that next advice will be to restore from backup?

You know that to reapply the whole LE, all servers must be done, am I correct? From my perspective you have a bug here, it is interested that there isn’t anything from the dumps why this is happened and is there probability to be repeated. Please don’t get me wrong, but putting the whole enclosure in maintenance with downtime is not what the customers expects for such operation of assigning mgmt. IP address.

From my perspective one of the most widespread infra technology, which is in the market since ages, must have built in functionality, where the Composer is able to compare the actual running device configuration with the template configuration in the internal OneView DB and apply just the differences, but not the whole config.
Regards,

[DanCernese]

"From my perspective one of the most widespread infra technology, which is in the market since ages, must have built in functionality, where the Composer is able to compare the actual running device configuration with the template configuration in the internal OneView DB and apply just the differences, but not the whole config."

That is in fact what it is doing.  You've got a bug somewhere.  I agree-- reapply the LI is all that should have been necessary unless the detailed analysis found something extra-ordinary and they're not telling you.

[DanCernese · Accepted solution]

Our support can manually correct the modules individually, please take them up on that offer for a virtual remote session.  They had already fake-reproduced the issue and found performing the LE reapply resolved it.. ..but HPE is aware of the risk that some unknown mismatch on a server might trigger an interruption even though if things match they should not be touched.

[CSKA]

@DanCernese 

Thanks again Dan,
So please confirm I understand you right. HPE expert will be able to fix the IP, without reapply LE, on sharing session, touching the interconnect only?