Fixing ESXi 6.5 On HPE G7 Servers3 min read

HP, you motherfuckers.

So there I was, moving VMs off of my main host (DL380 G7, 2x x5690’s, 192GB RDIMM), getting ready to replace the 8x 300GB RAID10 array I’ve been using for a while now with some 1TB disks and SSDs, awesome, right?

So the host was powered down, ready for an upgrade, and being the logical guy I am I decided to do some software upgrades.

I used the latest SPP and ran that through, for some reason I was using a BIOS from 2010? Once that was done I had updated firmware, so I moved onto ESXi. I’ve been running 6.0.0U2 for the longest time and thought this would be the ideal time to upgrade.

I got the HPE official ISO, ran through the upgrade, waited for it to boot annnddd….

PSOD.

After some digging, it would appear this is actually a known issue with most G7 servers (unsure about G6).

The issue.

It would appear that one driver is causing this in the 6.5 image,  “hpe-smx-provider” (650.03.11.00.17-4240417). Installing the standard ESXI 6.5 ISO does allow the server to boot, but is missing a lot of drivers and does not give the pretty all-inclusive system stats that the HPE ISO does.

So what now? I didn’t want to settle for the standard image, so I did the following to bundle the U2 hpe-smx-provider into the 6.5 image.

For the lazy: There is a link at the end of this post with a download to the compiled ISO.

Using PowerCLI Image Builder.

The first thing you want to do is download PowerCLI for Windows.

Once that is all installed, grab both of the following files and put it somewhere easy, I just shoved them in C:\HPE.
VMware-ESXi-6.5.0-OS-Release-4564106-HPE-650.9.6.0.28-Nov2016-depot.zip
VMware-ESXi-6.0.0-Update2-3620759-HPE-600.9.6.5.7-Dec2016-depot.zip

I also renamed them to make it easier for myself, “ESXIHP65” “ESXIHP6U2”.

Once that is done, fire up PowerCLI, and do the following commands:

  1. Add the 6.5 image bundle
    Add-EsxSoftwareDepot -DepotUrl C:\HPE\ESXIHP65.zip
  2. Check the profile was added.
    Get-EsxImageProfile
  3. Clone the profile so we can work on the clone.
    New-EsxImageProfile -CloneProfile HPE-ESXi-6.5.0-OS-Release-6* -Name "ESXICUST"
    Set Vendor to whatever, I used ‘Custom’.
  4. Check the profile was added.
    Get-EsxImageProfile
  5. Remove the driver from the clone.
    Remove-EsxSoftwarePackage ESXICUST hpe-smx-provider
  6. Add the 6.0U2 image bundle
    Add-EsxSoftwareDepot -DepotUrl C:\HPE\ESXIHP6U2.zip
  7. Check the profile was added.
    Get-EsxImageProfile
  8.  Now we can see both driver versions from the bundles.
    Get-EsxSoftwarePackage | findstr smx
  9. Add the good driver into the custom image.
    add-esxsoftwarepackage -imageprofile ESXICUST -softwarepackage "hpe-smx-provider 600.03.11.00.9-2768847"
  10. Export your custom bundle to ISO.
    Export-EsxImageProfile -ImageProfile ESXICUST -ExportToIso -filepath "c:\HPE\ESXI65CUST.iso"
    You should now have ‘ESXI65CUST.iso’ in your working folder.

Result.

So, once that’s all done mount the ISO however you like, I use iLO and go through the install as per normal, continue through the warning about CPUs of this socket being depreciated. And, eventually….

BOOM.

Post edit: You must do a clean install for this to work, upgrading will not work as the offending driver will not be overwritten. 

If you’re lazy and trust me enough then here is a link to the compiled ISO, ready to be installed.

I hope this helped some of you out, best of luck! MM~~

86 thoughts on “Fixing ESXi 6.5 On HPE G7 Servers3 min read

  1. Hello, Great post btw! I am having the same issue with my dl380’s. So I have 6U2 and 6.5 HPE images. Did you just extract the iso and save as zip? I’m stuck on step one. I extracted the iso saved contents as zipped ran powercli with same commands and the index.xml file cannot be found. Any help is appreciated!

    Reply

    1. So the files your looking for are actually the offline bundles for running on the host, not the ISO. These can be acquired on the internet if you search for the packages. They are also on the /r/homeab FTP, you can find that on the Homelab wiki…

      Reply

  2. I think the main problem here is that even everything is working correct this server isn’t supported for 6.5 by VMware neither by HP. So for homelab or labs this is totally fine but when running this server in production and you have further problems, you would be out of luck when opening a support ticket.

    http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=server&productid=12845&deviceCategory=server&details=1&keyword=%22dl380%20g7%22&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

    http://h17007.www1.hpe.com/us/en/enterprise/servers/supportmatrix/vmware.aspx#.WK6jFxLsLxg

    Reply

    1. Indeed, however even though it’s officially unsupported it kind of sucks that HP either didn’t bother checking before release or checked and didn’t care. I still see plenty of G7’s in the wild and considering you have to pay for even software updates one would think they would repay that with at least checking their updates…

      HP have now released a patch for this now, only a week or so after I posted this so all is well again :). I am getting a lot of questions about this and I can’t seem to find it now… If I do I will make sure to update this post.

      Reply

      1. Where did you did you find the patch that HP released? I can’t see that their ISO available for download has been updated since November 18th, and still no HP ISO for the 6.5a version either.

        Reply

  3. Can you direct me to the patch? I have installed the STD image and when trying to install the offline bundle for the HPE drivers I still get the PSOD.

    Reply

    1. Update not working. You must reinstall. I do not understand why … but that does work for me.

      Reply

      1. Probably because an update to the modified version doesn’t replace the broken driver with the older one.

        Reply

        1. So I did a complete reinstall to SD card as I had to replace it. Everything went well with the custom ISO but VC is not showing sensor data. I have removed host from VC and reset entire network to reconfigure it again then added it back. Any thoughts?

          Reply

          1. Also it is the Web Client that I am referring to as fat client is no more. I did run it and connect to the host though to see the hardware data. Just need Web Client working. 🙂

  4. Thx Bro – that helped me a lot

    a shame for HPE, it is still not fixed

    Reply

  5. I’m getting error 14 during the install of the custom image, as well as the vanilla 6.5, any ideas?

    Reply

  6. Hi.I have faced the same trouble too.I have tried your guide step by step,but it still does no work. (I used upgrade way,not reinstall way) So I’m interested about your firmware version.I used HP SPP released at 2016.10 to updated my DL380 G7 server.My BIOS is using ver 2015,P410i firmware is ver 6.64,and ILO ver 1.88.How about yours?Thanks.

    Reply

    1. OK,I solved it by myself.Here is my experience:DO NOT USE UPGRADE Mode.It’s necessary to format the whole ESXi system and execute a new installation.

      Reply

      1. Ah right, yes this is the case. I will add this into the tutorial.

        Glad to see you worked it out.

        Reply

  7. So when attempting to clone the profile I am getting an error that some of the VIBs failed to validate. Any Ideas on this?

    Reply

    1. Redownload and check hashes to make sure the file isn’t corrupted.

      Reply

      1. It was apparently the version of PowerCLI I was using. That being said after rebuilding the image I am still getting the PSOD. I am going to try a full install instead of an upgrade and see hwo that goes. THanks again.

        Reply

        1. You need to full install, an upgrade will not override the offending driver, I’m going to add that into the post now.

          Let me know how it goes!

          Reply

          1. Still no go. I had tried adding some other drivers from newer HPEUTils release so one of these may still be causing it. This is really frustrating that HPE won’t release a fixed image. Next round of servers we buy I will vote with my pocket book.

          2. Well I got it now. Initially I was trying to slip in some of the latest HP Updates and do this at the same time. I think my script was overwriting something when I did it. Here is a quick script to do it all.

            Get-EsxSoftwareDepot | remove-esxsoftwaredepot

            $HPImage = “A:\ESXi_ISOs\ESXI_HP6-5_2.zip”
            $BaseHPImage = “A:\ESXi_ISOs\ESXI_HP6-5_Cloned.zip”
            $NewDriverDepot1 = “A:\ESXi_ISOs\ESXi_HP6-0.zip”
            $NewDriverDepot2 = “A:\ESXi_ISOs\HPEUtils.zip”
            $FixedBaseHPImageIso = “A:\ESXi_ISOs\HPE_ESXi_6-5_Final_04272017.iso”
            $BadBaseImage = “A:\ESXi_ISOs\ESXI_HP6-5_2.zip”
            ###Make a clone of the HP Images
            Add-EsxSoftwareDepot $HPImage
            $BaseHPImageName = Get-EsxImageProfile
            New-EsxImageProfile -CloneProfile $BaseHPImageName -Name $BaseHPImage -Vendor HPCustom
            $BaseHPImageName = Get-EsxImageProfile -Vendor HPCustom
            Export-EsxImageProfile -ImageProfile $BaseHPImageName -ExportTobundle -Force -FilePath $BaseHPImage
            #Add HP custom Image
            Get-EsxSoftwareDepot | remove-esxsoftwaredepot
            Add-EsxSoftwareDepot $BaseHPimage
            Get-EsxSoftwarePackage -Name hpe-smx-provider
            $BaseHPImageName = Get-EsxImageProfile
            Remove-EsxSoftwarePackage -ImageProfile $BaseHPImageName -SoftwarePackage hpe-smx-provider
            Export-EsxImageProfile -ImageProfile $BaseHPImageName -ExportTobundle -Force -FilePath $BaseHPImage
            Get-EsxSoftwareDepot | remove-esxsoftwaredepot
            Add-EsxSoftwareDepot $BaseHPImage
            Get-EsxSoftwarePackage -Name hpe-smx-provider
            ####HOpefully the above get command is empty.
            Add-EsxSoftwareDepot $NewDriverDepot1
            Get-EsxSoftwarePackage -Name hpe-smx-provider
            ####HOpefully you just found my driver levi….
            Add-EsxSoftwarePackage -ImageProfile $BaseHPImageName -SoftwarePackage hpe-smx-provider
            Export-EsxImageProfile -ImageProfile $BaseHPImageName -ExportTobundle -Force -FilePath $BaseHPImage
            Get-EsxSoftwareDepot | remove-esxsoftwaredepot
            Add-EsxSoftwareDepot $BaseHPImage
            Get-EsxSoftwarePackage -Name hpe-smx-provider
            Get-EsxImageProfile | Export-EsxImageProfile -ExportToIso -Force -FilePath $FixedBaseHPImageIso

  8. Hi,

    Using VMware Update Manager, I tried to upgrade HP ProLiant BL460c G7 ESXi from 6.0 to 6.5 using HP image, but it failed with PSOD.

    Then I performed fresh installation using same HP image and PSOD encountered again. Then I tried fresh installation using VMware standard image and it worked fine.

    I’ve noticed you mentioned standard image would be missing a lot of drivers. Is there any drawback using standard VMware image? If we must use HP image then we must follow the above process?

    Please suggest.

    Kind regards,

    Mobin

    Reply

  9. Hey Muff. I just used the standard ISO from the VMWare site. What should I do to run the HPE image? Everything works (seemingly) at the moment so I’m not sure what to do!

    Reply

    1. So, there isn’t a real issue with running the base image however you will be missing some system stats, monitoring and drivers that are baked in.

      I pretty much need these things, and whilst you can add them into the stock image with VIBs it is a lot easier just to use the HP image with everything baked in, things like iLO management and RAID management from the ESX CLI can all be done with the HP image. If you also look at the hardware view of the host you will get a *much* better view with the HP image.

      To use the latest HP image on your G7 host just do the above.

      Cheers, MM

      Reply

      1. So, I tried it, followed the guide and ran the ISO, started the upgrade and got a random directory is not empty error at about 19% and then it stopped… Rebooted the box and ESXi failed to boot, ended up having to restore from a backup before the upgrade. I think I’ll stick with dodgy sensors I think!

        Reply

        1. If you tried to upgrade instead of do a clean install this would be the issue FYI.

          Reply

          1. My bad I missed that huge lettering! Do you just suggest to re-register the VMs or is there a better way to do it from a fresh install?

          2. It’s all good.

            So I normally vMotion VMs somewhere else but if that isn’t an option for you then just shut them down, do the clean install and readd them from the datastore once you’re back up and running.

            Should be pretty painless unless you have loads of VMs.

            If you ensure you setup networking with the same names as before the VMs should just import and start without issue.

            MM~~

  10. Great work! As i am totaly noob to VMs and VMware in general but recived an outsorted ML110 G7 to begin learn this stuff at home, i like to ask: is there a posibility that you share your custom made image with us? Kind regards, Adam

    Reply

  11. This is a great post, you saved me a lot of work.
    Great job and BIG THANKS!!!!

    Reply

  12. hi,

    thx for this documentation. On current HPE image I also need to disable hpe-cru after downgrade the smx.
    This was on dl385G7

    Reply

  13. Hi MM

    HPE released a new image recently:

    VMware-ESXi-6.5.0-OS-Release-5146846-HPE-650.9.6.5.27-May2017.iso

    Does that fix the problem? If YES, does it fix it

    – for UPDATE
    – only for FRESH INSTALL

    If you look at HPEs “content of the image”
    http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c04430318

    You see the following…

    6.5 May 2017 vSphere 6.5 Build 5146846

    hpe-build – 650.9.6.5.27-4240417
    hpe-cru 650.6.5.8.24-1.4240417
    hpe-ilo 650.10.0.2-2.4240417
    hpe-smx-provider 650.03.11.00.17-4240417
    hpe-esxi-fc-enablement 650.2.6.10-4240417
    amshelper- 650.10.6.0-24.4240417

    So is this image GOOD ?

    Did you test it?

    Any feedback would be appreciated 🙂

    Regards,
    Oliver

    Reply

    1. I am interested in the latest release also from HPE if it works or has same issue.

      Reply

  14. hi
    i test this image: VMware-ESXi-6.5.0-OS-Release-5146846-HPE-650.9.6.5.27-May2017.iso directly from hp with NO PATCH, and it’s NOT WORK :(.

    please would you like to update your iso file in the first page?

    many thanks….

    Reply

  15. Thank you for this post! Saved me a boatload of time! Fresh install worked like a charm – but there was a random reboot after the first install/reboot.

    Reply

  16. I try the new ISO with the described workaround and it works on a DL385 G7. But under the monitoring of the storage thjere seems to be an issue. There are no metrics available and I see just this text: “The Small Footprint CIM Broker Daemon (SFCBD) is running, but no data has been reported. You may need to install a CIM provider for your storage adapter.”

    Any idea?

    Reply

    1. I too am working on this issue. Does anyone know if this is caused by the smx driver (hpe-smx-provider–600.03.11.00.9-2768847–HPE–VMwareAccepted–2016-01-22) replacing hpe-smx-provider–650.03.11.00.17-4240417–HPE–VMwareAccepted–2017-11-15? I’m working on a comparison of all VIBs but its hard to tell what VIB provides what functionality.

      Reply

  17. I´ve tried it with a DL360 G7 – new install. It fixed the purble screen issue – but I wasn´t able to get it work. I had problems with the esxi web GUI – I wasn´t able to configure a network to my vm´s … And I´ve got many display problems and it was extremly slow so I decided to install the 6.5 from vm ware again. I really hope, that HP will get it work soon. It makes no fun …

    Reply

  18. Albeit I’m very clear on how to get iso images on to usb thumb drives, and other isos work perfectly for installing, I’m failing with the supplied MUFFINS6.5FIX.iso

    Has anyone gotten a DL380 to boot off of MUFFINS6.5FIX.iso and if so, what’s your RC5 checksum?

    Thanks

    Reply

    1. iLO image install worked. Thumb drive not. Thanks for the image.

      Reply

      1. I was unable to boot with any USB device connected, PSODs on VMKUSB — I have read that 6.5 added/replaced the USB drivers with this new module (however removing it doesnt help). If I boot the installer fully, with all USB removed (so yea, have to iLo, PXE, or burn a CD..yuck) then plug in my target USB stick I’m able to install no problem

        Reply

  19. Do you have a build for the Gen 7 servers with the latest ISO that was recently released?

    Reply

    1. I would try the generic VMware update and not the HPE one as the VMware ISO wont have any HP specific drivers. VUM should also run a check on compatibiliy issues first. But no guarantees on the outcome 😉

      Reply

  20. I have 6.5 installed on a ML350 G6 succesfully using the generic VMware ISO, so no HP drivers installed. Now I’m obviously missing out on hardware monitoring. What would happen if I install th hpe-smx-provider-600.03.11.00.9-2768847.vib on my ML? I’m a bit hessitant because I’m not a great fan of PSOD’s 😉

    Reply

          1. Nice, thumbs up. I need to create one before the Spectre and Meltdown mitigations were built in.

          2. Hi! I just tried this. It ran for 3 days, I rebooted the host, and then it wouldn’t boot past the PSOD of the -smx error.

    1. @Richard, did you do any changes to the iso? did you update anything after install? is your hw ok? I have it running still and have no issues so far.

      Reply

      1. Same here, no issues. Built one from scratch pre-meltdown patches.

        Reply

  21. Hi MM,

    Do you have updated .iso with release 7388607?
    This is the last ESXi issued from VMware which I have.
    Unfortunately suffers same illness as the previous image but I do not have those vib file “hpe-smx-provider”.
    Also do you have by any chance 1.89 BIOS for DL360G7?

    Regards,
    MSM

    Reply

  22. I am looking for a version without the spectre/meltdown patches, for example 6765664 from november 2017. I don’t want to cap this system.

    Reply

    1. For example can I just use muffin and update with the original ISO (non-hp) to 6765664?

      Reply

      1. I would suggest you build one from scratch. Also, would be nice if you could upload it somewhere and post the link 🙂

        Reply

  23. I used this on a fresh install and it worked great. I am curious. Can I use this to upgrade from 5.5 to 6.5 or would I need to perform a clean install? I have a few G7s. Here we are nearly a year later and the latest custom ISO doesn’t seem to work.

    Reply

  24. Just ran this on a refurbished G7, the 6.5 image worked flawlessly, thanks so much for the help!!!

    Reply

  25. Hi all,

    what do you think, is it working with the new HP 6,7 ESXi custom iso? thanks in advance!

    Reply

    1. Someone needs to test it. I think I can only do it in 2 or 4 weeks, not sooner.

      Reply

  26. Hello!
    I want to make a fresh build based on the latest releases
    VMware-ESXi-6.0.0-Update3-6921384-HPE-600.10.2.0.23-Feb2018-Depot
    and
    VMware-ESXi-6.5.0-Update2-8294253-HPE-Gen9plus-650.U2.10.2.0.14-May2018-Depot

    moved smx-provider,
    added drivers
    QLG-NetXtremeII-5.0-offline_bundle-4362634 and scsi-hpsa_6.0.0.124-1OEM.600.0.0.2494585

    What else should I add or remove drivers to get a full build?

    Thank you!

    Reply

    1. We use QLogic 10GB SFP adapters NC523SFP 2 Port 10GB adapters. I have been installing this after running your ISO.

      Reply

  27. Hi- This has been working well. I just updated a few G7s to 6.5 U2. It seems that this may have broken the health sensors. Whenever vCenter updates the hosts hardware information (seems to run a few times a day), I see an invalid fault in the logs for the hosts I’ve updated. I have to assume it’s something with a VIB the G7s are using. Any insight?

    Reply

  28. I have a server for testing. Can test iso image if yuor compile it.

    Reply

  29. Hey,

    Whats your login for the esxi backend of your image?

    Reply

  30. What I don’t understand is that the 6.5 works on the G6 servers no issues but not on the G7/G8 servers.

    Reply

    1. Mark, you mean to say that the HP 6.5 ISO works on an ML350G6? Because it doesnt…

      Reply

    2. DL380/360 G6 servers use iLO2 which does not cause issues with 6.5, which is why that works.

      Reply

      1. Well, I have an ML350G6 with ILO2, and I can guarantee you that upgrading from i.e. 5.5 or 6.0 to 6.5 using the HP ISO will lead to many, many hours of recovery fun!

        Reply

  31. Hi Guys – I have a HP DL380 G6 and concur that an unmodified HP 6.5 image doesnt work and recovering isn’t fun.

    Does anyone know if i inject hpe-smx-provider 600.03.11.00.9-2768847 driver into a HP ESXi 6.7 image will work?

    Thanks in advance.

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.