Resetting the VASA Certificates with purecert

Delete the existing VASA certificates:


# purecert delete vasa-ct0
Name
vasa-ct0

# purecert delete vasa-ct1
Name
vasa-ct1

# purecert list
Name        Status    Key Size  Issued To     Issued By  Valid From               Valid To                 Country  State/Province  Locality       Organization  Organizational Unit    Email                                   Common Name
management  imported  2048      10.21.88.112  Sub-CA     2019-09-26 12:45:28 PDT  2021-09-25 12:45:28 PDT  US       California      Mountain View  Pure Storage  Solutions Engineering  administrator@sso.alex.purestorage.com  10.21.88.112

Create self signed VASA Certificates that have the Organization and Organizational Unit set to 'Pure Storage':


# purecert create --common-name 10.21.88.113 --country US --self-signed --key-size 4096 --locality 'Mountain View' --organization 'Pure Storage' --organizational-unit 'Pure Storage' --state California vasa-ct0
Name      Status       Key Size  Issued To     Issued By     Valid From               Valid To                 Country  State/Province  Locality       Organization  Organizational Unit  Email  Common Name
vasa-ct0  self-signed  4096      10.21.88.113  10.21.88.113  2019-10-28 07:37:42 PDT  2029-10-25 07:37:42 PDT  US       California      Mountain View  Pure Storage  Pure Storage         -      10.21.88.113

# purecert create --common-name 10.21.88.114 --country US --self-signed --key-size 4096 --locality 'Mountain View' --organization 'Pure Storage' --organizational-unit 'Pure Storage' --state California vasa-ct1
Name      Status       Key Size  Issued To     Issued By     Valid From               Valid To                 Country  State/Province  Locality       Organization  Organizational Unit  Email  Common Name
vasa-ct1  self-signed  4096      10.21.88.114  10.21.88.114  2019-10-28 07:37:59 PDT  2029-10-25 07:37:59 PDT  US       California      Mountain View  Pure Storage  Pure Storage         -      10.21.88.114

Note:

When resetting the VASA certificate the O and OU must be set to "Everpure" for both. This is a requirement for resetting the certificate. This is due to how the VASA service determines if the certificate is a custom/imported certificate or if it is a default certificate that will allow a vCenter Server to import it's own certificate to the VASA service. Should the O and OU be anything else than "Everpure" this will not create the correct default certificate in VASA and vCenter will fail when registering the storage Provider.

Check that the new certificates show up in purecert list:


purecert list
Name        Status       Key Size  Issued To     Issued By     Valid From               Valid To                 Country  State/Province  Locality       Organization  Organizational Unit    Email                                   Common Name
management  imported     2048      10.21.88.112  Sub-CA        2019-09-26 12:45:28 PDT  2021-09-25 12:45:28 PDT  US       California      Mountain View  Pure Storage  Solutions Engineering  administrator@sso.alex.purestorage.com  10.21.88.112
vasa-ct0    self-signed  4096      10.21.88.113  10.21.88.113  2019-10-28 07:37:42 PDT  2029-10-25 07:37:42 PDT  US       California      Mountain View  Pure Storage  Pure Storage           -                                       10.21.88.113
vasa-ct1    self-signed  4096      10.21.88.114  10.21.88.114  2019-10-28 07:37:59 PDT  2029-10-25 07:37:59 PDT  US       California      Mountain View  Pure Storage  Pure Storage           -                                       10.21.88.114

Register the Storage Providers in vCenter: Follow the section "Registering FlashArray VASA Providers with the Plugin" from Web Guide: Implementing vSphere Virtual Volumes with FlashArray to register the Storage Providers in vCenter.

Note:

Note that the initial attempt to register the Storage Provider may fail. This could be due to the VASA Provider needing to update the Self Signed Certificate to include the SAN entry for the IP address. Generally speaking, registering the Storage Provider will work on the first attempt; however, there is a chance that you may want to wait 5 to 10 minutes. The job that updates the VASA certificates SAN entries is scheduled to run every 5 minutes. In the event that the Storage Provider needs to be re-registered as soon as possible, please open up a support case and the TSE working the case can restart the VASA service on each controller to force the SAN entry to be updated in the certificate stored in VASA.

Confirm that the VMCA signed certificate is imported to vCenter:


purecert list
Name        Status    Key Size  Issued To     Issued By  Valid From               Valid To                 Country  State/Province  Locality       Organization  Organizational Unit    Email                                   Common Name
management  imported  2048      10.21.88.112  Sub-CA     2019-09-26 12:45:28 PDT  2021-09-25 12:45:28 PDT  US       California      Mountain View  Pure Storage  Solutions Engineering  administrator@sso.alex.purestorage.com  10.21.88.112
vasa-ct0    imported  2048      10.21.88.113  CA         2019-10-27 07:39:49 PDT  2020-10-27 07:39:49 PDT  US       -               -              Pure Storage  Pure Storage           -                                       10.21.88.113
vasa-ct1    imported  2048      10.21.88.114  CA         2019-10-27 07:50:06 PDT  2020-10-27 07:50:06 PDT  US       -               -              Pure Storage  Pure Storage           -                                       10.21.88.114

Keep in mind that this does not configure and setup the VASA Providers for multi vCenter support. These VMCA signed certificates will only allow one VMware SSO to register and authenticate with the VASA Providers.

Null VP URL on ESXi Hosts after Re-Registering the Storage Provider

When resetting the VASA Provider Certificate, Pure has noticed that the vvold service on the ESXi hosts will have the old certificate cached and will fail to authenticate with the VASA Provider. Essentially, the vvold service needs to have the ssl_reset ran against that. The easiest way to do that is to refresh the CA certs from vCenter to the ESXi hosts. As part of that process, the ESXi hosts will update the ssl certs on the ESXi host's services. Here is an example of using PowerShell and PowerCLI to refresh the CA Root Certs on the ESXi hosts in vCenter.

Leverage the following workflow:


## Connect to the vCenter Server ##
Connect-VIServer -server vcenter-server
  
  
## Get the ESXi hosts and set it to a variable ##
$hosts = get-vmhost
  
  
## Start the Service Instance ##
$si = Get-View ServiceInstance
  
  
## Start the certificate Manager view ##
$certMgr = Get-View -Id $si.Content.CertificateManager
  
  
## Using the Cert Manager, refresh the ESXi hosts Certs ##
## This pushes all certificates in the TRUSTED_ROOTS store in the vCenter Server VECS store to the host. ##
$certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)
  
  
## Now in vCenter the vvol datastore should be accessible for each of those hosts.  No need to do the ssl_reset and restart on VVold ##

Here is an example of this workflow:


PS C:\> Connect-VIServer -Server dev-vcsa

Name                           Port  User                          
----                           ----  ----                          
dev-vcsa                       443   ALEX\Carver                   

PS C:\> Get-Cluster -Name "Dev Cluster"

Name                           HAEnabled  HAFailover DrsEnabled DrsAutomationLevel  
                                          Level                                     
----                           ---------  ---------- ---------- ------------------  
Dev Cluster                    True       1          True       FullyAutomated      

PS C:\> $ESXi_Cluster = Get-Cluster -Name "Dev Cluster"

PS C:\> $ESXi_Cluster | Get-VMHost

Name                 ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz   MemoryUsageGB   MemoryTotalGB Version
----                 --------------- ---------- ------ ----------- -----------   -------------   ------------- -------
esxi-7.alex.pures... Connected       PoweredOn      16         151       38304          14.586         255.897   6.7.0
esxi-6.alex.pures... Connected       PoweredOn      20         141       43880          16.166         255.892   6.7.0
esxi-4.alex.pures... Connected       PoweredOn      20          94       43880           8.945         255.892   6.7.0

PS C:\> $hosts = $ESXi_Cluster | Get-VMHost

PS C:\> $hosts

Name                 ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz   MemoryUsageGB   MemoryTotalGB Version
----                 --------------- ---------- ------ ----------- -----------   -------------   ------------- -------
esxi-7.alex.pures... Connected       PoweredOn      16         151       38304          14.586         255.897   6.7.0
esxi-6.alex.pures... Connected       PoweredOn      20         141       43880          16.166         255.892   6.7.0
esxi-4.alex.pures... Connected       PoweredOn      20          94       43880           8.945         255.892   6.7.0

PS C:\> $si = Get-View ServiceInstance

PS C:\> $certMgr = Get-View -Id $si.Content.CertificateManager

PS C:\> $certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)

PS C:\>

Once this is all completed, the vvold service should be able to authenticate with the VASA Provider.

Checking or Inspecting the VASA Certificate

There may be times that part of the troubleshooting storage providers failing to register needs to include inspecting the VASA Certificate. vSphere will not be able to authenticate and connect to a VASA Provider if the certificate is not following x509 security standards with the certificate. The most common issues that are seen with certificate issues are actually on the vSphere and VMCA side of things and not with VASA. This is not to say that issues with the VASA certificate can't happen though.

Here are a couple issues to check when inspecting the VASA Certificate:

Does the Common Name match the Subject Alternative Name (SAN)?
When resetting the certificate, does the O and OU match "Everpure" for both?

The easiest way to inspect the certificate would be by navigating to the VASA Provider URL in a web browser and then inspecting the certificate. The example that is shown below is of an imported CA Signed certificate to VASA-CT0.

From a Chrome Browser on either Mac or Windows, follow these steps:

Using Chrome on Windows

Browse to the URL of the VASA Provider + /version.xml , for example, https://10.21.149.22:8084/version.xml and accept the risk and proceed.
Once the page is up, click on the lock next to the URL browser and click on the Certificate option.
In the next window, notice that the Issued To will be the Common Name that was used in the self signed cert generation or from the CSR generation.
Click on Details, scroll down to the Subject Alternative Names and click on it to show the details.

Make sure that the Common Name has a SAN entry. If it's a FQDN, then a FQDN entry should be there. If the CN is a IP, then the IP should be in the SAN.

Using Chrome on Mac

Browse to the URL of the VASA Provider + /version.xml , for example, https://10.21.149.22:8084/version.xml and accept the risk and proceed.
Once the page is up, click on the lock next to the URL browser and click on the Certificate option.
In the next window, Click the > next to Details to expand the selection. The O, OU and CN will be listed in the first part.
Scroll down further and find the Subject Alternative Names.

Make sure that there is a SAN entry that matches the Certificates Common Name.

From a Firefox Browser on either Mac or Windows, follow these steps:

Using Firefox on Windows

Browse to the URL of the VASA Provider + /version.xml , for example, https://10.21.149.22:8084/version.xml and accept the risk and proceed.
Once the page is up, click on the lock next to the URL browser and click > to show connection details.

Then click on More Information.
In the next window click on the Security Tab and then click on View Certificate.
All the Certificate Details are shown here. Notice the Organization, Organizational Unit and Common Name are in the first section "Subject Name" and the Subject Alternative Names are listed in a section below that. Make sure that the Common Name has a SAN entry. If it's a FQDN, then a FQDN entry should be there. If the CN is a IP, then the IP should be in the SAN.

Using Firefox on Mac

Browse to the URL of the VASA Provider + /version.xml , for example, https://10.21.149.22:8084/version.xml and accept the risk and proceed.
Once the page is up, click on the lock next to the URL browser and click > to show connection details.

Then click More Information.
In the next window click Security Tab and then click View Certificate.
All the Certificate Details are shown here. Notice the Organization, Organizational Unit and Common Name are in the first section "Subject Name" and the Subject Alternetive Names are listed in a section below that. Make sure that the Common Name has a SAN entry. If it's a FQDN, then a FQDN entry should be there. If the CN is a IP, then the IP should be in the SAN.

The point of checking the certificate is really to make sure that the Organization and Organizational Unit match what was provided and that there is a SAN Entry for the Common Name given. This helps confirm that the certificate was regenerated or imported correctly and that vCenter should not have problems with the certificate for the VASA provider.