Storage Provider Certificate Has Expired

How-Tos for VMware Solutions

Audience
Public
Source Type
Documentation

Note:

Should the FlashArray be on Purity 5.3.0 or higher the steps to reset the certificate do not apply. Rather an Array Admin user can manually reset the VASA certificate or import a new certificate to the FlashArray. The process of removing and re-registering the Storage Providers will still be required. The workflow would be as follows.

  1. Remove the Storage Providers with expired certificates in vCenter
  2. Please refer to the KB about managing VASA certificates with purecert Web Guide: Managing the VASA Certificates with purecert via the CLI.
    1. If the certificate was signed by a CA, then the end user will need to generate a new CSR, get a newly signed certificate and import the newly signed certificate.
    2. If the certificate was the default vSphere certificate, then the end user will need to delete the existing vasa-ct0 and vasa-ct1 certificates, then create new default self signed certificates.
  3. Re-register the storage providers in vCenter.

This process isn't as easy or simple as renewing a cert that is about to expire. In this case you can't renew the cert; instead you have to remove the storage providers and then re-register the storage providers. However, as part of this process Everpure Support will need to manually clear the expired cert and restart the vasa provider. Otherwise the re-registration of the storage providers will fail. This process is something that Pure is working to improve moving forward.

The impact of having the storage providers cert expire will be impactful to any further vVol related operations that have to communicate with the VASA provider on the array. Such is powering on VMs, vMotioning VMs, deploying new VMs, etc. However, any currently running VMs will continue to run without impact. The process of removing the expired storage provider, clearing the expired cert and then restarting the VASA service will not be impactful to the currently running vVol VMs.

Here are the steps to follow in order to Renew a Storage Provider that has had it's certificate expire.

  1. Log into the vCenter that the Storage Providers are registered.
  2. Navigate to the hosts, vms or datastores tab and select the vCenter object.
  3. Select the configure tab and then the storage providers option. Locate the Storage providers that have the expired certificate.
  4. Here you can see if you try to renew the Certificate it will fail:

    Certificate is expired, unable to renew

  5. You will need to remove both Storage Providers

    Remove Both Expired Storage Providers

  6. If you have the storage providers registered with another vCenter that is in Enhanced Linked mode, be sure to remove them from all the vCenters that are registered with them.
  7. If the FlashArray is running Purity 5.3.0 or higher, Resetting the VASA Certificates with purecert.
  8. If the FlashArray is not running Purity 5.3.0 or higher then once both storage providers are removed from all vCenter Servers you will want to work with Pure Support. You will need to enable remote assist and reference this KB in your support case. There will be steps for Pure Support to follow.
Note:

Do not manually reset certificates if the FlashArray is running Purity 5.3.0 or higher in the same method as outlined in this KB for Purity 5.0, 5.1 or 5.2. All management of the VASA certificates must be done by the customer with purecert via the CLI on Purity 5.3.0 and higher.

Steps to Clear Keystore Cache and restart VASA Service

In order for the TSE to proceed, they will need the customer to open up Remote Assist to the Array. After that, confirm that this is the array that they need you to remove the expired Certificate.

Essentially all you will need to do is remove the keystore and truststore cache and restart the vasa service for both controllers.

Please refer to the How to check and clear Array Key Store for more information about this process, but here is what you will essentially do after the customer contacts you.

For Purity Versions 5.0.x and 5.1.0 - 5.1.5

These are the steps for Purity running below 5.1.6. If the array is running 5.1.6 or higher, the process is different

  1. Use Remote Assist to Connect with the Array (unless it's a dark site, then you'll need to do this over a zoom session)
  2. Get the output of the keystore, the password is "purestorage".
    
    keytool -list -v -keystore /cache/ui/keystore
    
    root@fs64-16-ct1:~# keytool -list -v -keystore /cache/ui/keystore
    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN
    Your keystore contains 1 entry
    Alias name: jetty
    Creation date: Jun 27, 2017
    Entry type: PrivateKeyEntry
    Certificate chain length: 1
    Certificate[1]:
    Owner: CN=Pure Storage, OU=Pure Storage, O=Pure Storage, C=US
    Issuer: OU=VMware Engineering, O=3m-vvol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
    Serial number: ff4ed0d401b85282
    Valid from: Mon Jun 26 12:51:30 PDT 2017 until: Wed Jun 27 12:51:30 PDT 2018
    Certificate fingerprints:
      MD5:  14:E9:2C:66:F8:74:95:8B:D6:61:FD:36:AC:AC:96:90
      SHA1: 9B:02:FC:BD:0B:D2:DF:08:C4:A5:B0:AA:51:EC:58:96:DD:78:8E:23
      SHA256: E1:54:21:A3:6C:1A:9D:FE:9C:A8:D5:19:32:61:8B:31:52:E7:9A:59:C2:06:85:FD:81:E9:0A:47:39:49:32:A9
      Signature algorithm name: SHA256withRSA
      Version: 3
    Extensions:
    #1: ObjectId: 2.5.29.35 Criticality=false
    AuthorityKeyIdentifier [
    KeyIdentifier [
    0000: 80 A3 05 66 4C 5C 84 33   45 A7 99 5F 52 78 10 49  ...fL\.3E.._Rx.I
    0010: 79 F9 FC 94                                        y...
    ]
    ]
    #2: ObjectId: 2.5.29.17 Criticality=false
    SubjectAlternativeName [
      IPAddress: 10.14.64.161
    ]
    *******************************************
    *******************************************
  3. Remove the keystore and truststore cache and restart the vasa service
    
    rm /cache/ui/keystore /cache/ui/truststore && service vasa restart
  4. Check the keystore and verify it's cleared
    
    keytool -list -v -keystore /cache/ui/keystore
    
    root@fs64-16-ct1:~# keytool -list -v -keystore /cache/ui/keystore
    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN
    Your keystore contains 1 entry
    Alias name: jetty
    Creation date: Feb 2, 2012
    Entry type: PrivateKeyEntry
    Certificate chain length: 1
    Certificate[1]:
    Owner: CN=Pure Storage, O=Pure Storage, OU=Pure Storage, C=US
    Issuer: CN=Pure Storage, O=Pure Storage, OU=Pure Storage, C=US
    Serial number: 4f2b166c
    Valid from: Thu Feb 02 15:04:12 PST 2012 until: Wed Jan 28 15:04:12 PST 2032
    Certificate fingerprints:
      MD5:  0D:E8:64:78:C5:66:5E:64:8C:4D:D8:AB:EC:78:20:FE
      SHA1: D8:8F:22:67:B8:9C:30:31:69:66:F4:23:E9:69:94:A4:B2:38:79:21
      SHA256: E2:3E:10:E5:A7:92:68:F0:63:0C:D2:81:ED:34:2F:97:B6:9E:08:7F:82:F6:E1:53:43:01:27:67:ED:2F:D9:DD
      Signature algorithm name: SHA1withRSA
      Version: 3
    
    *******************************************
    *******************************************
  5. Repeat the same steps above for the peer controller

  6. Update the customer that the expired certs have been cleared from the array cache and the VASA service has been restarted.

For Purity Versions 5.1.6+ and 5.2+

If the array is running Purity 5.1.6 or higher here is the process that should be followed for steps 2 through 4 above:

  1. Here is an example of printing the certificate information
    
    openssl x509 -in /cache/ssl/vasa.crt -text -noout
    
    
    root@SLC-OOTO-ct1:~# openssl x509 -in /cache/ssl/vasa.crt -text -noout
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 14093109525121868793 (0xc394c871d19023f9)
        Signature Algorithm: sha256WithRSAEncryption
            Issuer: CN=CA, DC=bootcamp-sso-1, DC=dev.purestorage.com, C=US, ST=California, O=Bootcamp-vCSA-1.dev.purestorage.com, OU=VMware Engineering
            Validity
                Not Before: Dec  9 13:47:32 2018 GMT
                Not After : Dec 10 13:47:32 2019 GMT
            Subject: C=US, O=Pure Storage, OU=Pure Storage, CN=10.204.120.145
            Subject Public Key Info:
                Public Key Algorithm: rsaEncryption
                    Public-Key: (2048 bit)
                    Modulus:
                        00:f2:16:fa:b1:c2:11:6f:f5:07:9d:9c:6c:11:d2:
                        0f:2d:a6:ad:66:35:5a:2f:04:17:07:46:98:76:02:
                        6d:33:36:03:ce:aa:c3:a9:5c:b9:e1:79:a1:9a:53:
                        ac:fe:a1:59:ca:26:32:b3:40:49:52:1a:3b:43:f7:
                        dd:9b:27:06:3f:ed:f8:91:b5:af:7f:07:a4:de:fc:
                        b7:d7:63:fc:3f:04:a1:f6:a6:38:94:5c:30:e0:c2:
                        f9:a6:c6:e3:9c:2f:ca:0a:c2:0d:97:72:32:4a:17:
                        02:e4:f3:60:6c:f8:aa:65:08:d0:3b:01:b0:9f:07:
                        3c:fe:4d:cb:4e:ee:93:76:4a:73:7a:9c:2e:63:2e:
                        82:e7:7a:da:5c:1b:50:60:ef:1e:75:8a:b7:c0:4d:
                        f0:da:dc:cd:e9:7b:e0:af:5e:c7:9a:ef:f2:d9:2d:
                        29:77:58:1c:c4:ac:4e:86:3f:70:46:4e:1f:a6:4f:
                        8c:08:0e:2a:05:fd:dd:f5:8d:b9:e8:fe:4e:09:15:
                        63:d8:88:62:f9:80:2a:e1:3a:cf:8b:b1:0e:02:ed:
                        ab:80:cf:25:25:6e:9a:08:d3:f9:66:29:e7:48:75:
                        8a:27:b6:3c:a1:c5:de:6c:40:f8:f4:0f:15:07:81:
                        3e:82:e2:e8:1d:45:da:fb:a7:42:35:80:8e:67:49:
                        96:8b
                    Exponent: 65537 (0x10001)
            X509v3 extensions:
                X509v3 Subject Alternative Name:
                    IP Address:10.204.120.145
                X509v3 Authority Key Identifier:
                    keyid:97:DE:EA:3E:0D:2F:80:B8:E1:C7:E9:FC:27:19:3D:FD:F3:0F:06:F0
    
                Authority Information Access:
                    CA Issuers - URI:https://Bootcamp-vCSA-1.dev.purestorage.com/afd/vecs/ca
    
        Signature Algorithm: sha256WithRSAEncryption
             46:98:e0:b9:c7:43:9c:8d:3b:d1:8e:37:2f:34:99:81:79:ad:
             c5:cb:cc:57:a3:ea:c0:13:73:8d:02:d4:9e:f1:f0:ef:92:d0:
             f5:45:6d:5b:e7:44:81:c0:d7:0d:72:9e:76:c6:9b:74:a9:b7:
             ee:ab:39:9a:b5:07:1b:63:6f:d9:f5:c7:ae:7e:77:18:e3:4b:
             97:10:bd:a5:bc:89:05:3c:55:4f:58:f0:d2:79:53:19:c2:51:
             9f:96:ab:d4:85:e4:0c:eb:db:a9:29:5e:a2:fa:51:7c:b3:5c:
             5b:b4:40:4c:bc:35:98:43:70:3c:53:f5:b7:b4:79:fe:ba:25:
             4e:3a:3c:43:a8:eb:cf:f6:a4:b3:12:07:64:e1:69:48:17:4c:
             84:68:fa:7b:63:05:96:75:0f:5c:a9:3a:86:99:b2:cc:a1:db:
             5d:b9:be:78:fa:c0:f5:95:eb:71:a2:da:bb:b3:8b:d4:3d:e1:
             86:29:14:e2:f9:e6:ff:3e:2b:63:87:94:7e:c3:1b:12:10:5f:
             6d:60:ac:22:b7:69:8e:e4:91:10:f8:e3:f1:6f:f8:08:fa:2a:
             73:ce:97:a4:21:8f:1a:e4:1c:26:c3:d9:5e:e8:aa:c1:ab:73:
             26:1f:0c:96:4a:a5:8a:e9:8e:33:83:6f:4f:ee:87:13:3c:25:
             0a:af:c3:f4
    
  2. Then here are the commands you would need to clear the certificate information on the controller
    
    openssl genrsa -out /cache/ssl/vasa.key 2048
    openssl req -new -key /cache/ssl/vasa.key -out /cache/ssl/vasa.csr -subj "/C=US/O=Pure Storage/OU=Pure Storage/CN=Pure Storage"
    openssl x509 -req -days 7300 -in /cache/ssl/vasa.csr -signkey /cache/ssl/vasa.key -out /cache/ssl/vasa.crt
    rm /cache/ssl/vasa.csr
    service nginx reload
    service vasa restart
    

    This would need to be done for both controllers.

  3. Then you can run the query in step one to confirm that the certificate is cleared to default.
  4. After the VASA Certificates have been reset the Storage Providers will need to be re-registered.
    1. Please follow theRegistering the FlashArray VASA Provider that works best for you.
  5. Sometimes even after re-registering the storage providers some ESXi hosts may not show connectivity to the vVol Datastore. This is because vCenter's certificate updates in VMCA may fail to dynamically push the updateded CA and CRLs to the ESXi hosts. In this event, you will need to refresh the CA/CRLs for the ESXi hosts that are mounted to the vVol DS. This can be done manually in the UI or with PowerShell.
    1. The process for doing this with PowerShell can be found here.

    Once you have your storage providers registered with the vCenters they were before, you can check to see that your vVol Datastore is accessible.

    If the hosts show the vVol datastore as inaccessible after re-registering the providers follow the steps in this KB to ensure the new certificate is passed properly to the host:

    Troubleshooting: vCenter Storage Provider Errors and Host Protocol Endpoint Connection Errors.

    Everpure's Virtual Volumes (vVols) implementation released in December of 2017. When registering the Flash Array storage providers the certificate is set to expire a year after the initial registration. Customers will want to renew the Certificates before they expire, but if they have expired they can still be renewed, but it will take some extra steps. Here are the steps to renew the storage provider certificate in both cases.