An inaccessible can be attributed to multiple potential causes as denoted in the first section of this KB. This section of the KB will provide suggested commands to run and components between the AVS and Everpure Cloud Dedicated integration to check to confirm connectivity, iSCSI path status and proper AVS ESXi host membership on the Everpure Cloud Dedicated array, as well as methods to initialize AVS hosts and rescan host HBAs.
Solution 1: Confirm AVS ESXi host membership is accurate on Everpure Cloud Dedicated
AVS ESXi hosts can be automatically replaced for numerous reasons, including ESXi code upgrades or if a hardware failure occurs. Customer-driven actions such as addition or removal of ESXi hosts from an existing AVS cluster can also account for host membership changes. It is for this reason that we strongly recommend deploying our host lifecycle management functionality as this will automatically check for AVS ESXi cluster membership updates and if one is detected, will automatically update both Everpure Cloud Dedicated as well as the AVS cluster to account for the new host or hosts.
To check that host membership is up to date, login to the Everpure Cloud Dedicated GUI, navigate to the impacted host group and review the AVS ESXi hosts listed. Compare this listing to what is shown in the AVS vCenter, as is depicted in the below screenshot.
In this example, we can see that there is a host mismatch between the AVS Cluster-1 ESXi host membership and what is shown in Everpure Cloud Dedicated. This often leads to the VMFS datastore or datastores utilized by that cluster not being available for the mismatched host.
If there is a discrepancy as shown above, proceed to Solution 5 and re-run the Storage Management | Azure VMware Solution and Everpure Cloud Dedicated against the impacted cluster and optionally deploy automated AVS host lifecycle management. The build-pcbscluster command will automatically update the Everpure Cloud Dedicated host group membership with the correct AVS ESXi hosts, configure SW iSCSI, populate the host object on Everpure Cloud Dedicated with the proper iSCSI IQN and lastly rescan the HBAs on all hosts within that cluster.
Solution 2: Confirm desired AVS VMFS Volume is connected to the AVS host group in Everpure Cloud Dedicated GUI
A simple, but important item to check is to confirm that the VMFS datastore you want to utilize with an AVS cluster is connected to the correct Everpure Cloud Dedicated host group. The new-pcbsvmfsdatastore command will automatically create the volume and connect it to the specified host group for Everpure Cloud Dedicated, however, it is possible for the volume to be manually disconnected inside of Everpure Cloud Dedicated by a storage administrator so it is important to confirm volume connectivity if there are accessibility issues.
To confirm, login to the Everpure Cloud Dedicated GUI, navigate to Storage > Hosts > Hosts Groups and confirm that the desired VMFS volume or volumes are connected to the correct AVS cluster.
If the VMFS volume has been disconnected from the desired host group, it can be reconnected by clicking on the three ellipses under Connected Volumes and selecting Connect Volumes. Note that if the volume has not previously been formatted with VMFS 6, then the datastore will not appear within AVS. For new VMFS volumes, the new-pbcsvmfsdatastore run command needs to be executed or the restore-pcbsvmfsdatastore can also be run to copy a VMFS snapshot to a volume and automatically connect it to the AVS cluster of your choosing. All commands shown in this section are detailed in the Storage Management | Azure VMware Solution and Everpure Cloud Dedicated of the AVS and Everpure Cloud Dedicated Implementation guide.
Solution 3: Confirm AVS ExpressRoute is connected to Everpure Cloud Dedicated for Azure vNet via Virtual Network Gateway
AVS and Everpure Cloud Dedicated network connectivity is accomplished via a virtual network gateway that is connected to the Azure vNet that Everpure Cloud Dedicated utilizes. If that connection is deleted or becomes otherwise unavailable, iSCSI traffic between AVS and Everpure Cloud Dedicated will be disrupted.
To confirm connectivity, from the Azure portal, navigate to the Ultra Virtual Network gateway item utilized for AVS and Everpure Cloud Dedicated connectivity. Click on 'Connections' and verify that the ExpressRoute connection is in a 'Succeeded' state. This scenario is shown in the below screenshot.
If no connection to your AVS cluster is shown, or it is not in a Succeeded state, refer to the Deploying an Azure Ultra/ErGw3AZ SKU Virtual Network Gateway section of the Implementation Guide to re-establish an connection between the two solutions. If you receive an error message when attempting to fix or establish the ExpressRoute connection, open a ticket with Azure support to troubleshoot.
Solution 4: Verify path connectivity between AVS and Everpure Cloud Dedicated
Data path connectivity can be verified from both the Azure VMware Solution vCenter instance as well as Everpure Cloud Dedicated.
To view path connectivity from Everpure Cloud Dedicated, login to the Everpure Cloud Dedicated GUI and navigate to Health > Connections. You can optionally sort by 'esx' to only show AVS ESXi hosts.
In this example below, we see that two hosts do not have active paths to Azure VMware Solution, while one does.
From the AVS vCenter click on the ESXi host of interest. Select Configure >Storage Adapters > vmhba64 / iSCSI Software Adapter > Paths and observe the path status listed. The below screenshot depicts a scenario where the paths between the AVS ESXi host and the Everpure Cloud Dedicated instance for all LUNs are down/dead.
If paths are down as shown above, proceed to solution 5 and solution 6 below to attempt to remediate. If solution 5 and solution 6 fail, open a support request with the Azure VMware Solution team and specify that iSCSI paths between AVS and Everpure Cloud Dedicated appear to be down and need to be looked at in more detail from the AVS ESXi hosts.
Solution 5: Initialize AVS hosts that have been replaced in AVS cluster and deploy AVS Host Lifecycle Management
If AVS Host Lifecycle Management is not in place, then the build-pscdcluster command must be manually executed against an AVS cluster anytime that the ESXi host membership is updated. The build command iterates through a cluster and if it has already been executed against an existing AVS ESXi hosts then it is skipped. This build command is required as any new ESXi host or hosts introduced to the cluster by default do not have the software iSCSI adapter enabled and Everpure Cloud Dedicated does not have the host objects with their IQNs listed. Instructions for executing the build-pcbscluster command can be found in the Storage Management | Azure VMware Solution and Everpure Cloud Dedicated section of the Implementation guide.
Solution 6: Rescan AVS ESXi host storage
The VMFS Run Command library in the AVS Azure portal item allows AVS administrators to manually perform a rescan of all ESXi host cluster HBAs to detect new VMFS data volumes connected to the cluster. How to access the Sync-ClusterVMHostStorage run command is detailed in the below screenshot.
For the ClusterName field (item 4), make sure to enter the AVS ESXi cluster that you want to be scanned. That value can be Cluster-1, Cluster-2 and so on.
After clicking Run, the execution status of the run command can be tracked in the 'Run Execution Status' tab within the Run Command Azure blade.