Routing: IP Routes to FlashBlade

IP routing to the FlashBlade array is determined by the network configuration at the DGX and the EVPN route configuration in the storage fabric.

At the DGX routes are defined based on the subnet and IP assigned to the storage fabric NIC port that will be used for data movement. EVPN routing will forward the packets to the FlashBlade endpoint.

Figure 1. NVIDIA DGX network configuration example

Route Configuration

Routes from each DGX should be defined as follows:

sudo ip route add 100.127.199.0/25 via 100.127.0.36 dev enp225s0f0np0 src 100.127.0.37

 sudo ip route add 100.127.199.0/25 via 100.127.128.37 dev enp97s0f0np0 src 100.127.128.37

Note: EVPN configuration is beyond the scope of this document.

FlashBlade and PAC Deployment Completion

After the NVIDIA DGX nodes have been rebooted and successfully PXE booted into the new OS and the routing has been configured on the NVIDIA DGS nodes, deployment is complete. The next steps are for validation purposes.

Validate PAC Install

Note: Validation steps listed below are dependent on a mountable file system being created on the FlashBlade prior to attempting. Please review and complete the Site Preparation and Physical Installation section above as well as the Configuring FlashBlade: Multiple DataVip Creation section are completed before attempting to validate the mount to a FlashBlade.

IP’s used in this example may change depending on install requirements

From the BCM head node, ssh into one of the compute nodes to validate install:

Checking the DMKS status should show the DKMS module loaded.
ssh node003 
root@node005:~# sudo dkms status
multipath-nfs/99.2-4.0.28, 5.19.0-45-generic, x86_64: installed

Mounting the FlashBlade

The FlashBlade is mounted automatically via a service controlled by NVIDIA called the system-roce-storage-startup.service. This service ensures that the FlashBlade mount is properly configured for each DGX. The mount in the system-roce-storage-startup.service is using the following syntax:

sudo mount -o vers=3,proto=rdma,remoteports=100.127.199.4~100.127.199.5,nconnect=16 100.127.199.4:/purefb /mnt/purefb

After the NVIDIA DGX nodes boot the service will run and automatically make sure the appropriate local mount directory is created as well as execute the mount to connect to the FlashBlade file system created in the Create Filesystem section.

Validate FlashBlade Mount from BCM Head Node

Once the NVIDIA DGX nodes have rebooted and the Validate PAC Install steps have been checked you can then validate the mounts exist on each NVIDIA DGX. The number of nodes in the ‘pdsh’ command will vary, in the example below we are showing 24.

To spot check a single NVIDIA DGX node ssh into a the node from a head node and run:

mount |grep purefb

You should see an existing mount with that ‘purefb’ name in the mount path.

Example:

root@dgx-01:~# mount |grep purefb
100.127.199.4:/purefb on /mnt/purefb type nfs (rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,forcerdirplus,proto=rdma,nconnect=16,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=100.127.199.4,mountvers=3,mountproto=tcp,local_lock=none,addr=100.127.199.4)
pdsh -w node[001-024] sudo mount |grep /mnt/purefb2

As an alternate larger check you can run the following from a BCM head node ssh session.

pdsh -w node[001-024] sudo mount |grep /mnt/purefb |wc

The node count and mount count from |wc should be the same.

To Remove Pure Aggregate Client, if needed

From a root ssh session on a BCM head node, run:

cm-chroot-sw-img /cm/images/<clonedsoftwareimagename>
sudo dpkg –remove  multipath-nfs/99.2-4.0.28

Exit the cm-chroot session:

'ctrl-d'