NVIDIA DGX SuperPOD environments utilize a single mount point from DGX hosts to a storage endpoint. Pure has provided a client side .deb file install to accommodate this capability. The Pure Aggregate Client (PAC) distributes traffic between two initiator ports to fully utilize the SuperPOD storage fabric connections per compute node. Below are the steps to install the PAC.
Note:
It is considered best practice to clone a category and software image in BCM and install the PAC into the cloned software image. This image can then be assigned to a cloned category for validation and testing.
Ensure the software image that has been cloned has the appropriate Nvidia OFED drivers installed. Cloning the default dgx-os-6.1-h100-image will provide the base image to work with but will require the NVIDIA OFED driver version appropriate for the install to be added. For details on the proper installation of Nvidia OFED drivers see the following link and reference the Nvidia OFED driver install options.
Install Pure Aggregate Client (PAC) Steps:
-
Download PAC (Pure Aggregate Client)
Download the PAC (Pure Aggregate Client) .deb installation file from here and copy it to a BCM head node.
-
Copy the .deb installation file into BCM a cloned software image on a head node
cp ./multipath-nfs-dkms_4.0.28-purestorage_all.deb /cm/images/<clonedsoftwareimagename>/Example:cp multipath-nfs-dkms_4.0.28-purestorage_all.deb / cm/images/pure-dgx-os-6.2-h100-image/ -
From a BCM head node, run
cmsh -c 'softwareimage use <softwareimagename>; get kernelversion'Example:cmsh -c 'softwareimage use pure-dgx-os-6.2-h100-image; get kernelversion' -
From a root ssh session on a BCM head node, run
cm-chroot-sw-img /cm/images/<clonedsoftwareimagename>Example:cm-chroot-sw-img cm/images/pure-dgx-os-6.2-h100-image/ mounted /cm/images/pure-dgx-os-6.2-h100- image/dev mounted /cm/images/pure-dgx-os-6.2-h100-image/dev/pts mounted /cm/images/pure-dgx-os-6.2-h100-image/proc mounted /cm/images/pure-dgx-os-6.2-h100-image/sys mounted /cm/images/pure-dgx-os-6.2-h100-image/run mounted /run/systemd/resolve/stub-resolv.conf -> /cm/images/pure-dgx-os-6.2-100-image/run/systemd/resolve/resolv. conf Using chroot with mounted virtual filesystems to chroot in /cm/images/pure-dgx-os-6.2-h100- image. ... Type 'exit' or ctrl-D to exit from the chroot in the software image. This also unmounts the above mentioned /dev /dev/pts /proc /sys /run filesystems in the software image. -
From within the cm-chroot-sw-img session, run
sudo apt install ./multipath-nfs-dkms_4.0.28-purestorage_all.debNote: Click ‘ok’ at kernel message pop-up, if seen.Example:sudo apt install ./multipath-nfs-dkms_4.0.28-purestorage_all.deb The following packages were automatically installed and are no longer required: python3-automat python3-bcrypt python3-constantly python3-hamcrest python3-hyperlink python3-incremental python3-pyasni python3-pyasn1-modules python3-service-identity python3-twisted python3-zope.interface Use 'sudo apt autoremove' to remove them. The following NEW packages will be installed: multipath-nfs-dkms 0 upgraded, 1 newly installed, 0 to remove and 235 not upgraded. Need to get 0 B/963 kB of archives. After this operation, 6,093 kB of additional disk space will be used. Get:1 /multipath-nfs-dkms_4.0.28-purestorage_all.deb multipath-nfs-dkms all 2:4.0.28-purestorage [963 kB] Selecting previously unselected package multipath-nfs-dkms. (Reading database ... 223753 files and directories currently installed.) Preparing to unpack .../multipath-nfs-dkms_4.0.28-purestorage_all.deb ... Unpacking multipath-nfs-dkms (2:4.0.28-purestorage) Setting up multipath-nfs-dkms (2:4.0.28-purestorage) Created symlink /etc/systemd/system/multi-user.target.wants/multipath-nfs-dkms.service → /lib/systemd/system/multipath-nfs-dkms. service. Scanningprocesses.•• Scanning processor microcode... Scanning linux images... The processor microcode seems to be up-to-date. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. -
After the .deb install finishes, run
dkms build multipath-nfs/99.2-4.0.28 -k <kernel version from step 3.>Example:dkms build multipath-nfs/99.2-4.0.28 -k 5.15.0-1053-nvidia Creating symlink /var/lib/dkms/multipath-nfs/99.2-4.0.28/source - /usr/src/multipath-nfs-99.2-4.0.28 Kernel preparation unnecessary for this kernel. Skipping... Building module: cleaning build area.... •/_dkms-run.sh -j16 KVER=5.15.0-1053-nvidia K_BUILD=/lib/modules/5.15.0-1053-nvidia/build................... Signing module: - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/rpcrdma. ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfsd.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/sunrpc.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfs_layout_nsv41_files. ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/blocklayoutdriver. ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfs.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfsv3.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfsv2. ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfsv4.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfs_layout_flexfiles.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/nfs_acl. ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/lockd. ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/auth_rpcgss.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/rpcsec_gss_krb5.ko - /var/lib/dkms/multipath-nfs/99.2-4.0.28/5.15.0-1053-nvidia/x86_64/module/compat_nfs_ssc.ko EFI variables are not supported on this system /sys/firmware/efi/efivars not found, aborting. cleaning build area.... -
After the build completes, run
dkms install multipath-nfs/99.2-4.0.28 -k <kernel version from step 3.>Example:dkms install multipath-nfs/99.2-4.0.28 -k 5.15.0-1053-nvidia Forcing installation of multipath-nfs rperdma. ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfsd.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ sunrpc.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfs_layout_nfsv41_files.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ blocklayoutdriver.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfs.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfsv3.ko: Running module version sanity check. -Originalmodule - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfsv2.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfsv4. ko: Running module version sanity check. -Originalmodule - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfs_layout_flexfiles.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ nfs_acl. ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ lockd. ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ auth_rpgss. ko: Running module version sanity check. -Originalmodule - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ rpcsec_gss_krbs.ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ compat_nfs_ssc. ko: Running module version sanity check. - Original module - Installation - Installing to /lib/modules/5.15.0-1053-nvidia/updates/dkms/ depmod.... -
After the install completes, exit the cm-chroot-sw-img session
'ctrl-d'Example:'ctrl-d' exit unmounted /cm/images/pure-dgx-os-6.2-h100-image/run/systemd/resolve/resolv.conf unmounted /cm/images/pure-dgx-os-6.2-h100-image/dev/pts unmounted /cm/images/pure-dgx-os-6.2-h100-image/dev unmounted /cm/images/pure-dgx-os-6.2-h100-image/proc unmounted /cm/images/pure-dgx-os-6.2-h100-image/sys unmounted /cm/images/pure-dgx-os-6.2-h100-image/run -
From the head node ssh session, create new ramdisk for the PAC (Pure Aggregate Client) image
Example:cmsh [bch1]% softwareimage [bch1->softwareimage]% use pure-dgx-os-6.2-h100-image [bch1->softwareimage [pure-dgx-os-6.2-h100-image]]% createramdisk [bch1->softwareimage [pure-dgx-os-6.2-h100-image]]% Sat Oct 12 13:05:07 2024 [notice] bchl: Initial ramdisk for image pure-dgx-os-6.2-h100-image is being generated [bch1->softwareimage [pure-dgx-os-6.2-h100-image]]% Sat Oct 12 13:05:59 2024 [notice] bchl: Initial ramdisk for image pure-dgx-os-6.2-h100-image was generated successfully [bmh1->softwareimage[pure-dgx-os-6.2-h100-image]]% -
Reboot Nodes
From within CMSH, add nodes to the category that has the modified software image above and reboot the nodes or reboot nodes already in the category.
Note: Ensure the software image that has been cloned and updated with the PAC is appropriately assigned to the cloned category before rebooting any nodes in the category.