Scale Out Configuration

Install the SAP HANA Scale Out system on both compute 1 and compute 2 (note compute 1 and compute 2 in this instance are separate and distinct groups of servers which match one another. If compute 1 is made up of 4 nodes (3 workers and 1 standby) then compute 2 needs to be made up of the same number of nodes and topology.

Use the same instance SID for both installations.
Use the same <sid>adm user password on both sets of systems.
The shared location (typically and NFS mount point) for the source and target systems are completely separate.
The global.ini file found at the following location - /hana/shared/<sid>/global/hdb/custom/config/global.ini - is the exact same on both sets of hosts.

Configure the data, catalog and log streaming backup location on both SAP HANA systems.

If using backint ensure that both sets of systems have access to the same backup set in the ISV.
If using a filesystem based backup ensure that either a shared filesystem or NFS mount point is used, ensure that any subfolders in the backup location are owned by the <sid>adm user.

Stop the SAP HANA instances on both sets of systems by running the following command on each of them:


/usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function StopSystem HDB

4. Wait until the instances have stopped on each system by using the following command:


/usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function GetSystemInstanceList

When all of the instances have stopped the output will be shown as below:


13.07.2020 03:16:42
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
shn1, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GRAY
shn4, 0, 50013, 50014, 0.3, HDB|HDB_STANDBY, GRAY
shn3, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GRAY
shn2, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GRAY

5. SAP HANA will unmount the log and data volumes when the systems are stopped, but sometimes this does not happen. It is worth checking if any volumes are still mounted on any nodes at this point. During installation, it is possible to either use a different set of volumes for each scale out landscape or the same volumes. If the former option is used then before going live with this solution one set of the volumes must be chosen, and the alternatives destroyed. Note that if using the latter method (the same volumes on both the source and target) then during installation it needs to be ensured that the original SAP HANA Scale Out cluster must be shut down and the volumes unmounted.

Note:

If using the alternative volumes method to install SAP HANA in the scale out the landscape, after deleting the alternative volumes the global.ini file on the source and target set of systems must be updated to reflect the WWID's of the permanent volumes to be kept.

6. Connect the arrays using synchronous replication, optionally add a third array for asynchronous replication.

7. On Array 1, In Protection->ActiveCluster Create a new POD, add the volumes to the POD and then add in Array 2 to stretch the pod. At this point the volumes will begin initial synchronization. Once Initial synchronization is complete both Arrays should as online.

8. At this point the volumes are online on both arrays, connect the host for compute 1 and compute 2 to the volumes in the POD on array 1 and array 2.

9. Ensure that the correct configuration for ActiveCluster DM-Multipath has been done on all nodes - find the configuration in the ActiveCluster Requirements and Best Practices.

10. If required, set the preferred paths for each host in FlashArray.

11. Start the SAP HANA Instance on Compute 1 using the following command:


/usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function StartSystem HDB

12. If required for third site availability add the POD to a Protection Group with Array 3 as the target.

Failover Process

In the event of losing a single array (array 1 or 2) there will be no impact on application availability as failover is transparent.

In the event of losing the compute upon which the SAP HANA instance is running at that point in time - the below process will need to be followed to bring the SAP HANA instance up on alternative compute.

Note:

If restoring service to compute 1 or compute 2 (systems attached to the synchronous replication volumes) then the volumes will already exist on the array and should be connected to the relevant host. If the SAP HANA instance is being failed over to a third site replicated to using asynchronous replication then the snapshots must first be restored to volumes and connected to the relevant host(s).

Step 1. Ensure the log and data volumes are not mounted on any of the source nodes.

Using the "df" command on each node:


df -h

Example of mounted SAP HANA data and log volumes; note the mount path in a Scale Out landscape using the Storage API Connector is in the form <basepath_datavolumes>/mnt0000#:


Filesystem                                     1K-blocks      Used Available Use% Mounted on
devtmpfs                                       264013600         0 264013600   0% /dev
tmpfs                                          397605660         4 397605656   1% /dev/shm
tmpfs                                          264021864     18840 264003024   1% /run
tmpfs                                          264021864         0 264021864   0% /sys/fs/cgroup
/dev/sdz2                                       62883840  14824852  48058988  24% /
Fileserver.puredoes.local:/mnt/nfs/SHN_Backup 1073485824 160157696 913328128  15% /hana/backup
Fileserver.puredoes.local:/mnt/nfs/SHN_Shared 1073485824 160157696 913328128  15% /hana/shared
tmpfs                                           52804372        24  52804348   1% /run/user/469
tmpfs                                           52804372         0  52804372   0% /run/user/468
tmpfs                                           52804372         0  52804372   0% /run/user/1001
tmpfs                                           52804372         0  52804372   0% /run/user/0
/dev/mapper/3624a9370884890ea83bd488200011c47  536608768   5362804 531245964   1% /hana/data/SH1/mnt00001
/dev/mapper/3624a9370884890ea83bd488200011c4a  402456576   5762904 396693672   2% /hana/log/SH1/mnt00001

Step 2. (Optional) verify that the global.ini configuration file on the source and target match. More specifically ensure that the [storage] section with the volume WWID's is exactly the same. Note the global.ini file can be located at /hana/shared/<SID>/global/hdb/custom/config/global.ini.

Source System


# global.ini last modified 2020-07-09 02:42:08.450368 by /usr/sap/SH1/HDB00/exe/hdbnsutil -initTopology --hostnameResolution=global --workergroup=default --set_user_system_pw
[communication]
listeninterface = .global

[multidb]
mode = multidb
database_isolation = low
singletenant = yes

[persistence]
basepath_datavolumes = /hana/data/SH1
basepath_logvolumes = /hana/log/SH1
basepath_shared = yes
use_mountpoints = yes

[storage]
ha_provider = hdb_ha.fcClient
partition_*_*__prtype = 5
partition_1_data__wwid = 3624a9370884890ea83bd488200011c47
partition_1_log__wwid = 3624a9370884890ea83bd488200011c4a
partition_2_data__wwid = 3624a9370884890ea83bd488200011c48
partition_2_log__wwid = 3624a9370884890ea83bd488200011c4b
partition_3_data__wwid = 3624a9370884890ea83bd488200011c49
partition_3_log__wwid = 3624a9370884890ea83bd488200011c4c

[trace]
ha_fcclient = info

Target System


# global.ini last modified 2020-07-09 03:09:58.950952 by /usr/sap/SH1/HDB00/exe/hdbnsutil -initTopology --hostnameResolution=global --workergroup=default --set_user_system_pw
[communication]
listeninterface = .global

[multidb]
mode = multidb
database_isolation = low
singletenant = yes

[persistence]
basepath_datavolumes = /hana/data/SH1
basepath_logvolumes = /hana/log/SH1
basepath_shared = yes
use_mountpoints = yes

[storage]
ha_provider = hdb_ha.fcClient
partition_*_*__prtype = 5
partition_1_data__wwid = 3624a9370884890ea83bd488200011c47
partition_1_log__wwid = 3624a9370884890ea83bd488200011c4a
partition_2_data__wwid = 3624a9370884890ea83bd488200011c48
partition_2_log__wwid = 3624a9370884890ea83bd488200011c4b
partition_3_data__wwid = 3624a9370884890ea83bd488200011c49
partition_3_log__wwid = 3624a9370884890ea83bd488200011c4c

[trace]
ha_fcclient = info

Step 3. As the <sid>adm user use the hdbnsutil to change the system name.

Note:

Note failover in this example is done as follows:


SHN1 , Worker Node -> SHN5, Worker Node
SHN2, Worker Node -> SHN6, Worker Node
SHN3, Worker Node -> SHN7, Worker Node
SHN4, Standby Node -> SHN8, Standby Node


hdbnsutil -convertTopology

A successful topology conversion will output the following:


nameserver shn5:30001 not responding.
checking 1 master lock file(s) ....................................... ok
load(/usr/sap/SH1/HDB00/exe/python_support/hdb_ha/fcClient.py)=1
attached device '/dev/mapper/3624a9370884890ea83bd488200011c47' to path '/hana/data/SH1/mnt00001'
attached device '/dev/mapper/3624a9370884890ea83bd488200011c4a' to path '/hana/log/SH1/mnt00001'
Opening persistence ...
sh1adm: no process found
hdbrsutil: no process found
run as transaction master
converting topology from cloned instance...
- keeping instance 00
- changing host shn1 to shn5
- changing host shn2 to shn6
- changing host shn3 to shn7
- changing host shn4 to shn8
- keeping DatabaseName SH1
detached device '/dev/mapper/3624a9370884890ea83bd488200011c47' from path '/hana/data/SH1/mnt00001'
detached device '/dev/mapper/3624a9370884890ea83bd488200011c4a' from path '/hana/log/SH1/mnt00001'
done.

Step 4. Start the SAP HANA system.

Use the sapcontrol utility to start the SAP HANA system.


/usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function StartSystem HDB

A successful start request will respond as follows:


09.07.2020 08:46:57
Start
OK

To check on the status of the startup process for SAP HANA use the sapcontrol command with the GetProcessList function.


 /usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function GetSystemInstanceList

Once the instances on each node have been started they should be displayed as "GREEN".


13.07.2020 04:33:51
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
shn7, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN
shn8, 0, 50013, 50014, 0.3, HDB|HDB_STANDBY, GREEN
shn5, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN
shn6, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN