A reprotect operation automates the replication of failed over volumes back to the original FlashArray. In order to run reprotect, a fully successful recovery must occur. If the disaster recovery operation was executed and skipped multiple steps due to failures, a reprotect might not be possible. In this case manual reprotection might be required-which is essentially the same process as setting up replication for the first time. If a reprotect fails, try again with the Force Cleanup option checked (it will only appear if a reprotect has failed once).
Once a recovery operation has completed, it is recommended to run the reprotect as soon as possible. This will ensure the data being generated on the now production site is being protected.
The reprotect operation does the following things:
- Sets up replication on the FlashArray(s) that are now running the VMs
- Reverses the SRM protection groups and recovery plans
- Initiates a synchronization of the data
To start a reprotect, click on the desired SRM recovery plan and click Reprotect.
Confirm the action and click Next, then Finish.
Prior to reprotect, if the original pod was stretched, it must be unstretched. This is due to the fact that protection groups cannot replicate back into a pod and volumes cannot be moved into a stretched pod. In order to make sure that a failback will work, the SRA ensures that the original pod has been unstretched first. If you did not unstretch it, you will see the following error in SRM:
It is important to understand that the SRA will never unstretch or stretch a pod back for the environment- this must be done manually.
The pod can be unstretched from either array-just ensure that the array is properly configured as a target in the SRM array managers.
The FlashArray will then look for the FlashArray protection group or groups of the source volume(s).
The process follows these rules:
- The SRA will look for the original source volume on the pod; if it cannot find it, the SRA will setup a new protection group on the asynchronous target FlashArray-see below for details. The source volume is identified via the volume name-so if it has been renamed on the source the lookup will fail
- Protection groups on the pod that include the original source volume will be created on the asynchronous target FlashArray even if they do not have replication enabled and/or replicate to a different array
- Protection group name matching is not case-sensitive. If the pod has a protection group called srm-pg and the asynchronous target FlashArray has one called SRM-PG, they will be considered the same and no new group will be created
- For protection groups created by the SRA, it will match the replication and local snapshot policy
- Protection groups that are created by the SRA during reprotect will only add the original FlashArray as a replication target-no matter how many targets were in the original protection group. If you would like the created protection group to also replicate to other arrays, add them manually as targets later
- If the original source volume in the pod is in more than one protection group, the SRA will re-create all protection groups on the asynchronous target FlashArray
- If a protection group with the name already exists on the asynchronous target FlashArray, the SRA will not re-create the group and will put the volume in that identified group
- If a pre-existing protection group policy name does not match the original source protection group policy, it will still be used. The SRA will not update the identified pre-existing protection group with the policy on the original array
Once the protection has been re-created on the now production side, the original source volume(s) will be removed from protection groups that replicate to the now production array. If the volume is in protection groups that do not replicate to the now production array, the volume will be left in those.
The SRA will re-create (or re-use) the protection groups on the now production FlashArray. Newly-created (and potentially pre-existing) protection groups will be replicating back to the FlashArray where the pod currently exists. It is recommended to manually add the 2nd FlashArray (the FlashArray which the pod was unstretched from prior to reprotect) to these protection groups as well as a replication target.
The pod was previously on flasharray-50-1 and flasharray-m50-2.
And the pod was unstretched to only exist on flasharray-m50-1.
The protection group that was created on the target (now production) FlashArray only will have the array that hosts the pod currently, in this case flasharray-m50-1.
It is recommended to manually add flasharray-m50-2 as a target as well.
The reason for this recommendation is that if the 3rd array is replicating snapshots back to both arrays, both arrays will be seeded with the data. So when the original pod is stretched back to the 2nd array (in this case flasharray-m50-2) it will take much less time as most of the data is already there via periodic "seeding" from the protection group. Note that this is a recommendation, not a requirement.
If the SRA cannot find the original source volume (and therefore cannot identify the correct protection groups), the SRA will instead create a protection group with the default replication policy with the name PureSRADefaultProtectionGroup replicating back to the original array. This protection group can be edited and changed (or even removed) as desired afterwards; ensure that the desired volumes are still replicated by a protection group.
The reprotect operation will then complete swapping directions of the SRM objects and reconfiguring protection of the virtual machines in the recovery plan.
The reprotect operation will rename the volume on the target array to remove the applied suffix of -puresra-failover.
If failures occur, the Force Cleanup option will become available-it is highly encouraged to not resort to that option immediately. It is advised to attempt figure out and resolve the underlying problem and re-run the reprotect without Force Cleanup selected until successful.
The main reasons that a reprotect could fail are often in the VMware environment (placeholders aren't there, mappings are incorrect or missing, etc). Though some FlashArray failures can cause this too:
- Replication connection does not exist back to the array. If so, re-create this connection
- Array managers are configured incorrectly
- Original source volume was renamed. It should be in the form of <volume name>-puresra-demoted. If it is not, rename it back. You will see an error like: SRA command 'prepareReverseReplication' failed for device '<volume name>'. Cannot find the volume <volume name> on the array <arrayname> Please make sure that replication setup is correct". If it fails in the prepareReverseReplication phase, likely the source volume has been renamed manually or destroyed. Either rename it back, or if it is gone, run an SRM SRM User Guide: Discovering Replicated Devices with the FlashArray SRA and then re-attempt the reprotect without force cleanup checked. Only if that fails, should you then retry with force cleanup checked
- The source pod was renamed. If so, rename it back to the original name, or remove the SRM protection group and create a new one
- The pod is stretched across two arrays. Unstretch it from one of the arrays. If you do not want to do so, remove the SRM protection group and create a new one
- Target volume was renamed. It should be in the form of <volume name>-puresra-failover. If it is not, rename it back. You will see an error like: SRA command 'reverseReplication' failed for device '<volume name>'''. Cannot find the volume '<volume name>' on the array <arrayname>
- If this is the case, fix the name of the volume AND ensure that the original volume is still in a protection group replicating to the target. The reprotect operation will have removed it from any replication groups at this stage causing device discovery to fail
- A volume exists with the original name but no suffix. If the original volume was srm-DS1 and it was failed over to the 2nd array, the failover volume will be called srm-DS1-puresra-failover. If there is a volume on the failover array called srm-DS1 already the initial recovery will not fail, but the reprotect will as it will try to rename the volume from srm-DS1-puresra-failover to srm-DS1 which will fail because there is already a volume with that name. While this is unlikely to occur, it could. You will see an error upon reprotect like: Failed to reverse replication for device 'peer-of-53f027a7-828d-4b4d-a3a8-d4b2c8364507:srmDS-08'. SRA command 'reverseReplication' failed for device 'peer-of-53f027a7-828d-4b4d-a3a8-d4b2c8364507:srmDS-08'. Could not rename failed over volume srmDS-08 to peer-of-53f027a7-828d-4b4d-a3a8-d4b2c8364507:srmDS-08 on array flasharray-m20-1. Note that the volume might be in the destroyed volume folder awaiting eradication if it is not visible
Re-creating a Destroyed Pod Prior to a Failback
If prior to a failback an error is encountered in array discovery such as the following.
One of two things have likely happened:
-
The pod was renamed. In this case, rename it back, or create a new pod with the correct name
In the case of the first situation, follow these instructions:
You should re-create the pod with that name (in this case srmPod) on any array configured in an array manager that has a replication connection back to the current production array.
Once re-created, array discovery will succeed.
A failback to the re-created pod can now occur and the recovered volumes will be created in the pod.