Splunk Multisite SmartStore (Active/Active) Architecture

Splunk Multisite SmartStore is generally deployed to meet the disaster recovery requirements. Splunk Multisite SmartStore deployment on premises is limited to two sites, with each site hosted in an on-premises data center, in an active-active mode. This means, input data can be ingested through both the sites.

Prerequisites

1) To host the Splunk Multisite data on FlashBlade, you would need two FlashBlades, each located in an on-premises data center with the minimum Purity//FB version of 3.3.3 that offers the "multi-site-writable" bucket option. This feature includes all the object-related functionalities that are required to host Splunk SmartStore index data on FlashBlades using S3 protocol as well as the object replication requirements as laid out by Splunk in their documentation.

Please see the How to setup Splunk Multisite SmartStore with FlashBlade to setup the FlashBlades for Splunk Multisite SmartStore.

2) Third-party VIP or Global Server Load Balancing (GSLB) to route the traffic from the peer nodes to the object store (FlashBlade) hosted on its site's location. In case of a site failure or the local FlashBlade failure, the VIP or GSLB should be able to reroute the peer's traffic as necessary to the FlashBlade in the other site.

The remote.s3.endpoint refers to the FlashBlade data vip that is configured through the GSLB and hence should refer a URI instead of the hard coded IP address.

3) The bucket level versioning on both the FlashBlades should be enabled and the Splunk indexes.conf parameter remote.s3.supports_versioning should be set to false. Not setting either one of them can cause object counts mismatch between the two FlashBlades.

Please see Splunk's documentation for the Splunk Multisite SmartStore deployment requirements on premises.

Remote Volume

The volume definition for the remote storage in indexes.conf points to the remote object store where Splunk SmartStore stores the warm data. The remote volume definition looks like the following.

[volume:remote_store]
storageType = remote
# The bucket name should be same across both FlashBlades
path = s3://<bucket name>  

# The access_key and secret_key should be same across both FlashBlades 
remote.s3.access_key = <access key of the account that holds the bucket>
remote.s3.secret_key = <secret key of the account that holds the bucket>

# Make sure to include the Object store URI through third-party VIP or GSLB
remote.s3.endpoint = http://<FlashBlade-URI>

remote.s3.supports_versioning = false 
remote.s3.list_objects_version = v2

[splunk_index]
remotePath = volume:remote_store/$_index_name
repFactor = auto
homePath = <home path specification>

• As the same indexes.conf file is deployed across the two sites, the parameters under the volume:remote_store are applicable to both the FlashBlades.
  • The S3 bucket name as referred by the path parameter should be the same on both the FlashBlades.
  • The access_key, secret_key should be the same on both the FlashBlades.
• The remote.s3.endpoint should point to the FlashBlade URI through a third-party VIP (Virtual IP) or GSLB (Global Server Load Balancing).
  • The VIP or GSLB routes traffic from the Splunk indexer nodes to the FlashBlade hosted on its site's location. In case of a FlashBlade failure, the VIP or GSLB reroutes the traffic as necessary to the remaining active FlashBlade.
  • Using the IP address from the FlashBlade Data vip will not work with the Multisite.
• Make sure to set the parameter remote.s3.supports_versioning is set to false and the bucket level versioning is enabled on both the FlashBlades.

Please see Splunk's documentation for more details on the deployment topology.

Replication Lag

Based on the distance between the two FlashBlades, there will be a replication lag time between them when the FlashBlades replicate the objects asynchronously. To eliminate the peer indexer on the other site to upload its copy prior to the completion of the object store replication, the following parameter should be updated.

remote_storage_upload_timeout

The parameter remote_storage_upload_timeout under [clustering] stanza on the server.conf for all the indexers across the site should be set to a time (in seconds) higher than the maximum replication lag time between the two FlashBlades.

Recommended setting:

remote_storage_upload_timeout = 600

Note: This setting should be updated to a higher value if you notice the replication lag goes beyond 600 seconds. If the latency between two FlashBlades is over 200ms, there is a good chance the 600 seconds might not be sufficient. Please review and update accordingly.