Splunk Enterprise software would not start and displays the message
# $SPLUNK_HOME/bin/splunk start
Splunk> The IT Search Engine.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
coldPath='/cold/splunk/_introspection/colddb' of index=_introspection on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
Performing locktest or locktool utilities would work fine but Splunk still fails to start.
Applies to
- Splunk Enterprise version between 8.1.0 and 8.1.2 with cold tier pointing to network filesystem (NFS) from FlashBlade mounted on the Splunk indexers.
- Both NFS v3 and NFS v4.1
Cause
This is a known Splunk issue introduced in Splunk 8.1.0. Splunk versions prior to 8.1.0 work fine.
|
Date filed |
Issue number |
Description |
|---|---|---|
|
2021-01-21 |
SPL-199608, SPL-200661 |
Indexer is not starting up because of locktest failure after upgrade to 8.1.0 Workaround: To startup the indexer, the workaround done was to set OPTIMISTIC_ABOUT_FILE_LOCKING=1 in $SPLUNK_HOME/etc/splunk-launch.conf to bypass locktest for Splunk to startup. Refer to: https://docs.splunk.com/Documentation/Splunk/8.1.0/Troubleshooting/FSLockingIssues
or upgrade to 8.1.3+ |
Resolution
Upgrade Splunk to 8.1.3 that has the filesystem locking issues fixed.
|
Date resolved |
Issue number |
Description |
|---|---|---|
|
2021-03-02 |
SPL-199608, SPL-200661 |
Indexer is not starting up because of locktest failure after upgrade to 8.1.0 |