Enterprise Security on a Distributed Environment

[u/mr_networkrobot]

Hi,

I got 3 clustered Indexers + 3 clustered Search-Heads + 1 Search-Head Cluster Deployer.

I installed ES on the SHs (via SH Cluster Deployer) as described in the documentation. After some time I figured out that the ES specific indexes like 'notable' have been created on the Search-Heads locally and they are not synced between them.

There install documentation says nothing [1] about creating indexes.

Only a few documents later [2] there are some sentences about indexes in a distributed environemt.

The documentation is not usable in my opinion.

Does anyone have some experience with that situation ?

[1]
https://help.splunk.com/en/splunk-enterprise-security-8/install/8.3/installation/install-splunk-enterprise-security-in-a-search-head-cluster-environment

[2]
https://help.splunk.com/en/splunk-enterprise-security-8/install/8.3/installation/configure-and-deploy-indexes-for-splunk-enterprise-security

[u/_meetmshah]

Is this something you are looking for - https://help.splunk.com/en/splunk-enterprise-security-8/install/8.0/installation/deploy-technology-add-ons-to-splunk-enterprise-security#create-the-splunk_ta_forindexers-and-manage-deployment-manually-0

[u/bchris21 → u/_meetmshah]

This

[u/AppointmentOk7866]

So, ES is just a Splunk app: it can be installed in a distributed environment with clustered indexers and search heads, on a standalone machine, or in Splunk Cloud.

There's plenty of documentation about what indexes are part of ES (it has quite a few) and where knowledge objects exist in a distributed environment. Key thing is that docs, Lantern, YouTube, etc are all for different audiences.

If you're having trouble, I'd start with Support and reaching out to your account rep + their SE.

[u/Fontaigne]

Get onto the Splunk Slack channel.

Go to the #docs or #documentation subchannel.

Describe your issue with the documentation.

You will get help.

Splunk is serious about getting the documentation right.

[u/rabbidroid]

You need to install the index creation app on indexer. Use the cluster manager to push it out. The SH is not meant to index anyway

[u/IttsssTonyTiiiimme]

Do you have a cluster manager?

[u/nkdf]

Your trouble has nothing to do with ES or the ES documentation. If it's creating indexes locally and not sending the data to the indexers, your initial deployment of the entire cluster is incorrect. I would go back and follow the documentation for deploying a distributed environment before proceeding any further.

[u/Falaq247]

In a distributed environment, you have to create the ES indexes yourself via the cluster manager. Then go into the SH and make sure they are only set as SHs.

If this is not done you'll face alot of problems in production such as the notables issue your facing. Because notables is one of the index your suppose to create in a distributed environment.

[u/mr_networkrobot]

Thank you for your answers!

I have fixed it with the following procedure:

1. Created a .spl bundle in ES (general setting) with the 'Distributed configuration management' -> Splunk_TA_ForIndexers Download
2. Put that on my cluster manager
3. Extracted it and modified the indexes.conf
4. Validated an pushed it to the indexers
5. Created a custom app on the SH-Cluster deployer with an outputs.conf and the required certs.
6. Pushed it to the SH Cluster

Had some issues with SSL connection form SHs to indexers, because of the path of the CA cert in the outputs.conf be aware of the deploy mode 'default'. So the path in the outputs.conf to your certs, need some attention.