Indexes customizations and management¶

Default indexes model¶

The application stack uses by default a 4 layer indexes models:

Index name	Purpose
os-unix-nmon-metrics	metric store index where metrics data is being stored
os-unix-nmon-events	nmon data stored as regular events
os-unix-nmon-config	nmon configuration data
os-unix-nmon-internal	internal data generated by the application for various processing tasks

indexes.conf¶

The indexes are defined in the Support Addon SA-metricator-for-nmon:

SA-metricator-for-nmon/default/indexes.conf

macros.conf¶

The front-end application metricator-for-nmon uses macro definitions for the indexes to be used by the application:

metricator-for-nmon/default/macros.conf

List of indexes macros:

macro name	indexes target
nmon_index	index=os-unix-nmon-*
nmon_idx	idx=os-unix-nmon-*
nmon_events_index	index=os-unix-nmon-events*
nmon_metrics_index	index=os-unix-nmon-metrics*

Wildcards are automatically replaced by any suite of characters.

eventtypes.conf¶

The front-end application metricator-for-nmon uses events definition in eventtypes.conf:

eventtype	indexes target
nmon	index=os-unix-nmon-events* OR index=os-unix-nmon-internal*
nmon:events	index=os-unix-nmon-events* sourcetype=nmon_data
nmon:config	index=os-unix-nmon-config* sourcetype=nmon_config
nmon:collect	index=os-unix-nmon-internal* sourcetype=nmon_collect
nmon:processing	index=os-unix-nmon-internal* sourcetype=nmon_processing
nmon:clean	index=os-unix-nmon-internal* sourcetype=nmon_clean
uptime	index=os-unix-nmon-events* sourcetype=nmon_data type=UPTIME
inventory	index=os-unix-nmon-config* sourcetype=nmon_config

inputs.conf¶

The Technical Addon contains a default definition of indexes to be used for data indexing:

TA-metricator-for-nmon/default/inputs.conf

With following inputs stanza specifying their target index:

[script://./bin/metricator_helper.sh]
index = os-unix-nmon-internal

[script://./bin/metricator_consumer.sh]
index = os-unix-nmon-internal

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*metrics.csv]
index = os-unix-nmon-metrics

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*events.csv]
index = os-unix-nmon-events

[monitor://$SPLUNK_HOME/var/log/metricator/var/config_repository/*events.csv]
index = os-unix-nmon-config

[script://./bin/metricator_cleaner.sh --cleancsv]
index = os-unix-nmon-internal

Customizing indexes names and multi-tenant configuration¶

For various reasons and purposes, you may need to customize the indexes names:

You have your own naming convention and you wish to be able to use for the application deployment as well
You are a large company with multiple regions across the globe and you want a geographical or per environment segregation
You want to use access control based indexes segregation to allow specific populations to access specific indexes
Any combination of these

The applications stack is compatible with any kind of customization.

Recommended customization scenario:

As we use a naming convention which matches indexes using wildcards by default, you are free to use any convention starting by the index names, such as:

Default Index name	Automatic target
os-unix-nmon-metrics	os-unix-nmon-metrics*
os-unix-nmon-events	os-unix-nmon-events*
os-unix-nmon-config	os-unix-nmon-config*
os-unix-nmon-internal	os-unix-nmon-internal*

Using our naming convention¶

If you are using our naming built-in naming convention then good news for you, you have absolutely nothing else to do than creating your own indexes.

Example of configuration: per environment segregation¶

Let’s take the example of having 3 environments where you are deploying the application stack:

Engineering (-eng)
UAT (-uat)
Production (-prod)

Because you may want to use a much higher retention for Production systems rather than Eng and UAT and have a larger period of metrics available, this is valuable to split the ingestion into multiple-indexes stacks.

You proceed the following way:

For each of your indexer layer, prepare the stack of indexes and apply your custom indexes
Create a local version of the inputs.conf shipped with the Technical Addon and publish it to the instances managed by the deployment servers
Respect Splunk best practices in every step!

Based on the example above::

Indexing layer preparation:¶

Production configuration, assuming clustered indexers.

On cluster master:

cd /opt/splunk/etc/master-apps/SA-metricator-for-nmon
if [ ! -d local ]; then mkdir local; fi
cp -p default/indexes.conf local/

Then achieve the following configuration, adapt if using Splunk volumes or for any other need:

# nmon data ingested as metrics
[os-unix-nmon-metrics-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-metrics-prod/colddb
datatype = metric
homePath = $SPLUNK_DB/os-unix-nmon-metrics-prod/db
splitByIndexKeys = metric_name,host
thawedPath = $SPLUNK_DB/os-unix-nmon-metrics-prod/thaweddb
repFactor = auto

# nmon data ingested as regular events
[os-unix-nmon-events-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-events-prod/colddb
homePath = $SPLUNK_DB/os-unix-nmon-events-prod/db
thawedPath = $SPLUNK_DB/os-unix-nmon-events-prod/thaweddb
repFactor = auto

# nmon config ingested as regular events
[os-unix-nmon-config-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-config-prod/colddb
homePath = $SPLUNK_DB/os-unix-nmon-config-prod/db
thawedPath = $SPLUNK_DB/os-unix-nmon-config-prod/thaweddb
repFactor = auto

# nmon internal data
[os-unix-nmon-internal-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-internal-prod/colddb
homePath = $SPLUNK_DB/os-unix-nmon-internal-prod/db
thawedPath = $SPLUNK_DB/os-unix-nmon-internal-prod/thaweddb
repFactor = auto

Publish the cluster bundle

Technical Addon configuration¶

On Deployment Servers:

cd /opt/splunk/etc/deployment-apps/TA-metricator-for-nmon
if [ ! -d local ]; then mkdir local; fi

Add the following configuration to local/inputs.conf (create the file)

[script://./bin/metricator_helper.sh]
index = os-unix-nmon-internal-prod

[script://./bin/metricator_consumer.sh]
index = os-unix-nmon-internal-prod

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*metrics.csv]
index = os-unix-nmon-metrics-prod

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*events.csv]
index = os-unix-nmon-events-prod

[monitor://$SPLUNK_HOME/var/log/metricator/var/config_repository/*events.csv]
index = os-unix-nmon-config-prod

[script://./bin/metricator_cleaner.sh --cleancsv]
index = os-unix-nmon-internal-prod

Achieve the same configuration on any server running the TA-metricator-for-nmon and not managed by Deployment Servers (cluster master, Deployment Servers…)

Reload or restart the Deployment Servers to publish this new configuration to managed servers

And finally repeat the operation for each environment that needs to be managed.

It is not required to achieve any modification on the front-end user application since the default configuration takes in charge any index respecting the naming convention using wildcards definitions.

Using a different naming convention¶

If the builtin indexes naming convention is not option for you, it is still possible to customize the different applications to match your needs.

However for easy of use and maintenance, we would recommend using the builtin naming convention when possible to avoid additional steps in your deployments.

You will proceed as following:

Customize indexes in local/indexes.conf (Indexer layer in SA-metricator-for-nmon)
Customize macros in local/macros.conf (Search Head layer in metricator-for-nmon)
Customize eventtypes in local/eventtypes.conf (Search Head layer in metricator-for-nmon)
Customize inputs.conf in local/inputs.conf (Technical Addon in TA-metricator-for-nmon)