In today’s article about Splunk monitoring we want to monitor the version of the Splunk components which are connecting to Forwarder Management. The former name of Forwarder Management was Deployment Server – which I personally prefer more as it not only configures and manages your Forwarders, but all Splunk components including Indexers and Search Heads.
Unfortunately the Forwarder Mangement WebGUI only displays the OS platform of the client but not the Software version.
With the Nagios plugin we want ensure that all your clients (Indexers, Search Heads, Forwarders) are running at least the defined Splunk version. Otherwise an error will be generated.
You need to have Forwarder Management implemented for this check. On the client you just need to point it to one Forwarder Management server, which can be any Splunk server in your environment.
You can set the Forwarder to the Forwarder Management server with the command
$SPLUNK_HOME$/bin/splunk set deploy-poll bd20.bwlab.loc:8089
and check using
$SPLUNK_HOME%/bin/splunk list deploy-poll
The Nagios plugin queries Forwarder Management for the client list and compares every client against a minimum build level you can define. The plugin is a Powershell script communicating with the REST API of Splunk. For that reason the script has to be executed from a Windows device. That does not meanthe Splunk instance running the Forwarder Management role has to be installed on the Windows machine. If yourun Splunk on Linux or Mac you just need a Windows machine in your environment which executes the script against the non-Windows Splunk instance.
Setup monitoring using nsclient++ on Windows
In this example the Forwarder Management server runs on the same machine like the Nagios client nsclient++. If you need to do an indirect query, because your Splunk server is running on a non-Windows machine simply adjust the IP in the Nagios service definition.Download and extract the files to C:\Program Files\NSClient++\scripts\splunk
- Download and extract the files to C:\Program Files\NSClient++\scripts\splunk
- Adjust your “C:\Program Files\NSClient++\nsclient.ini” and add the external script
splunkfwmanagementversion = cmd /c echo scripts\\splunk\\check-deploymentclientsversion.ps1 -servername $ARG1$ -username $ARG2$ -password $ARG3$ -minbuild $ARG4$; exit($lastexitcode) | powershell.exe -command –
- 3. On the Nagios server: create a new command using NRPE
# ‘nt_nrpe_splunkfwmanagementversion’ command definition
command_line /usr/lib/nagios/plugins/check_nrpe -t 30 -H $HOSTADDRESS$ -p 5666 -c splunkfwmanagementversion -a $ARG1$ $ARG2$ $ARG3$ $ARG4$
- 4. On the Nagios server: add a service to your host definition
use generic-service ; Name of service template to use
service_description Splunk FW Management Clients Version
After reloading the Nagios config you should verify the status of the check. It should look like this if everything is running smoothly.
You can also run the PowerShell script manually for testing. The script accepts multiple parameters
Servername or IP address of the Deployment Server/Forwarder Management
Port of splunkd – default 8089
Protocol to use to communicate with splunkd – default: https
Connection timeout to splunkd in milliseconds – default 5000
Username to use to login to splunkd
Password to use with splunkd
Build version of client to check. has to be passed as an integer value. if client runs on a lower build version a critical message will be is generated
example build numbers
version 4.3.3 = build 128297
version 6.0.2 = build 196940
version 6.1.1 = build 207789
version 6.1.3 = build 220630
version 6.1.4 = build 233537
version 6.1.5 = build 239630
version 6.2.0 = build 237341
Splunk 6.2.1 = build 245427