RSS

Install Splunk and Forwarder on Linux

Step by step install Splunk, Splunk Forwarder, Splunk app free trial version on Linux.

Install Splunk

Download from https://www.splunk.com/en_us/download/splunk-enterprise

Download Splunk 8 via Command Line (wget):

# .deb For Debian and Ubuntu $ wget -O splunk-8.0.3-a6754d8441bf-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=splunk&filename=splunk-8.0.3-a6754d8441bf-linux-2.6-amd64.deb&wget=true' # .rpm For Redhat and CentOS $ wget -O splunk-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=splunk&filename=splunk-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm&wget=true' # .tgz For all Linux platforms $ wget -O splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=splunk&filename=splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz&wget=true'

Install Splunk on Debian Buster using .deb:

$ sudo dpkg -i splunk-8.0.3-linux-2.6-amd64.deb Selecting previously unselected package splunk. (Reading database ... 89422 files and directories currently installed.) Preparing to unpack splunk-8.0.3-linux-2.6-amd64.deb ... Unpacking splunk (8.0.3) ... Setting up splunk (8.0.3) ... complete

The Splunk install location is /opt/splunk:

$ ls -l /opt/splunk total 2936 -r--r--r-- 1 splunk splunk 842 Mar 31 02:36 README-splunk.txt drwxr-xr-x 4 splunk splunk 4096 Apr 9 15:02 bin -r--r--r-- 1 splunk splunk 57 Mar 31 02:33 copyright.txt drwxr-xr-x 15 splunk splunk 4096 Apr 9 15:02 etc -rw-r--r-- 1 splunk splunk 426 Apr 9 15:02 ftr drwxr-xr-x 4 splunk splunk 4096 Apr 9 15:02 include drwxr-xr-x 8 splunk splunk 4096 Apr 9 15:02 lib -r--r--r-- 1 splunk splunk 85709 Mar 31 02:33 license-eula.txt drwxr-xr-x 3 splunk splunk 4096 Apr 9 15:02 openssl drwxr-xr-x 4 splunk splunk 4096 Apr 9 15:02 share -r--r--r-- 1 splunk splunk 2881101 Mar 31 02:56 splunk-8.0.3-linux-2.6-x86_64-manifest

Start, Stop and Restart Splunk

Optional add splunk to PATH:

export PATH="$PATH:/opt/splunk/bin"

Start, stop and restart splunk:

$ splunk start $ splunk stop $ splunk restart

Note: Should use splunk user to execute all splunk related command and config change.

Example:

$ sudo -u splunk splunk start --accept-license This appears to be your first time running this version of Splunk. Splunk software must create an administrator account during startup. Otherwise, you cannot log in. Create credentials for the administrator account. Characters do not appear on the screen when you type in credentials. Please enter an administrator username: admin Password must contain at least: * 8 total printable ASCII character(s). Please enter a new password: Please confirm new password: Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'. Generating RSA private key, 2048 bit long modulus .....................................................................................................+++++ ..................................................................................................................................+++++ e is 65537 (0x10001) writing RSA key Generating RSA private key, 2048 bit long modulus ..........+++++ ....................................................+++++ e is 65537 (0x10001) writing RSA key Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'. Splunk> CSI: Logfiles. Checking prerequisites...	Checking http port [8000]: open	Checking mgmt port [8089]: open	Checking appserver port [127.0.0.1:8065]: open	Checking kvstore port [8191]: open	Checking configuration... Done.	Creating: /opt/splunk/var/lib/splunk	Creating: /opt/splunk/var/run/splunk	Creating: /opt/splunk/var/run/splunk/appserver/i18n	Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css	Creating: /opt/splunk/var/run/splunk/upload	Creating: /opt/splunk/var/run/splunk/search_telemetry	Creating: /opt/splunk/var/spool/splunk	Creating: /opt/splunk/var/spool/dirmoncache	Creating: /opt/splunk/var/lib/splunk/authDb	Creating: /opt/splunk/var/lib/splunk/hashDb New certs have been generated in '/opt/splunk/etc/auth'.	Checking critical directories...	Done	Checking indexes...	Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket history main summary	Done	Checking filesystem compatibility... Done	Checking conf files for problems...	Done	Checking default conf files for edits...	Validating installed files against hashes from '/opt/splunk/splunk-8.0.3-a6754d8441bf-linux-2.6-x86_64-manifest'	All installed files intact.	Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Generating a RSA private key ..............................................................................................+++++ ..........+++++ writing new private key to 'privKeySecure.pem' ----- Signature ok subject=/CN=buster/O=SplunkUser Getting CA Private Key writing RSA key Done Waiting for web server at http://127.0.0.1:8000 to be available........ Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://127.0.0.1:8000

sudo /opt/splunk/bin/splunk enable boot-start -user 

$ sudo /opt/splunk/bin/splunk enable boot-start -user splunk Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.

Splunk ports

  • 8000: For clients to the Splunk Web (webserver)
  • 8089: For Splunk Management port (inter Splunk communication)
  • 9997: For forwarders to the Splunk indexer. (forwarding and receiving data) This need manually enable, see blow Splunk Forwarder.

Config Splunk

Disable telemetry to splunk

If you donot want to send your splunk usage to Splunk Inc. edit /opt/splunk/etc/apps/splunk_instrumentation/local/telemetry.conf to disable telemetry.

$ sudo -u splunk cat /opt/splunk/etc/apps/splunk_instrumentation/local/telemetry.conf [general] telemetrySalt = ffbebefe-512d-4aa7-b832-634c9b854f6b deploymentID = 7bfc7c72-c062-520a-8a3c-78975cdf5d38 reportStartDate = 2020-04-09 sendAnonymizedUsage = false sendAnonymizedWebAnalytics = false sendLicenseUsage = false optInVersionAcknowledged = 4 sendSupportUsage = false showOptInModal = false

see telemetry.conf reference for more configuration.

Add local file monitor

sudo -u splunk vim /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/nginx/splunk/access.log] disabled = false index = splunk_web sourcetype = access_combined [monitor:///var/log/nginx/splunk/error.log] disabled = false index = splunk_web sourcetype = nginx_error_log [monitor:///var/log/nginx/buster/access.log] disabled = false index = buster_web sourcetype = access_combined [monitor:///var/log/auth.log] disabled = false index = buster_os sourcetype = linux_secure [monitor:///var/log/syslog] disabled = false index = buster_os sourcetype = syslog

Splunk Forwarder

Enable forwarder receiver on Splunk server

Before use splunk forwarder, you need enable receiver on splunk server:

Settings -> Forwarding and receiving -> Receive data -> Add new

Enter port you want (9997 by default). For example, 9997 will receive data on TCP port 9997.

Or use command line:

$ sudo -u splunk /opt/splunk/bin/splunk enable listen 9997

Install Forwarder

Download forwarder at https://www.splunk.com/en_us/download/universal-forwarder.html

Download the current latest version 8.0.3 via Command Line (wget):

# .rpm format (For Redhat, CentOS) $ wget -O splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=universalforwarder&filename=splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm&wget=true' # .deb format (For Debian, Ubuntu) $ wget -O splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=universalforwarder&filename=splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-amd64.deb&wget=true' # .tgz format (For all linux platform) $ wget -O splunkforwarder-8.0.3-a6754d8441bf-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=universalforwarder&filename=splunkforwarder-8.0.3-a6754d8441bf-Linux-x86_64.tgz&wget=true'

Install on CentOS:

# sudo rpm -ivh splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm warning: splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY Preparing... ################################# [100%] Updating / installing... 1:splunkforwarder-8.0.3-a6754d8441b################################# [100%] complete

Splunk forwarder install location: /opt/splunkforwarder/

Start Splunk Forwarder

# cd /opt/splunkforwarder/bin # sudo -u splunk ./splunk start --accept-license This appears to be your first time running this version of Splunk. Splunk software must create an administrator account during startup. Otherwise, you cannot log in. Create credentials for the administrator account. Characters do not appear on the screen when you type in credentials. Please enter an administrator username: fadmin Password must contain at least: * 8 total printable ASCII character(s). Please enter a new password: Please confirm new password: Splunk> Finding your faults, just like mom. Checking prerequisites...	Checking mgmt port [8089]: open	Creating: /opt/splunkforwarder/var/lib/splunk	Creating: /opt/splunkforwarder/var/run/splunk	Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n	Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css	Creating: /opt/splunkforwarder/var/run/splunk/upload	Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry	Creating: /opt/splunkforwarder/var/spool/splunk	Creating: /opt/splunkforwarder/var/spool/dirmoncache	Creating: /opt/splunkforwarder/var/lib/splunk/authDb	Creating: /opt/splunkforwarder/var/lib/splunk/hashDb New certs have been generated in '/opt/splunkforwarder/etc/auth'.	Checking conf files for problems...	Done	Checking default conf files for edits...	Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64-manifest'	All installed files intact.	Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done [ OK ]

Config Forwarder

The first step is use splunk add forward-server to add a forwarder server.

$ ./splunk add forward-server <splunk server>:<receiver port>

Example:

# ./splunk add forward-server 1.2.3.4:9997 Splunk username: fadmin Password: Added forwarding to: 1.2.3.4:9997.

Remove forwarder server:

# ./splunk remove forward-server 1.2.3.4:9997 Stopped forwarding to: 1.2.3.4:9997

Install Splunk forwarder as system service so it can startup at boot time:

# ./splunk enable boot-start Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.

Test Forwarder connection:

# /opt/splunkforwarder/bin/splunk list forward-server Splunk username: fadmin Password: Active forwards:	None Configured but inactive forwards:	1.2.3.4:9997

Add data to monitor

./splunk add monitor /var/log/auth.log -sourcetype linux_secure ./splunk add monitor /var/log/syslog -sourcetype syslog

Note: This is not permanent, need use inputs.conf to make it permanent.

Forwarder Config File - inputs.conf

You can configure data inputs on a forwarder by editing the inputs.conf configuration file.

In nearly all cases, edit inputs.conf in the $SPLUNK_HOME/etc/system/local directory. If you have an app installed and want to make changes to its input configuration, edit $SPLUNK_HOME/etc/apps/<appname>/local/inputs.conf. For example, if you have the Splunk Add-on for Unix and Linux installed, you would make edits in $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/inputs.conf.

Do not make changes to the inputs.conf in $SPLUNK_HOME/etc/system/default. When you upgrade, the installation overwrites that file, which removes any changes you made.

Whenever you make a change to a configuration file, you must restart the forwarder for the change to take effect.

Edit inputs.conf

  1. Using your operating system file management tools or a shell or command prompt, navigate to $SPLUNK_HOME/etc/system/local.
  2. Open inputs.conf for editing (e.g. /opt/splunkforwarder/etc/system/local/inputs.conf). You might need to create this file if it does not exist.
  3. Add your data inputs.
  4. Once you have added your inputs, save the file and close it.
  5. Restart the forwarder.

Example inputs.conf:

# /opt/splunkforwarder/etc/system/local/inputs.conf # # Linux login log. # For Debian, the log path is /var/log/auth.log # For CentOS, the log path is /var/log/secure [monitor:///var/log/secure] index=foo_os_log sourcetype=linux_secure [monitor:///var/log/syslog] index=foo_os_log sourcetype=syslog

Add splunk user to root (CentOS/Redhat) or adm (Debian/Ubuntu) group to have read access to /var/log/secure:

# usermod -a -G root splunk

Install Splunk app

Get Splunk app at https://splunkbase.splunk.com/

To mannually install apps and add-ons directly into Splunk Enterprise

  1. Put the downloaded file in the $SPLUNK_HOME/etc/apps directory.
  2. Untar and ungzip your app or add-on, using a tool like tar -xvf (on *nix) or WinZip (on Windows).
  3. Restart Splunk.

Troubleshooting

Enable debug log

Change $SPLUNKFORWARDER/etc/log.cfg to enable DEBUG logging, change category.TailingProcessor and category.WatchedFile to DEBUG:

[splunkd] rootCategory=WARN,A1 # TailingProcessor is meant to be used at level INFO -- without it, analyzing a # normal diag becomes much harder. Do NOT remove the TailingProcessor logger. category.TailingProcessor=DEBUG category.WatchedFile=DEBUG

Log file: /opt/splunkforwarder/var/log/splunk/splunkd.log

Forwarder can not connect to Splunk server error:

WARN TcpOutputProc - Cooked connection to ip=1.2.3.4:9997 timed out

Sample success log of forwarder connected to Splunk server error:

INFO TcpOutputProc - Connected to idx=1.2.3.4:9997, pset=0, reuse=0. INFO TcpOutputProc - Found currently active indexer. Connected to idx=1.2.3.4:9997, reuse=1.

Related pages:

References

OmniLock - Block / Hide App on iOS

Block distractive apps from appearing on the Home Screen and App Library, enhance your focus and reduce screen time.

DNS Firewall for iOS and Mac OS

Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps

Ad