Send Docker container log to Splunk

Docker engine has Splunk logging driver to send container logs to Splunk via HEC(http event collector. It is easy to setup, however it could suffer data loss if HEC is down. That is the advantage of file based data ingestion which can retry and resume even splunk service is down for long period of time. The default Docker logging driver is json-file, there is no problem for Splunk agent to read the file, however how could you set index for different application, since the container id is random so the file path is random. The trick to use jounald driver and use rsyslog rules to read the jounald log and write to different file names based on any docker label.


Overview:
steps included:
- enable persistent journald storage
- create rsyslog rule to read journald and write to json log file,file name depends on docker_appnane

remaining steps to be achieved by other Ansible roles
- docker container need to set log driver to journald and expose label docker-appname e.g
docker run --log-driver=journald \
--log-opt labels=docker-appname \
--label docker-appname=mulesoft \

- Splunk UF read /var/log/docker-{{docker_appname}}.log and forward to splunk cloud
- logrorate rule to rotate /var/log/docker-{{docker_appname}}.log

Rsyslog rule in RHEL7

#imjournal module is loaded in main syslogd.conf

$IMJournalStateFile imjournal.state
$imjournalRatelimitInterval 300
$imjournalRatelimitBurst 30000

module(load="mmjsonparse") 

action(type="mmjsonparse")

#output all json fileds and remove redundant last msg field
#template(name="jsonformat" type="string" string="%$!all-json:R,ERE,1,FIELD:(.*), (\"msg\":.*)--end% }\n")
template(name="jsonformat" type="string" string="%$!all-json%\n")

if ($!DOCKER_APPNAME == "{{docker_appname}}") then {
    action(type="omfile" file="/var/log/docker-{{docker_appname}}.log" template="jsonformat")
    stop
} 

Splunk customized source type to be set in Heavy forwarder or Indexer(universal forwarder doesn't support sourcetype defination)

props.conf

[json_realtime_timestamp]
KV_MODE = json
MAX_TIMESTAMP_LOOKAHEAD = 16
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_FORMAT = %s%6N
TIME_PREFIX = "_SOURCE_REALTIME_TIMESTAMP":\s*"
pulldown_type = 1
TZ=UTC

inputs.conf in universal forwarder in Docker host

[monitor:///var/log/docker-mulesoft.log]
disabled = true
sourcetype=json_realtime_timestamp
index = mulesoft