I install a fluentd chart on OKD 3.11 with:
helm install kiwigrid/fluentd-elasticsearch \ --name fluentd \ -f ../../tools/fluentd/values.yaml \ --namespace logging The pod is created, I can enter in it and I can curl to external elasticsearch successfully. However the liveness probe keeps failing and the pods on timeout get recreated:
Normal Killing 10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated. And I can't find any indexes being created in ES.
The liveness config is the default for the chart:
Liveness: exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then exit 1; fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then rm -rf /var/log/fluentd-buffers; exit 1; fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then exit 1; fi; ] delay=600s timeout=1s period=60s #success=1 #failure=3 The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.
