fluentd

"Fluentd is an open source data collector for unified logging layer." - https://www.fluentd.org/

Snips

Parse the tail pos file into decimal position, inode, and inspect the position

This pos_file's columns are Filename,Position,Inode. In the below examples we don't actually do anything with the inode number, but you could use in in debugfs etc..

POS_FILES="/var/log/fluentd-containers.log.pos"
while read -r file pos inode ; do
    echo "$file $((16#$pos)) $((16#$inode))"
done < "$POS_FILE"

This will output something like:

/var/log/containers/calico-node-0am...cb0.log 2797 5347425

You can feed this through some math to see how far behind fluentd is for each file, and then into dd to see what the data is that fluentd has yet to process:

while read -r file pos _ ; do
    if f=$(readlink -f "$file") && [ -f "$f" ] ; then  # resolve symlinks and check that file exists
        f_size="$(stat -c "%s" "$f")"  # get file size in bytes
        pos_dec=$((16#$pos))  # convert pos from hex to dec
        if [[ "$f_size" -gt "${pos_dec}" ]]; then
            echo "$file f_size=${f_size}, pos=${pos_dec}, diff=$(( f_size - 16#$pos ))"
            dd status=none bs=1 if="$f" skip="${pos_dec}" count=256
            echo
        fi
    fi
done < "$POS_FILE"

Which will output a bunch of lines like:

/var/log/containers/network-metering-agent-tsl6s_kube-system_agent-25c3e4bc7bd0ddfdda571d8279b040d0a2f3dac03786a40b19dac11873a6af5a.log f_size=1996377, pos=1995147, diff=1230
{"log":"W0809 18:03:09.184540       1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 1489908695 (1489955501)\n","stream":"stderr","time":"2021-08-09T18:03:09.184786383Z"}
{"log":"W0