Kotej
How to compare the logs of two Kubernetes pods
Two replicas of the same deployment, same image, same config. One serves traffic, the other keeps failing its readiness probe. The answer is almost always in the logs, in the one line that one pod writes and the other doesn't. The problem is finding it, because every single line of the two logs is different.
Why diff on two logs is useless
Every log line carries values that change on every run without meaning anything: the timestamp, the
process ID, the thread name, the pod or host name, request and trace IDs, durations. Run a plain
diff on two pod logs and all of them differ, so the diff tells you nothing.
Step 1: export the logs
# the current logs of each pod
kubectl logs my-app-7c9f8d5b6-abcde > good.log
kubectl logs my-app-7c9f8d5b6-fghij > bad.log
# the previous run, if the container restarted (CrashLoopBackOff)
kubectl logs my-app-7c9f8d5b6-fghij --previous > bad-previous.log
# one container of a multi-container pod
kubectl logs my-app-7c9f8d5b6-fghij -c app > bad.logUseful variations: --since=10m to keep both logs to the same window, and comparing the
--previous log of a crashing pod with the log of a healthy one, which shows how far the crashing
one got before it stopped.
Step 2: set the volatile fields aside
The classic workaround is to strip them with sed or cut before diffing. It works
for one format and breaks on the next: the timestamp has a different shape, a stack trace spreads one entry
over twenty lines, the message itself contains an ID.
Kotej's log comparison does this for you. It recognises the format from the
content (Spring Boot, Log4j and Logback, JBoss, Python, klog, the CRI format of container runtimes, syslog,
access logs, Serilog, JSON lines and logfmt), splits each line into fields and compares the entries
without their timestamp, PID, thread and host. Inside the message, IDs and numbers are masked, and so is
the pod's own name, so Connected to pod my-app-abcde and Connected to pod my-app-fghij
count as the same line.
Step 3: read what's left
What remains are the entries one pod logged and the other didn't, or logged differently: the configuration value that wasn't loaded, the dependency that never answered, the exception the healthy pod never threw. Stack traces are kept together with the entry they belong to, so an exception shows up as one difference, not forty.
If a field you expected to matter is being ignored, or the opposite, choose which fields count. For an
in-house format Kotej doesn't know, describe it with a regular expression with named groups
(timestamp, level, message…) and the same comparison applies.
Logs never leave your machine
Production logs carry hostnames, internal URLs and, sometimes, customer data. Kotej runs in the browser: the two files are read and compared on your device and are not uploaded anywhere, so there is nothing to redact first.
Beyond two pods
The same approach answers other "what changed?" questions: the log before and after a deploy, the same job on two nodes, a staging run against a production run. If you saved many logs at once, compare the two folders, and each pair of logs is compared the same way.