When something breaks in production, you cannot attach a debugger and step through it as it happens to thousands of users. What you have instead is evidence: the logs and metrics your system left behind. Learning to read that evidence — and to leave good evidence in the first place — is the difference between diagnosing an outage in minutes and staring at it for hours.
The skill has two halves: reading logs when things go wrong, and designing what to log so that the logs are worth reading.
Reading logs under pressure
The first move during an incident is to narrow the time window and search, not scroll. Filter logs to the minutes around the failure, then search for error levels and known keywords. Follow the timeline: what was the last thing that succeeded, and what was the first that failed? Errors often cascade, so the loudest, most numerous errors are frequently symptoms — hunt upstream for the first, quietest one that started the avalanche.
Correlation is the other key. A single request that touches several services is far easier to trace if every log line carries a shared request or trace id, letting you follow one user's journey across the whole system instead of guessing which log lines belong together.
Logging that is actually useful
Good logs are designed, not accidental. Log at meaningful levels — errors for genuine failures, warnings for recoverable oddities, info for significant events — so you can filter to the signal. Include context: what operation, which user or request id, what the relevant values were. A log that says "error" tells you nothing; one that says "failed to charge order 4821 for user 77: gateway timeout after 30s" tells you almost everything.
And avoid the two failure modes: logging so little that failures are invisible, and logging so much that the signal drowns. Never log secrets or sensitive personal data — logs are widely readable and long-lived. Structured logging (machine-readable key-value fields) makes logs searchable and filterable in ways plain text never will.
Beyond logs: the three pillars
Logs are one of three complementary views into a running system. Metrics are numbers over time — request rates, error rates, latency, resource use — that tell you something is wrong and how bad. Traces follow a single request across services to show where time went. Logs give the detailed narrative of individual events. Together they are called observability, and each answers a different question.
Metrics tell you the site is slow; traces tell you which service is the bottleneck; logs tell you exactly what that service did. You do not need a heavyweight platform to start — decent logging with request ids and a few key metrics already transforms your ability to understand production. The teams that recover fast are simply the ones that invested in being able to see.