When something breaks in production — often at an inconvenient hour — logs are frequently your only window into what happened. Yet logging is treated as an afterthought, and the result is either near-silence, leaving you blind, or a deluge of noise that buries the one line that mattered. Good logging is a genuine design skill.
The goal is notes that your future, tired, debugging self will thank you for.
The two failure modes
Too little logging leaves you unable to reconstruct what went wrong, guessing in the dark. Too much logging — recording everything indiscriminately — is arguably worse, because the important signal is drowned in noise, and searching through it becomes its own ordeal. Both extremes fail you at the moment you need the logs most.
Effective logging lives in the middle: enough to reconstruct events, structured enough to search, and free of pointless chatter.
What makes a log useful
Useful log entries carry context: what was happening, relevant identifiers, and enough detail to understand the situation without exposing sensitive data. Using appropriate severity levels lets you separate routine information from genuine problems, so you can focus on what matters. Consistency and structure make logs searchable rather than a wall of freeform text.
A good log entry answers "what was the program doing, and what happened?" for someone who was not there when it ran.
Logging with care
A serious caution: logs must never become a security hole. Passwords, keys, personal data and other sensitive information should be kept out of logs, since logs are often widely accessible and long-lived. Think about who can read them and for how long.
Approached deliberately — meaningful context, sensible levels, no sensitive data, neither famine nor flood — logging turns a production mystery into a readable story. It is worth designing rather than sprinkling in as an afterthought.
The test that actually matters: could a stranger use this at 3am
The engineer who gets paged for a service at 3am very often did not write the code that just failed, does not remember its internals, and has minutes rather than hours to figure out what is wrong before an SLA is breached. Every logging decision — what to log, at what level, with what fields — should be judged against that specific reader rather than against the person who wrote the code and already understands it. A log line that makes perfect sense to its author in the moment they wrote it ("retry limit hit") is nearly useless to a stranger six months later with no other context: retry limit for what operation, hit how many times, affecting which customer, with what underlying error on the final attempt. The habit that separates logging that helps from logging that merely exists is writing every message as though the reader has never seen this code before, because at 3am, functionally, they usually have not.
Structured fields over prose, every time
A log line written as a sentence — "Failed to process order 4471 for customer acme-corp: insufficient inventory" — is comfortable to read once but nearly impossible to query reliably at scale, because pulling every failure for a given customer means writing a fragile string match against free text that breaks the moment anyone tweaks the wording. The same information logged as structured fields — `order_id: 4471, customer: acme-corp, reason: insufficient_inventory` — reads slightly less naturally on a terminal but can be filtered, grouped, and counted directly by any log query tool without guessing at phrasing. The right habit is not choosing one over the other but doing both: a short human-readable message for the person scanning a terminal, plus the same information duplicated as structured fields for the person — or the alerting rule — that needs to query across a million lines rather than read one.
What NOT to log is as much a design decision as what to log
Verbose logging on a hot path has a real, measurable performance cost, and logging sensitive data — full credit card numbers, plaintext passwords, complete request bodies containing personal information — creates a compliance and security liability that is often worse than the debugging convenience it buys, because a log aggregator is itself an attack surface and a breach of it can leak exactly the data the application was careful to protect everywhere else. Mature logging practice treats redaction as a first-class concern applied automatically at the logging layer — masking known-sensitive field names before they ever leave the process — rather than trusting every individual call site to remember to redact manually, because the one place someone forgets is the one place a real leak happens.
Log levels as an on-call contract, not a suggestion
The most useful convention a team can enforce is treating `error` level as an implicit promise: something at this level is worth a human's attention, possibly right now, possibly at 3am. The moment that promise is broken by logging routine, expected conditions at `error` level out of convenience, the whole signal degrades — the same alert-fatigue mechanism that ruins paging systems ruins log levels just as thoroughly, training whoever reads them to skim past `error` lines because most of them turn out to be nothing. Keeping that promise intact — routine conditions at `info` or below, no matter how tempting it is to make something more visible by escalating its level — is what keeps a log level filter actually meaningful six months and a few team turnovers later, instead of degrading into noise nobody trusts.
Timestamps and clocks: the detail that quietly wrecks correlation
A log aggregator pulling lines from many machines is implicitly trusting that every machine's clock agrees closely enough that ordering events by timestamp actually reflects the order they happened in. Clock drift between hosts — a few hundred milliseconds is common even with NTP running, and it can be far worse if NTP is misconfigured or blocked on some hosts — can make logs from two services appear out of order even when the causal chain between them was the other way around, which is exactly the kind of subtle error that makes an investigator draw the wrong conclusion about what caused what. The practical fixes are boring but effective: use a properly synchronized time source everywhere logs are generated, always log in UTC rather than local time so that a per-host timezone misconfiguration cannot silently shift one machine's logs relative to every other machine's, and prefer request-scoped ordering — the sequence implied by a trace or correlation ID — over raw wall-clock timestamp ordering whenever the two disagree, because the request's own causal order is the one that is actually true regardless of what any individual clock says.
Sampling what you log without losing the incident you needed
A service under enough load that full logging becomes a real cost problem still needs the one property that makes sampling safe: it must never be the routine 99% that gets kept while the 1% documenting an actual failure gets silently dropped along with everything else. The practical pattern is asymmetric by design — sample successful, routine requests aggressively to control volume, but always keep, at full detail, anything associated with an error, an unusually slow response, or any other flagged anomaly. Getting this backwards, applying a single uniform sampling rate across every log line regardless of what it represents, produces a logging pipeline that looks like it is working right up until the one incident where the specific line that would have explained everything happened to fall on the wrong side of a coin flip.
Making log volume itself an observable, not an afterthought
Teams that only think about logging as a debugging tool, and never as a system with its own resource budget, are routinely surprised by how quickly log volume itself becomes an incident: a misconfigured retry loop or a newly deployed bug that logs on every iteration of a hot path can generate orders of magnitude more volume than normal within minutes, silently overwhelming the shipping pipeline or blowing through a cost budget long before anyone notices from the application's own behavior. Mature setups treat log volume as a first-class metric in its own right — graphed, alerted on, and reviewed with the same seriousness as request latency or error rate — specifically so that a sudden spike in logging itself is caught early, as a symptom worth investigating on its own, rather than discovered days later as an unpleasant surprise on an infrastructure bill or a suddenly-overwhelmed aggregation pipeline.
The paradox of the perfect log line nobody wrote
Every postmortem eventually produces some version of the same regret: if only this one specific field had been logged at this one specific point, the incident would have been diagnosed in minutes instead of hours. That regret is not evidence of an isolated oversight; it is the predictable, permanent output of the fact that no one can log everything that might someday matter, because the cost of comprehensive logging on every hot path would be prohibitive and most of the additional detail would never be needed. The realistic, sustainable response is not trying to anticipate every future question in advance, it is closing the specific gap a real incident just revealed, which is exactly why treating postmortems as an input to logging decisions, not just to code fixes, compounds over time into a logging setup that reflects the actual failure modes a particular system has actually had, rather than a generic list of fields someone guessed might be useful before anything had gone wrong.
Idempotency keys and request IDs are worth logging even when nothing is wrong
It is tempting to only log identifiers when the code path fails, since that is when they seem to matter, but the request that later turns out to matter is rarely known to be interesting at the moment it happens — logging the request ID, idempotency key, and correlation ID on every request, success or failure, at a cheap debug or info level is what makes it possible to trace a customer's specific complaint back to the exact request in question hours or days later, instead of discovering that the one request that mattered was the one nobody thought to log fully because at the time it looked completely routine.
Consistent field naming across services is worth enforcing centrally
When one service logs `user_id` and another logs `userId` and a third logs `uid` for the exact same concept, a query that needs to correlate activity across all three has to know and handle every variant by name, which is a small tax paid on every single cross-service investigation — enforcing a shared naming convention centrally, rather than leaving it to each team's own preference, is unglamorous work that pays for itself the first time an incident spans more than one service.