In the fast-paced world of software development, it's easy to get caught up in the latest trends and technologies. However, one crucial aspect of modern software development often gets overlooked: observability. Observability refers to the ability to monitor and understand the behavior of complex systems, making it easier to identify and resolve issues before they impact users.
But why is observability so important? Simply put, it's about building a culture of monitoring and feedback that allows developers to make data-driven decisions and improve the overall quality of their software.
What is Observability?
So, what exactly is observability, and how does it differ from monitoring? While monitoring is primarily focused on detecting issues, observability is about gaining a deeper understanding of the underlying system and its behavior. This involves collecting and analyzing data from various sources, such as logs, metrics, and traces, to identify patterns and trends.
To achieve observability, you need to adopt a holistic approach that involves not only the technical aspects but also the cultural and organizational changes required to make it work. This includes establishing clear goals, defining key performance indicators (KPIs), and fostering a culture of experimentation and learning.
Implementing Observability in Practice
Implementing observability in practice requires a combination of technical and organizational changes. From a technical standpoint, this involves setting up monitoring tools, such as Prometheus and Grafana, and collecting data from various sources. However, it's not just about collecting data; it's also about making sense of it and using it to inform decision-making.
Organizational changes are also crucial. This includes establishing clear goals and KPIs, defining roles and responsibilities, and fostering a culture of experimentation and learning. By doing so, you can create a feedback loop that allows developers to make data-driven decisions and improve the overall quality of their software.
Challenges and Best Practices
While implementing observability can be challenging, there are several best practices that can help. One key challenge is the sheer volume of data being collected, which can be overwhelming to analyze. To address this, it's essential to focus on the most critical metrics and KPIs, and use tools like data visualization to make sense of the data.
Another challenge is the cultural and organizational changes required to make observability work. To overcome this, it's essential to establish clear goals and KPIs, define roles and responsibilities, and foster a culture of experimentation and learning. By doing so, you can create a feedback loop that allows developers to make data-driven decisions and improve the overall quality of their software.
Conclusion
In conclusion, observability is a critical aspect of modern software development that requires a combination of technical and organizational changes. By adopting a holistic approach that involves collecting and analyzing data, establishing clear goals and KPIs, and fostering a culture of experimentation and learning, you can build a culture of monitoring and feedback that allows developers to make data-driven decisions and improve the overall quality of their software.
By following the best practices outlined in this article, you can overcome the challenges of implementing observability and create a more efficient and effective software development process.
A metric is a number that forgot the story behind it
A counter that says "request_errors_total: 4,281" is cheap to store, cheap to query across a year of history, and completely silent about which four thousand requests failed or why. That is the fundamental trade metrics make: they are aggregates, computed and stored as numbers over time, which is exactly what makes them fast to graph and fast to alert on, and exactly what makes them useless for answering "show me one example." A metrics backend built on this model — a fixed set of numeric time series, each identified by a name and a small set of labels — can hold years of history for a fraction of the storage a single day of raw logs would need, because it never stores the individual event, only the running aggregate.
This is precisely why the discipline of keeping label cardinality low matters so much in practice: a label like `status_code` has a handful of possible values and costs almost nothing; a label like `user_id` or `request_id` has millions of possible values, and adding it turns one cheap time series into millions of expensive ones, which is the single most common way teams accidentally blow up their metrics storage bill.
A log line is one event, told in as much detail as someone bothered to write
Where a metric is an aggregate, a log line is a single, timestamped fact: this request came in, this exception was thrown, this connection was refused, with whatever fields the code happened to attach. Unstructured logs — free text meant for a human eye scanning a terminal — are fast to write and miserable to query at scale, because answering "how many of these happened for this customer in the last hour" means parsing prose with regular expressions. Structured logs — each line a JSON object with named fields — cost a little more discipline to write but turn every log aggregator into something closer to a database: filter by `customer_id`, group by `error_code`, and the question that used to require grep and a lot of hope now runs as a query.
The practical rule of thumb that separates teams who log well from teams who log a lot: attach the fields you will actually filter and group by later — request ID, user ID, tenant, code path — as structured fields rather than folding them into a sentence, because a sentence is for a human reading one line, and a field is for a machine answering a question across a million of them.
A trace is the shape of one request as it crosses the whole system
Distributed tracing answers a question neither metrics nor logs answer well on their own: for this one slow request, where exactly did the time go, across every service it touched? A trace is a tree of spans — one span per unit of work, each carrying a start time, a duration, and a parent — propagated via a trace ID that follows the request across every service boundary it crosses. Looking at a trace waterfall for a 2-second request and seeing that 1.7 of those seconds were spent in one downstream call to a service three hops away is the kind of answer that would otherwise take an afternoon of correlating timestamps across five different services' logs by hand.
The catch is cost: capturing a full trace for every single request adds real overhead, and storing them all is expensive at scale, which is why production tracing systems almost always sample — tracing every request in low-traffic services and a deliberately chosen fraction in high-traffic ones, biased toward keeping traces for the slow or failed requests that are actually interesting to look at later.
How the three fit together in an actual investigation
A realistic incident does not start with a trace or a log line, it starts with a metric: error rate crossed a threshold, or p99 latency doubled. The metric answers "something changed, at this time, of this magnitude" cheaply and instantly. From there the investigation pivots to traces from that exact time window to see the shape of the affected requests — which service in the call graph is where the extra time or the failure is concentrated. Only then does it narrow to logs: pull the specific log lines from that one service, in that time window, ideally filtered by the same request or trace ID the trace surfaced, to see the actual error message or stack trace that explains the failure.
Each pillar is the right tool for exactly one step of that funnel — cheap and broad, then precise about shape, then precise about content — and a team that has wired the three together so a person can click from a metric spike into the relevant traces and from a trace into its own log lines has built something qualitatively different from a team that has the same three tools sitting in three separate, disconnected dashboards.
Push versus pull, and why it changes what "cheap" means
Metrics systems split into two collection models with genuinely different operational trade-offs. In a pull model — Prometheus is the canonical example — the monitoring system itself reaches out on a schedule and scrapes each target's current values, which means a target that is down simply fails to be scraped and that absence is itself a clear, unambiguous signal. In a push model, each instrumented process actively sends its metrics to a central collector on its own schedule, which handles short-lived jobs (a batch process that finishes before any scheduled scrape could reach it) far better, but loses that same clean signal — a service that stops pushing looks identical, from the collector's point of view, to a service that has simply gone quiet for a normal reason.
Neither model is strictly better; pull suits long-running services with a stable, discoverable set of targets, and push suits short-lived or highly ephemeral workloads where waiting to be scraped is not a reliable option. Most organizations running both batch and long-lived services end up needing both models represented somewhere in their metrics pipeline, often via a push gateway that lets short-lived jobs push their final numbers into a system that is otherwise pull-based.
What instrumentation libraries actually standardized
Before OpenTelemetry, every observability vendor shipped its own proprietary instrumentation library, which meant switching vendors or adding a second one meant re-instrumenting an entire codebase from scratch — a real, expensive lock-in that had nothing to do with which vendor's backend was actually better. OpenTelemetry's contribution was standardizing the instrumentation layer itself: a single, vendor-neutral API and wire format for metrics, logs and traces that any backend can consume, so a codebase is instrumented once and the choice of where that data ultimately gets stored and queried becomes a configuration decision rather than a rewrite. This matters for exactly the reason the three-pillar framing matters throughout this cluster of articles: the signals themselves — what a metric is, what a trace is — are more fundamental and more durable than any specific vendor's product, and standardizing how they are emitted is what finally let teams treat backend choice as replaceable infrastructure instead of a permanent commitment baked into application code.
Exemplars: the bridge that links a metric spike to one real trace
A histogram bucket showing that 2% of requests took over a second tells a team that a slow tail exists, but not which specific requests were in it — exemplars close exactly that gap by attaching a small sample of real trace IDs directly to the metric data point that produced them, so clicking on the spike in a latency histogram can jump straight to an actual trace from that exact bucket rather than requiring a separate, manual search through a trace store hoping to find a matching example from the same time window.