In today's complex software systems, understanding what's happening under the hood is crucial for delivering high-quality products and services. However, building a culture of monitoring and feedback can be a daunting task, especially for organizations with limited resources. Observability, a concept that originated in the field of control theory, has become a crucial aspect of modern software development. In this article, we'll delve into the world of observability, exploring its best practices and providing actionable tips for building a culture of monitoring and feedback in your organization.
What is Observability?
Observability is the ability to understand the internal state of a system by analyzing its external outputs. In the context of software development, observability refers to the ability to monitor and understand the behavior of complex systems, including their performance, errors, and other key metrics. By collecting and analyzing data from various sources, developers can gain insights into the system's behavior, identify bottlenecks, and make data-driven decisions to improve the overall quality of the system.
There are three main components of observability: monitoring, logging, and tracing. Monitoring involves collecting and analyzing data from various sources, such as application performance metrics, error rates, and user behavior. Logging involves collecting and storing log data from various sources, such as application logs, network logs, and database logs. Tracing involves analyzing the flow of requests through a system, including the time it takes for each request to complete and any errors that may occur along the way.
Best Practices for Building a Culture of Observability
Building a culture of observability requires a combination of technical and organizational skills. Here are some best practices to get you started:
- **Instrument your code**: Use libraries and frameworks to instrument your code and collect data on key metrics, such as application performance, error rates, and user behavior.
- **Use a centralized logging solution**: Collect and store log data from various sources in a centralized logging solution, such as ELK or Splunk.
- **Implement tracing**: Use libraries and frameworks to implement tracing and analyze the flow of requests through your system.
- **Make data-driven decisions**: Use data from monitoring, logging, and tracing to make informed decisions about the system's behavior and performance.
Overcoming Common Challenges
Building a culture of observability can be challenging, especially for organizations with limited resources. Here are some common challenges and solutions:
- **Lack of resources**: Start small and focus on key metrics and logs. Use open-source tools and libraries to reduce costs.
- **Complexity**: Break down the system into smaller components and focus on one component at a time.
- **Resistance to change**: Communicate the benefits of observability to stakeholders and involve them in the process.
Conclusion
Building a culture of monitoring and feedback is crucial for delivering high-quality products and services. By understanding the concept of observability and implementing best practices, you can improve the overall quality of your system and make data-driven decisions. Remember to start small, focus on key metrics and logs, and involve stakeholders in the process.
The problem an SLO actually solves
Every team with a dashboard eventually has the same argument: is the current error rate acceptable, or bad enough to halt deploys and drop everything? Without a pre-agreed number, that argument gets re-litigated every single time, usually under time pressure, usually with whoever is loudest in the incident channel winning. A service level objective — an explicit target like "99.9% of requests succeed, measured over a rolling 30 days" — exists to have that argument exactly once, in a calm meeting, and then turn every future instance of it into arithmetic: is the current rate above or below the number the team already agreed to.
The objective has to be chosen deliberately rather than borrowed from a vendor's marketing page. A target of 99.99% sounds better than 99.9% but allows roughly four minutes of full downtime a month instead of about forty-three minutes, and hitting that tighter number can require an entirely different, more expensive architecture. The right target is the loosest one users will not visibly notice being violated, not the tightest one that looks good in a slide deck.
The error budget: the same number, spent rather than watched
An SLO of 99.9% over 30 days is mathematically the same statement as "this service is allowed 43 minutes and some seconds of failure this month" — and reframing it that way, as a budget to be spent rather than a wall not to be touched, is what actually changes team behavior. A service that has burned through half its monthly budget in the first three days is telling its owners something concrete and actionable: slow down on risky deploys until the trend recovers. A service that has barely touched its budget by day twenty-five is telling its owners the opposite: there is room to take a calculated risk on a change that would otherwise feel too dangerous to ship.
This is the mechanism that turns observability data into an actual decision-making tool rather than a wall of graphs nobody consults until something is already on fire — the budget is checked before a risky deploy, not just after an incident, and it gives teams a shared, non-political basis for saying no to a release that would otherwise be pushed through on schedule regardless of the service's current health.
Choosing what to measure is the hard part
The mechanics of an error budget are simple arithmetic; the actual difficulty is choosing which service level INDICATOR to measure in the first place, because the wrong one produces a number that is technically true and practically useless. Measuring server-side success rate, for instance, misses every request that never reached the server because a client-side network failure or a CDN outage dropped it first — a service can show a perfect SLI while users experience nothing but failures. The indicators that actually track user experience are usually measured as close to the user as practically possible: successful page loads as observed by real browsers, not just successful responses as observed by the origin server.
Latency SLIs have the same trap in a different shape: averaging latency hides the tail, where a small percentage of very slow requests can represent a large fraction of unhappy users while barely moving the mean. This is why latency SLOs are almost always expressed as a percentile — 95% of requests under 300ms — rather than an average, because the percentile is the number that actually correlates with how many real users had a bad experience.
What this changes about the on-call rotation
Once a team has a real error budget, on-call stops being purely reactive — respond when paged — and gains a proactive half: watch the burn rate, and treat a fast burn as an early warning rather than waiting for the budget to fully deplete before anyone notices. Burn-rate alerting — paging when the budget is being consumed at a rate that would exhaust it in, say, two hours if it continued — catches problems earlier and with fewer false positives than a threshold alert on the raw error rate, because a brief spike that recovers on its own barely dents the budget and correctly does not page anyone, while a sustained regression that would actually violate the SLO if left alone gets caught well before the full budget is gone.
Multi-window burn-rate alerts as the practical compromise
A single burn-rate alert has an inherent tension: a threshold sensitive enough to catch a fast, severe regression within minutes is also sensitive enough to fire on brief, self-resolving blips that no human needed to see, and a threshold loose enough to ignore blips is too slow to catch a genuine fast burn before real budget damage is done. The practical fix, used widely in SRE practice, is running several burn-rate windows simultaneously at different sensitivities — a short window (checking the last hour) paired with a longer one (checking the last six hours), both required to agree before paging. A short, sharp spike that recovers within the hour never satisfies the six-hour window and is correctly suppressed; a sustained regression satisfies both windows within a reasonable time and pages promptly.
Tuning these windows and thresholds is iterative, not a one-time setup: a team ships an initial guess, tracks how often it pages for nothing versus how often a real incident was caught late, and adjusts the windows based on that actual track record rather than a formula copied from someone else's system with a completely different traffic pattern and failure profile.
What happens when the budget is already spent
An error budget that reaches zero before the measurement window ends is not a purely theoretical event; it is meant to trigger a real, pre-agreed policy change, and teams that define an SLO without also defining what happens at zero have only done half the work. The common convention is a graduated response: at partial exhaustion, riskier deploys get extra scrutiny or a second reviewer; at full exhaustion, feature work pauses entirely and the team's priority shifts to reliability work until the budget has recovered enough headroom for normal risk-taking to resume. Making this consequence concrete and pre-agreed, rather than negotiated fresh under pressure every time it happens, is what gives the whole error-budget mechanism its teeth — without it, an SLO is just a dashboard number nobody is actually bound by, and the culture reverts to whoever argues loudest in the incident channel deciding whether to ship, exactly the problem the SLO was introduced to solve.
Composite SLOs for systems built from many services
A single user-facing action often depends on several backend services succeeding in sequence, and a naive SLO measured only at the outermost edge can hide which internal dependency is actually responsible for a shortfall. Mature setups define an SLO per service along the dependency chain and roll them up mathematically — since a request that depends on three services each individually meeting 99.9% will, if the failures are independent, succeed at closer to 99.7% overall — so that a team can tell in advance whether its own internal targets are actually tight enough to support the external promise the whole chain is trying to keep, rather than discovering the gap only after the aggregate SLO has already been missed.
Why the objective belongs to the team, not to whoever is loudest
An SLO set unilaterally by one manager, or copied wholesale from a different service with different traffic and different user expectations, rarely survives contact with the team that has to actually be paged against it, and a target nobody upstream actually agreed to tends to get quietly renegotiated during the first real incident anyway, defeating the entire point of having pre-agreed to a number. The objectives that hold up under pressure are the ones the on-call engineers themselves helped set, informed by real historical data about what the service has actually achieved rather than an aspirational number nobody has checked against reality.