Speed Up Issue Resolution with Full-Stack Observability

In recent years, observability has become an important capability for IT teams to develop within their technology stacks. Organizations often want clarity about how observability and automation differ; they’re distinct concepts, yet intertwined. Frequently, the goal of observability is to enable automation by providing greater visibility into IT errors, events and incidents.

Full-stack observability enhances the traditional network and systems monitoring that IT teams have always performed, making it easier to resolve issues more rapidly. As organizations try to do more with less, this capability becomes even more important. By automating issue resolution, organizations extend the reach of small IT teams. However, this capability requires visibility into incidents and their impact, not only on technology but also on revenue. For example, when we work with customers, we can measure how many transactions were abandoned due to an application problem and the corresponding effect on revenue.

Typically, the alerts IT teams receive from technology resources — in network operating centers, for instance — often lack context, making it difficult to understand what is actually happening. As a result, staffers tend to intervene only when an issue becomes severe and starts to affect revenue, the customer experience or another critical area. Then, the response is all hands on deck as everyone pursues identification, remediation and root-cause analysis. Unfortunately, the mean time to identification often turns into “mean time to innocence,” with each organizational silo denying responsibility. Of course, the business doesn’t care whose fault it is, it just wants the problem fixed.

The next generation of observability helps organizations avoid these issues by adding contextual information to telemetry data. If a construction crew accidentally cuts a fiber cable, causing a data center to go dark, that will set off alarms everywhere. Observability makes it possible to pull relevant data into a context-and-correlation engine that can see the problem: The internet is down, which means the wide area network team can resolve the issue and everyone else can return to work. Another benefit of this shift is that it helps to create a culture more focused on resolution than blame.

A dark data center is an extreme example that requires human intervention, but other issues may be handled by automated tools. For example, if an organization’s cloud instances are overtaxed, an automated solution can recognize that the environment will be short on resources and automatically spin up what is needed to augment the shortfall. When organizations reach the point of automating this type of workflow and adjusting on the fly, they can start to resolve issues before they begin to affect the business.

When we partner with customers to improve their observability capabilities, we start by understanding the true nature of the impact when an incident occurs. Building this type of capability is a significant change for most organizations, not only in their tools but also in their skills and operations. Making this work correctly and getting everyone on the same page is often a matter of breaking down silos.

The next step in our process is mapping the customer journey to understand where observability can affect customers’ experiences. Understanding the workflow is essential because automating a poor process doesn’t fix anything; it only makes the process happen more quickly. We also analyze customers’ existing tools to identify gaps and determine which solutions will help them get where they want to be.

Finally, we conduct workshops on automation and observability to help stakeholders build a framework to pull these capabilities together holistically and strategically. When it’s time to move forward, we help customers develop a plan to implement full-stack observability, including design, implementation and integration.

Story by Mark Beckendorf, the head of full-stack observability for Digital Velocity at CDW.