The Evolution of Observability

Historical Context

Historically, observability has relied on three key pillars that form the foundation for understanding and managing system performance:

1. Logging: Recording events and activities within systems.
2. Metrics: Capturing numerical data about system performance.
3. Tracing: Following the journey of requests across system components.

These elements were essential for gaining a complete picture of an IT landscape. However, these pillars often operated in silos, with different platforms handling each pillar separately. This made it difficult to connect data from different sources and gain a clear, complete view of a system’s health.

Standardization Efforts

This siloed approach led to efforts to standardize observability. Initiatives like Opentracing (focused on tracing) and OpenCensus (focused on metrics and logging) emerged to address these gaps. Eventually, these two standards merged into OpenTelemetry. This highlights the need for a unified approach to logging, metrics, and tracing, allowing teams to gain deeper, more comprehensive insights into system behavior.

Current Pitfalls in Observability

Even though logging, metrics, and tracing are crucial for observability, several challenges still exist:

• Data Silos: Many platforms manage logging, metrics, and tracing independently, making it difficult and time-consuming to connect and analyze data across these sources.
• Boundary Issues: Traces often stop at the edges of applications, making it hard to link traces from different applications. This forces IT teams to rely solely on timestamps and estimations, which, of course, can be inaccurate.

OpenTelemetry: A Unified Framework

OpenTelemetry’s unified framework has been a game changer for tackling data silos and boundary issues in observability.

What is OpenTelemetry?

OpenTelemetry is a framework that integrates logging, metrics, and tracing into a single, unified system. It introduces the concept of signals as a generic term for these telemetry data types. OpenTelemetry is designed to be tool-independent, language-agnostic, and free from creating new dependencies.

Key Principles of OpenTelemetry

1. Tool independence: Works across various programming languages and tools.
2. No dependencies: Does not require specific collectors to function, avoiding issues related to library versions.
3. Backward compatibility: Ensures that updates to the standard do not require rewriting existing applications.

The Reality Check

While OpenTelemetry makes it much easier to spot patterns in system data, it still takes a human touch to tell the difference between correlation and causation. Technology can show us when things are related, but figuring out if one thing is actually causing the other is where human judgment comes into play. In other words, understanding what’s really driving system behaviour requires thoughtful analysis and real-world context. With AI and machine learning on the rise, we might soon be able to automate more of this process—making it easier to pinpoint root causes and get clearer insights with less manual effort.

Deep Dive into Signals

To fully understand system behaviour, OpenTelemetry focuses on three core types of signals: tracing, metrics, and logging. Each provides unique insights, working together to give a complete view of system performance and health.

Tracing

Tracing captures the complete path of a request through a system, composed of multiple spans that represent individual units of work.

• Span: A unit of work within a distributed system, containing information about its execution.
• Trace: A record of the journey of a request, consisting of linked spans.

Metrics

Metrics are real-time measurements of a service’s performance, such as CPU usage, memory consumption, and request counts. They are crucial for monitoring system availability and performance. Metrics can also trigger alerts or prompt adjustments in deployment when thresholds are met.

Logging

Logs are timestamped records of events and activities, which are crucial for debugging and diagnostics. OpenTelemetry serves as a bridge, integrating existing logging solutions with tracing and metrics to provide a more comprehensive view of system performance.

Fundamentals of OpenTelemetry

OpenTelemetry’s framework is built on key components that simplify capturing and managing telemetry data. Core elements like auto-instrumentation, protocols, context propagation, and sampling work together to deliver a complete observability solution.

Auto-Instrumentation

OpenTelemetry offers automatic instrumentation for popular programming languages like .NET, Java, Node.js, Python, and Go. It dynamically modifies bytecode to collect telemetry data without the need for manual code changes.

Protocol & Collector

Data is transmitted using standardized protocols to a collector. The collector processes and exports the data to an observability backend, which serves as a central hub for data management.

Context, Baggage, and Propagators

OpenTelemetry introduces key concepts like context, baggage, and propagators to manage and share data across services:

• Context: Stores information that links signals across different services.
• Propagators: Handle the transfer of context between services.
• Baggage: Contextual information shared across spans, useful for carrying metadata like CustomerId.

Sampling

Sampling helps manage data volume by selectively capturing telemetry data, which reduces costs and focuses on the most relevant traces.

Application Architecture

Application Architecture focuses on how a system’s components interact. In OpenTelemetry, it ensures smooth instrumentation and data flow for better observability.

Implementing OpenTelemetry

Implementing OpenTelemetry in an existing landscape requires a strategic and thoughtful approach. The goal is to roll it out in a way that maximizes impact while keeping the implementation manageable.

That’s why we recommend the following approach:

1. Identify high-value processes: Start with an end-to-end process that is valuable or problematic. This will provide meaningful insights from the start.
2. Ensure Comprehensive Coverage: Implement OpenTelemetry across the entire process to fully harness its potential. This ensures that you’re gathering data from all critical points.
3. Prioritize Breadth Over Perfection: Start with auto-instrumentation and capture existing logs before optimizing further. Focus on getting broad visibility first, and then improve and fine-tune later.