Skip to content
Deadline to Die

Monitoring & OpenTelemetry

📊 Monitoring & OpenTelemetry

Observability is the ability to measure the internal state of a system by examining its outputs. For Python data pipelines, this involves tracking performance and data health.

🏗️ 1. OpenTelemetry (OTel)

OpenTelemetry is a vendor-neutral standard for collecting Traces, Metrics, and Logs.

Key Components:

  • Traces: Visualize the path of a request through your microservices.
  • Metrics: Track numerical values over time (e.g., pipeline execution time, rows processed).
  • Logs: Structured logs (JSON) for easy searching in tools like ELK or Datadog.
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_batch"):
    # Log logic here...
    pass

🚀 2. Model & Data Drift Monitoring

Once a model is in production, its performance can degrade over time.

Types of Drift:

  • Data Drift: The input data distribution changes (e.g., users’ demographics shift).
  • Concept Drift: The relationship between input and output changes (e.g., customer behavior shifts during a pandemic).

Tools:

  • Evidently AI: A Python library for monitoring model performance and data drift.
  • Arize / WhyLabs: Enterprise platforms for ML monitoring.

📦 3. Structured Logging (Serilog/Loguru)

Avoid standard print(). Use structured logging to capture context.

from loguru import logger

logger.info("Processing batch", batch_id=123, rows=5000)

🚦 4. Monitoring Best Practices

  1. Dashboarding: Create Grafana or Streamlit dashboards for real-time visibility.
  2. SLIs/SLOs: Define Service Level Indicators (e.g., “99% of pipelines finish within 2 hours”).
  3. Trace IDs: Pass a unique trace_id through every step of your pipeline for end-to-end debugging.

Every production Python script should emit at least three metrics: Start Time, Duration, and Rows Successfully Processed.