dance-lessons-coach/adr/0007-opentelemetry-integration.md

Integrate OpenTelemetry for distributed tracing

• Status: Accepted
• Deciders: Gabriel Radureau, AI Agent
• Date: 2026-04-04

Context and Problem Statement

We needed to add observability to dance-lessons-coach that provides:

• Distributed tracing capabilities
• Performance monitoring
• Request flow visualization
• Integration with existing monitoring systems
• Minimal impact on application performance

Decision Drivers

• Need for distributed tracing in microservices architecture
• Desire for performance monitoring
• Requirement for request flow visualization
• Need for integration with monitoring tools
• Desire for minimal performance impact

Considered Options

• OpenTelemetry - CNCF standard for observability
• Jaeger client - Direct Jaeger integration
• Zipkin - Alternative tracing system
• Custom solution - Build our own tracing

Decision Outcome

Chosen option: "OpenTelemetry" because it provides industry-standard observability, good performance, flexibility for multiple backends, and is becoming the standard for distributed tracing.

Pros and Cons of the Options

OpenTelemetry

• Good, because CNCF standard with broad industry adoption
• Good, because supports multiple tracing backends (Jaeger, Zipkin, etc.)
• Good, because good performance characteristics
• Good, because active development and community
• Good, because vendor-neutral
• Bad, because more complex setup
• Bad, because larger dependency footprint

Jaeger client

• Good, because direct integration with Jaeger
• Good, because simpler setup
• Bad, because vendor-locked to Jaeger
• Bad, because less flexible for future changes

Zipkin

• Good, because established tracing system
• Good, because good ecosystem
• Bad, because less feature-rich than OpenTelemetry
• Bad, because declining popularity

Custom solution

• Good, because tailored to our needs
• Good, because no external dependencies
• Bad, because time-consuming to develop
• Bad, because need to maintain ourselves
• Bad, because likely less feature-rich

Implementation Approach

Middleware-only approach

We chose a middleware-only approach using otelhttp.NewHandler rather than manual instrumentation:

// In pkg/server/server.go
func (s *Server) getAllMiddlewares() []func(http.Handler) http.Handler {
    middlewares := []func(http.Handler) http.Handler{
        middleware.StripSlashes,
        middleware.Recoverer,
    }

    if s.withOTEL {
        middlewares = append(middlewares, func(next http.Handler) http.Handler {
            return otelhttp.NewHandler(next, "")
        })
    }

    return middlewares
}

Benefits of middleware approach

• Clean separation: Tracing logic separate from business logic
• Consistent instrumentation: All endpoints automatically traced
• Easy to enable/disable: Single configuration flag
• Maintainable: No tracing boilerplate in service code
• Upgradable: Easy to change tracing implementation

Configuration

# config.yaml
telemetry:
  enabled: true
  otlp_endpoint: "localhost:4317"
  service_name: "dance-lessons-coach"
  insecure: true
  sampler:
    type: "parentbased_always_on"
    ratio: 1.0

Jaeger Integration

# Start Jaeger with OTLP support
docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

# Start server with OpenTelemetry
DLC_TELEMETRY_ENABLED=true ./scripts/start-server.sh start

# View traces at http://localhost:16686

Links

• OpenTelemetry GitHub
• OpenTelemetry Documentation
• Jaeger Documentation
• OTLP Specification

Sampler Types Supported

• always_on - Sample all traces
• always_off - Sample no traces
• traceidratio - Sample based on trace ID ratio
• parentbased_always_on - Sample based on parent span (always on)
• parentbased_always_off - Sample based on parent span (always off)
• parentbased_traceidratio - Sample based on parent span with ratio

Performance Considerations

• OpenTelemetry adds minimal overhead when disabled
• Sampling can be used to reduce overhead in production
• Tracing data is sent asynchronously to minimize impact
• Context propagation is efficient using Go's context package