DEV Community

Jones Charles
Jones Charles

Posted on

Building Observable Microservices: Distributed Tracing with Jaeger and GoFrame

#go #microservices #jaeger #tracing

Ever found yourself lost in a maze of microservices, wondering where that request disappeared to? 🤔 You're not alone! In this guide, I'll show you how to implement distributed tracing in your Go applications using Jaeger and GoFrame. By the end, you'll be able to track requests across your entire system like a pro! 🚀

What We'll Cover 📋

  • Setting up Jaeger with Docker
  • Integrating Jaeger with GoFrame
  • Creating and managing traces
  • Handling errors gracefully
  • Visualizing and analyzing traces

Prerequisites

  • Basic knowledge of Go and microservices
  • Docker installed on your machine
  • A GoFrame project (or willingness to start one!)

Getting Started with Jaeger 🐋

First things first, let's get Jaeger up and running. The easiest way is using Docker:

docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ -p 5775:5775/udp \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 14268:14268 \ -p 14250:14250 \ -p 9411:9411 \ jaegertracing/all-in-one:1.21
Enter fullscreen mode Exit fullscreen mode

This command launches a complete Jaeger setup in one container. Pretty neat, right? 👌

Setting Up the Tracer in GoFrame 🔧

Let's dive into a complete setup example. Here's how to configure Jaeger with different sampling strategies and options:

First, grab the Jaeger client library:

go get github.com/uber/jaeger-client-go
Enter fullscreen mode Exit fullscreen mode

Now, let's set up our tracer. Here's a simple initialization:

package main import ( "github.com/opentracing/opentracing-go" "github.com/uber/jaeger-client-go" "github.com/uber/jaeger-client-go/config" ) func main() { // Get config from environment cfg, _ := config.FromEnv() // Create the tracer tracer, closer, _ := cfg.NewTracer(config.Logger(jaeger.StdLogger)) defer closer.Close() // Set as global tracer opentracing.SetGlobalTracer(tracer) // Start your server... } // A more detailed configuration example func initJaeger(service string) (opentracing.Tracer, io.Closer, error) { cfg := &config.Configuration{ ServiceName: service, Sampler: &config.SamplerConfig{ Type: "const", Param: 1, }, Reporter: &config.ReporterConfig{ LogSpans: true, LocalAgentHostPort: "localhost:6831", BufferFlushInterval: 1 * time.Second, QueueSize: 1000, }, Tags: []opentracing.Tag{ {Key: "environment", Value: "development"}, {Key: "version", Value: "1.0.0"}, }, } tracer, closer, err := cfg.NewTracer( config.Logger(jaeger.StdLogger), config.ZipkinSharedRPCSpan(true), ) if err != nil { return nil, nil, err } return tracer, closer, nil }
Enter fullscreen mode Exit fullscreen mode

Complete Tracing Setup Example 🎯

Let's look at a complete example of how to set up tracing in your application:

package main import ( "github.com/gogf/gf/v2/frame/g" "github.com/gogf/gf/v2/net/ghttp" "github.com/opentracing/opentracing-go" ) type App struct { Server *ghttp.Server Tracer opentracing.Tracer } func NewApp() (*App, error) { // Initialize Jaeger tracer, closer, err := initJaeger("my-service") if err != nil { return nil, err } defer closer.Close() // Create server server := g.Server() app := &App{ Server: server, Tracer: tracer, } // Register middleware and routes server.Use(app.TracingMiddleware) server.Group("/api", func(group *ghttp.RouterGroup) { group.POST("/orders", app.HandleOrder) group.GET("/orders/:id", app.GetOrder) }) return app, nil } // Complete example of a traced HTTP client func (app *App) makeTracedRequest(parentSpan opentracing.Span, url string) error { // Create a child span span := app.Tracer.StartSpan( "http_request", opentracing.ChildOf(parentSpan.Context()), ) defer span.Finish() // Create request req, err := http.NewRequest("GET", url, nil) if err != nil { span.SetTag("error", true) span.LogKV("event", "error", "message", err.Error()) return err } // Inject tracing headers carrier := opentracing.HTTPHeadersCarrier(req.Header) err = app.Tracer.Inject(span.Context(), opentracing.HTTPHeaders, carrier) if err != nil { span.SetTag("error", true) span.LogKV("event", "error", "message", "failed to inject tracing headers") return err } // Make the request client := &http.Client{} resp, err := client.Do(req) if err != nil { span.SetTag("error", true) span.LogKV("event", "error", "message", err.Error()) return err } defer resp.Body.Close() // Add response info to span span.SetTag("http.status_code", resp.StatusCode) return nil }
Enter fullscreen mode Exit fullscreen mode

Creating Your First Trace 📝

Let's create a middleware to trace all incoming requests:

func TracingMiddleware(r *ghttp.Request) { // Extract any existing trace from headers spanCtx, _ := opentracing.GlobalTracer().Extract( opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(r.Request.Header), ) // Start a new span span := opentracing.GlobalTracer().StartSpan( r.URL.Path, opentracing.ChildOf(spanCtx), ) defer span.Finish() // Pass the span through headers opentracing.GlobalTracer().Inject( span.Context(), opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(r.Request.Header), ) // Add to request context r.SetCtx(opentracing.ContextWithSpan(r.Context(), span)) r.Middleware.Next() }
Enter fullscreen mode Exit fullscreen mode

Adding Business Logic Traces 💼

Now for the fun part - tracing your actual business logic:

func ProcessOrder(r *ghttp.Request) { // Get the current span span := opentracing.SpanFromContext(r.Context()) // Add some business context span.SetTag("order_id", "12345") // Log important events span.LogKV("event", "order_received") // Your business logic here... processPayment() updateInventory() sendConfirmation() span.LogKV("event", "order_completed") }
Enter fullscreen mode Exit fullscreen mode

Error Handling Like a Pro 🛠️

Let's make our error handling more traceable:

// Define custom errors type MyError struct { Code int Message string } func (e *MyError) Error() string { return fmt.Sprintf("error code: %d, message: %s", e.Code, e.Message) } // Error handling in your handlers func HandleOrder(r *ghttp.Request) { span := opentracing.SpanFromContext(r.Context()) err := processOrder() if err != nil { // Mark the span as failed span.SetTag("error", true) span.SetTag("error.code", err.(*MyError).Code) // Log detailed error info span.LogKV( "event", "error", "message", err.Error(), "stack", string(debug.Stack()), ) // Handle the error appropriately r.Response.WriteJson(g.Map{ "error": err.Error(), }) return } }
Enter fullscreen mode Exit fullscreen mode

Viewing Your Traces 👀

Once everything is set up, you can view your traces at http://localhost:16686. The Jaeger UI lets you:

  • Search for traces across services
  • View detailed timing information
  • Analyze error patterns
  • Export traces for further analysis

Pro Tips 💡

  1. Use Meaningful Span Names: Instead of generic names like "process", use descriptive names like "order_processing" or "payment_validation".

  2. Add Relevant Tags: Tags help filter and analyze traces. Add tags for things like:

    • User IDs
    • Request IDs
    • Environment information
    • Business-specific identifiers

  3. Log Key Events: Use LogKV to mark important points in your process:

 span.LogKV("event", "cache_miss", "key", "user:123")
Enter fullscreen mode Exit fullscreen mode

Common Pitfalls to Avoid ⚠️

  1. Memory Leaks: Always remember to call span.Finish()
  2. Over-instrumentation: Don't trace everything; focus on important operations
  3. Missing Context: Always propagate context through your service calls

Advanced Troubleshooting Guide 🔍

1. Common Issues and Solutions

Missing Traces 

// Problem: Traces not showing up in Jaeger UI // Solution: Check sampling configuration cfg := &config.Configuration{ Sampler: &config.SamplerConfig{ Type: "const", // Try different sampling strategies Param: 1, // 1 = sample all requests }, } // Verify spans are being created span := opentracing.SpanFromContext(ctx) if span == nil { // No span in context - check your middleware log.Println("No span found in context") }
Enter fullscreen mode Exit fullscreen mode

Context Propagation Issues 

// Problem: Broken trace chains // Solution: Properly propagate context through your application // Wrong ❌ func (s *Service) ProcessOrder(orderID string) error { // Starting new trace chain span := tracer.StartSpan("process_order") defer span.Finish() // ... processing } // Correct ✅ func (s *Service) ProcessOrder(ctx context.Context, orderID string) error { span, ctx := opentracing.StartSpanFromContext(ctx, "process_order") defer span.Finish() // Pass ctx to other functions return s.updateInventory(ctx, orderID) }
Enter fullscreen mode Exit fullscreen mode

Performance Issues 

// Problem: Too many spans affecting performance // Solution: Use batch processing for spans cfg := &config.Configuration{ Reporter: &config.ReporterConfig{ QueueSize: 1000, // Buffer size BufferFlushInterval: 1 * time.Second, LogSpans: true, // Set to false in production }, }
Enter fullscreen mode Exit fullscreen mode

2. Debugging Tools 

// Debug span creation func debugSpan(span opentracing.Span) { // Get span context spanContext, ok := span.Context().(jaeger.SpanContext) if !ok { log.Println("Not a Jaeger span") return } // Print span details log.Printf("Trace ID: %s", spanContext.TraceID()) log.Printf("Span ID: %s", spanContext.SpanID()) log.Printf("Parent ID: %s", spanContext.ParentID()) } // Monitor span metrics type SpanMetrics struct { TotalSpans int64 ErrorSpans int64 AverageLatency time.Duration } func collectSpanMetrics(span opentracing.Span) *SpanMetrics { metrics := &SpanMetrics{} // Add your metric collection logic if span.BaggageItem("error") != "" { atomic.AddInt64(&metrics.ErrorSpans, 1) } return metrics }
Enter fullscreen mode Exit fullscreen mode

3. Best Practices for Problem Resolution

1. Validate Configuration 

func validateJaegerConfig(cfg *config.Configuration) error { if cfg.ServiceName == "" { return errors.New("service name is required") } if cfg.Reporter.LocalAgentHostPort == "" { return errors.New("reporter host:port is required") } return nil }
Enter fullscreen mode Exit fullscreen mode

2. Implement Health Checks 

func jaegerHealthCheck() error { span := opentracing.GlobalTracer().StartSpan("health_check") defer span.Finish() carrier := opentracing.HTTPHeadersCarrier{} err := opentracing.GlobalTracer().Inject( span.Context(), opentracing.HTTPHeaders, carrier, ) if err != nil { return fmt.Errorf("jaeger injection failed: %v", err) } return nil }
Enter fullscreen mode Exit fullscreen mode

3. Monitor Trace Quality 

type TraceQuality struct { MissingParentSpans int BrokenChains int HighLatencyTraces int } func monitorTraceQuality(span opentracing.Span) *TraceQuality { quality := &TraceQuality{} // Check for parent span if span.BaggageItem("parent_id") == "" { quality.MissingParentSpans++ } // Check latency if duration, ok := span.BaggageItem("duration"); ok { if d, err := time.ParseDuration(duration); err == nil { if d > 1*time.Second { quality.HighLatencyTraces++ } } } return quality }
Enter fullscreen mode Exit fullscreen mode

Wrapping Up 🎉

Distributed tracing with Jaeger and GoFrame gives you x-ray vision into your microservices. You can:

  • Track requests across services
  • Identify performance bottlenecks
  • Debug issues faster
  • Understand system behavior

What's Next?

  • Explore Jaeger sampling strategies
  • Add metrics and logging
  • Implement trace-based alerts

Found this helpful? Follow me for more Go tips and tricks! And don't forget to drop a comment if you have questions or suggestions! 🚀

Resources:

Top comments (0)