DEV Community

Cover image for Building Enterprise-Level Monitoring: From Prometheus to Grafana Dashboards
Aleksandr Murzin
Aleksandr Murzin

Posted on

Building Enterprise-Level Monitoring: From Prometheus to Grafana Dashboards

#prometheus #grafana #monitoring #python

Cover Image: Main Grafana dashboard with HTTP and performance metrics

Introduction

Once your web application hits production, the most critical question becomes: how is it performing right now? Logs tell you what happened, but you want to spot problems before users start complaining.

In this article, I'll share how I built a complete monitoring system for Peakline — a FastAPI application for Strava data analysis that processes thousands of requests daily from athletes worldwide.

What's Inside:

  • Metrics architecture (HTTP, API, business metrics)
  • Prometheus + Grafana setup from scratch
  • 50+ production-ready metrics
  • Advanced PromQL queries
  • Reactive dashboards
  • Best practices and pitfalls

Architecture: Three Monitoring Levels

Modern monitoring isn't just "set up Grafana and look at graphs." It's a well-thought-out architecture with several layers:

┌─────────────────────────────────────────────────┐ │ FastAPI Application │ │ ├── HTTP Middleware (auto-collect metrics) │ │ ├── Business Logic (business metrics) │ │ └── /metrics endpoint (Prometheus format) │ └──────────────────┬──────────────────────────────┘ │ scrape every 5s ┌──────────────────▼──────────────────────────────┐ │ Prometheus │ │ ├── Time Series Database (TSDB) │ │ ├── Storage retention: 200h │ │ └── PromQL Engine │ └──────────────────┬──────────────────────────────┘ │ query data ┌──────────────────▼──────────────────────────────┐ │ Grafana │ │ ├── Dashboards │ │ ├── Alerting │ │ └── Visualization │ └─────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Why This Stack?

Prometheus — the de-facto standard for metrics. Pull model, powerful PromQL query language, excellent Kubernetes integration.

Grafana — the best visualization tool. Beautiful dashboards, alerting, templating, rich UI.

FastAPI — async Python framework with native metrics support via prometheus_client.

Basic Infrastructure Setup

Docker Compose: 5-Minute Quick Start

First, let's spin up Prometheus and Grafana in Docker:

# docker-compose.yml version: '3.8' services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - "9090:9090" volumes: - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml - prometheus_data:/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention.time=200h' # 8+ days of history - '--web.enable-lifecycle' networks: - monitoring extra_hosts: - "host.docker.internal:host-gateway" # Access host machine grafana: image: grafana/grafana:latest container_name: grafana ports: - "3000:3000" environment: - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD} # Use .env! - GF_SERVER_ROOT_URL=/grafana # For nginx reverse proxy volumes: - grafana_data:/var/lib/grafana - ./monitoring/grafana/provisioning:/etc/grafana/provisioning depends_on: - prometheus networks: - monitoring volumes: prometheus_data: grafana_data: networks: monitoring: driver: bridge
Enter fullscreen mode Exit fullscreen mode

Key Points:

  • storage.tsdb.retention.time=200h — keep metrics for 8+ days (for weekly analysis)
  • extra_hosts: host.docker.internal — allows Prometheus to reach the app on the host
  • Volumes for data persistence

Prometheus Configuration 

# monitoring/prometheus.yml global: scrape_interval: 15s # How often to collect metrics evaluation_interval: 15s # How often to check alerts scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'webapp' static_configs: - targets: ['host.docker.internal:8000'] # Your app port scrape_interval: 5s # More frequent for web apps metrics_path: /metrics
Enter fullscreen mode Exit fullscreen mode

Important: scrape_interval: 5s for web apps is a balance between data freshness and system load. In production, typically 15-30s.

Grafana Datasource Provisioning

To avoid manual Prometheus setup in Grafana, use provisioning:

# monitoring/grafana/provisioning/datasources/prometheus.yml apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus:9090 isDefault: true editable: true
Enter fullscreen mode Exit fullscreen mode

Now Grafana automatically connects to Prometheus on startup.

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Level 1: HTTP Metrics

The most basic but critically important layer — monitoring HTTP requests. Middleware automatically collects metrics for all HTTP requests.

Metrics Initialization 

# webapp/main.py from prometheus_client import Counter, Histogram, CollectorRegistry, generate_latest, CONTENT_TYPE_LATEST from fastapi import FastAPI, Request from fastapi.responses import PlainTextResponse import time app = FastAPI(title="Peakline", version="2.0.0") # Create separate registry for metrics isolation registry = CollectorRegistry() # Counter: monotonically increasing value (request count) http_requests_total = Counter( 'http_requests_total', 'Total number of HTTP requests', ['method', 'endpoint', 'status_code'], # Labels for grouping  registry=registry ) # Histogram: distribution of values (execution time) http_request_duration_seconds = Histogram( 'http_request_duration_seconds', 'HTTP request duration in seconds', ['method', 'endpoint'], registry=registry ) # API call counters api_calls_total = Counter( 'api_calls_total', 'Total number of API calls by type', ['api_type'], registry=registry ) # Separate error counters http_errors_4xx_total = Counter( 'http_errors_4xx_total', 'Total number of 4xx HTTP errors', ['endpoint', 'status_code'], registry=registry ) http_errors_5xx_total = Counter( 'http_errors_5xx_total', 'Total number of 5xx HTTP errors', ['endpoint', 'status_code'], registry=registry )
Enter fullscreen mode Exit fullscreen mode

Middleware for Automatic Collection

The magic happens in middleware — it wraps every request:

@app.middleware("http") async def metrics_middleware(request: Request, call_next): start_time = time.time() # Execute request  response = await call_next(request) duration = time.time() - start_time # Path normalization: /api/activities/12345 → /api/activities/{id}  path = request.url.path if path.startswith('/api/'): parts = path.split('/') if len(parts) > 3 and parts[3].isdigit(): parts[3] = '{id}' path = '/'.join(parts) # Record metrics  http_requests_total.labels( method=request.method, endpoint=path, status_code=str(response.status_code) ).inc() http_request_duration_seconds.labels( method=request.method, endpoint=path ).observe(duration) # Track API calls  if path.startswith('/api/'): api_type = path.split('/')[2] if len(path.split('/')) > 2 else 'unknown' api_calls_total.labels(api_type=api_type).inc() # Track errors separately  status_code = response.status_code if 400 <= status_code < 500: http_errors_4xx_total.labels(endpoint=path, status_code=str(status_code)).inc() elif status_code >= 500: http_errors_5xx_total.labels(endpoint=path, status_code=str(status_code)).inc() return response
Enter fullscreen mode Exit fullscreen mode

Key Techniques:

  1. Path normalization — critically important! Without this, you'll get thousands of unique metrics for /api/activities/1, /api/activities/2, etc.

  2. Labels — allow filtering and grouping metrics in PromQL

  3. Separate error counters — simplifies alert writing

Metrics Endpoint 

@app.get("/metrics") async def metrics(): """Prometheus metrics endpoint""" return PlainTextResponse( generate_latest(registry), media_type=CONTENT_TYPE_LATEST )
Enter fullscreen mode Exit fullscreen mode

Now Prometheus can collect metrics from http://localhost:8000/metrics.

What We Get in Prometheus 

# Metrics format in /metrics endpoint: http_requests_total{method="GET",endpoint="/api/activities",status_code="200"} 1543 http_requests_total{method="POST",endpoint="/api/activities",status_code="201"} 89 http_request_duration_seconds_bucket{method="GET",endpoint="/api/activities",le="0.1"} 1234
Enter fullscreen mode Exit fullscreen mode

Level 2: External API Metrics

Web applications often integrate with external APIs (Stripe, AWS, etc.). It's important to track not only your own requests but also dependencies.

External API Metrics 

# External API metrics external_api_calls_total = Counter( 'external_api_calls_total', 'Total number of external API calls by endpoint type', ['endpoint_type'], registry=registry ) external_api_errors_total = Counter( 'external_api_errors_total', 'Total number of external API errors by endpoint type', ['endpoint_type'], registry=registry ) external_api_latency_seconds = Histogram( 'external_api_latency_seconds', 'External API call latency in seconds', ['endpoint_type'], registry=registry )
Enter fullscreen mode Exit fullscreen mode

API Call Tracking Helper

Instead of duplicating code everywhere you call the API, create a universal wrapper:

async def track_external_api_call(endpoint_type: str, api_call_func, *args, **kwargs): """ Universal wrapper for tracking API calls Usage: result = await track_external_api_call( 'athlete_activities', client.get_athlete_activities, athlete_id=123 ) """ start_time = time.time() try: # Increment call counter  external_api_calls_total.labels(endpoint_type=endpoint_type).inc() # Execute API call  result = await api_call_func(*args, **kwargs) # Record latency  duration = time.time() - start_time external_api_latency_seconds.labels(endpoint_type=endpoint_type).observe(duration) # Check for API errors (status >= 400)  if isinstance(result, Exception) or (hasattr(result, 'status') and result.status >= 400): external_api_errors_total.labels(endpoint_type=endpoint_type).inc() return result except Exception as e: # Record latency and error  duration = time.time() - start_time external_api_latency_seconds.labels(endpoint_type=endpoint_type).observe(duration) external_api_errors_total.labels(endpoint_type=endpoint_type).inc() raise e
Enter fullscreen mode Exit fullscreen mode

Usage in Code 

@app.get("/api/activities") async def get_activities(athlete_id: int): # Instead of direct API call:  # activities = await external_client.get_athlete_activities(athlete_id)  # Use wrapper with tracking:  activities = await track_external_api_call( 'athlete_activities', external_client.get_athlete_activities, athlete_id=athlete_id ) return activities
Enter fullscreen mode Exit fullscreen mode

Now we can see:

  • How many calls to each external API endpoint
  • How many returned errors
  • Latency for each call type

Level 3: Business Metrics

This is the most valuable part of monitoring — metrics that reflect actual application usage.

Business Metrics Types 

# === Authentication === user_logins_total = Counter( 'user_logins_total', 'Total number of user logins', registry=registry ) user_registrations_total = Counter( 'user_registrations_total', 'Total number of new user registrations', registry=registry ) user_deletions_total = Counter( 'user_deletions_total', 'Total number of user deletions', registry=registry ) # === File Operations === fit_downloads_total = Counter( 'fit_downloads_total', 'Total number of FIT file downloads', registry=registry ) gpx_downloads_total = Counter( 'gpx_downloads_total', 'Total number of GPX file downloads', registry=registry ) gpx_uploads_total = Counter( 'gpx_uploads_total', 'Total number of GPX file uploads', registry=registry ) # === User Actions === settings_updates_total = Counter( 'settings_updates_total', 'Total number of user settings updates', registry=registry ) feature_requests_total = Counter( 'feature_requests_total', 'Total number of feature requests', registry=registry ) feature_votes_total = Counter( 'feature_votes_total', 'Total number of votes for features', registry=registry ) # === Reports === manual_reports_total = Counter( 'manual_reports_total', 'Total number of manually created reports', registry=registry ) auto_reports_total = Counter( 'auto_reports_total', 'Total number of automatically created reports', registry=registry ) failed_reports_total = Counter( 'failed_reports_total', 'Total number of failed report creation attempts', registry=registry )
Enter fullscreen mode Exit fullscreen mode

Incrementing in Code 

@app.post("/api/auth/login") async def login(credentials: LoginCredentials): user = await authenticate_user(credentials) if user: # Increment successful login counter  user_logins_total.inc() return {"token": generate_token(user)} return {"error": "Invalid credentials"} @app.post("/api/activities/report") async def create_report(activity_id: int, is_auto: bool = False): try: report = await generate_activity_report(activity_id) # Different counters for manual and automatic reports  if is_auto: auto_reports_total.inc() else: manual_reports_total.inc() return report except Exception as e: failed_reports_total.inc() raise e
Enter fullscreen mode Exit fullscreen mode

Level 4: Performance and Caching

Cache Metrics

Cache is a critical part of performance. Need to track hit rate:

cache_hits_total = Counter( 'cache_hits_total', 'Total number of cache hits', ['cache_type'], registry=registry ) cache_misses_total = Counter( 'cache_misses_total', 'Total number of cache misses', ['cache_type'], registry=registry ) # In caching code: async def get_from_cache(key: str, cache_type: str = 'generic'): value = await cache.get(key) if value is not None: cache_hits_total.labels(cache_type=cache_type).inc() return value else: cache_misses_total.labels(cache_type=cache_type).inc() return None
Enter fullscreen mode Exit fullscreen mode

Background Task Metrics

If you have background tasks (Celery, APScheduler), track them:

background_task_duration_seconds = Histogram( 'background_task_duration_seconds', 'Background task execution time', ['task_type'], registry=registry ) async def run_background_task(task_type: str, task_func, *args, **kwargs): start_time = time.time() try: result = await task_func(*args, **kwargs) return result finally: duration = time.time() - start_time background_task_duration_seconds.labels(task_type=task_type).observe(duration)
Enter fullscreen mode Exit fullscreen mode

PromQL: Metrics Query Language

Prometheus uses its own query language — PromQL. Not SQL, but very powerful.

Basic Queries 

# 1. Just get metric (instant vector) http_requests_total # 2. Filter by labels http_requests_total{method="GET"} http_requests_total{status_code="200"} http_requests_total{method="GET", endpoint="/api/activities"} # 3. Regular expressions in labels http_requests_total{status_code=~"5.."} # All 5xx errors http_requests_total{endpoint=~"/api/.*"} # All API endpoints # 4. Time interval (range vector) http_requests_total[5m] # Data for last 5 minutes
Enter fullscreen mode Exit fullscreen mode

Rate and irate: Rate of Change

Counter constantly grows, but we need rate of change — RPS (requests per second):

# Rate - average rate over interval rate(http_requests_total[5m]) # irate - instantaneous rate (between last two points) irate(http_requests_total[5m])
Enter fullscreen mode Exit fullscreen mode

When to use what:

  • rate() — for alerts and trend graphs (smooths spikes)
  • irate() — for detailed analysis (shows peaks)

Aggregation with sum, avg, max 

# Total app RPS sum(rate(http_requests_total[5m])) # RPS by method sum(rate(http_requests_total[5m])) by (method) # RPS by endpoint, sorted sort_desc(sum(rate(http_requests_total[5m])) by (endpoint)) # Average latency avg(rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))
Enter fullscreen mode Exit fullscreen mode

Histogram and Percentiles

For Histogram metrics (latency, duration) use histogram_quantile:

# P50 (median) latency histogram_quantile(0.5, rate(http_request_duration_seconds_bucket[5m])) # P95 latency histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) # P99 latency (99% of requests faster than this) histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) # P95 per endpoint histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) by (endpoint)
Enter fullscreen mode Exit fullscreen mode

Complex Queries

1. Success Rate (percentage of successful requests) 

( sum(rate(http_requests_total{status_code=~"2.."}[5m])) / sum(rate(http_requests_total[5m])) ) * 100
Enter fullscreen mode Exit fullscreen mode

2. Error Rate (percentage of errors) 

( sum(rate(http_requests_total{status_code=~"4..|5.."}[5m])) / sum(rate(http_requests_total[5m])) ) * 100
Enter fullscreen mode Exit fullscreen mode

3. Cache Hit Rate 

( sum(rate(cache_hits_total[5m])) / (sum(rate(cache_hits_total[5m])) + sum(rate(cache_misses_total[5m]))) ) * 100
Enter fullscreen mode Exit fullscreen mode

4. Top-5 Slowest Endpoints 

topk(5, histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]) ) by (endpoint) )
Enter fullscreen mode Exit fullscreen mode

5. API Health Score (0-100) 

( ( sum(rate(external_api_calls_total[5m])) - sum(rate(external_api_errors_total[5m])) ) / sum(rate(external_api_calls_total[5m])) ) * 100
Enter fullscreen mode Exit fullscreen mode

Grafana Dashboards: Visualization

Now the fun part — turning raw metrics into beautiful and informative dashboards.

HTTP and Performance

Dashboard 1: HTTP & Performance

Panel 1: Request Rate

sum(rate(http_requests_total[5m]))
Enter fullscreen mode Exit fullscreen mode
  • Type: Time series
  • Color: Blue gradient
  • Unit: requests/sec
  • Legend: Total RPS

Panel 2: Success Rate

( sum(rate(http_requests_total{status_code=~"2.."}[5m])) / sum(rate(http_requests_total[5m])) ) * 100
Enter fullscreen mode Exit fullscreen mode
  • Type: Stat
  • Color: Green if > 95%, yellow if > 90%, red if < 90%
  • Unit: percent (0-100)
  • Value: Current (last)

Panel 3: Response Time (P50, P95, P99)

# P50 histogram_quantile(0.5, rate(http_request_duration_seconds_bucket[5m])) # P95 histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) # P99 histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
Enter fullscreen mode Exit fullscreen mode
  • Type: Time series
  • Unit: seconds (s)
  • Legend: P50, P95, P99

Panel 4: Errors by Type

sum(rate(http_requests_total{status_code=~"4.."}[5m])) by (status_code) sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (status_code)
Enter fullscreen mode Exit fullscreen mode
  • Type: Bar chart
  • Colors: Yellow (4xx), Red (5xx)

Panel 5: Request Rate by Endpoint

sort_desc(sum(rate(http_requests_total[5m])) by (endpoint))
Enter fullscreen mode Exit fullscreen mode
  • Type: Bar chart
  • Limit: Top 10

Dashboard 2: Business Metrics

This dashboard shows real product usage — what users do and how often.

Authentication and Users

Panel 1: User Activity (24h)

# Logins increase(user_logins_total[24h]) # Registrations increase(user_registrations_total[24h]) # Deletions increase(user_deletions_total[24h])
Enter fullscreen mode Exit fullscreen mode
  • Type: Stat
  • Layout: Horizontal

Panel 2: Downloads by Type

sum(rate({__name__=~".*_downloads_total"}[5m])) by (__name__)
Enter fullscreen mode Exit fullscreen mode
  • Type: Pie chart
  • Legend: Right side

Panel 3: Feature Usage Timeline

rate(gpx_fixer_usage_total[5m]) rate(search_usage_total[5m]) rate(manual_reports_total[5m])
Enter fullscreen mode Exit fullscreen mode
  • Type: Time series
  • Stacking: Normal

Dashboard 3: External API

Critical to monitor dependencies on external services — they can become bottlenecks.

External API

Panel 1: API Health Score

( sum(rate(external_api_calls_total[5m])) - sum(rate(external_api_errors_total[5m])) ) / sum(rate(external_api_calls_total[5m])) * 100
Enter fullscreen mode Exit fullscreen mode
  • Type: Gauge
  • Min: 0, Max: 100
  • Thresholds: 95 (green), 90 (yellow), 0 (red)

Panel 2: API Latency by Endpoint

histogram_quantile(0.95, rate(external_api_latency_seconds_bucket[5m])) by (endpoint_type)
Enter fullscreen mode Exit fullscreen mode
  • Type: Bar chart
  • Sort: Descending

Panel 3: Error Rate by Endpoint

sum(rate(external_api_errors_total[5m])) by (endpoint_type)
Enter fullscreen mode Exit fullscreen mode
  • Type: Bar chart
  • Color: Red

Variables: Dynamic Dashboards

Grafana supports variables for interactive dashboards:

Creating a Variable

  1. Dashboard Settings → Variables → Add variable
  2. Name: endpoint
  3. Type: Query
  4. Query: 
label_values(http_requests_total, endpoint)
Enter fullscreen mode Exit fullscreen mode

Using in Panels 

# Filter by selected endpoint sum(rate(http_requests_total{endpoint="$endpoint"}[5m])) # Multi-select sum(rate(http_requests_total{endpoint=~"$endpoint"}[5m])) by (endpoint)
Enter fullscreen mode Exit fullscreen mode

Useful Variables 

# Time interval Variable: interval Type: Interval Values: 1m,5m,10m,30m,1h # HTTP method Variable: method Query: label_values(http_requests_total, method) # Status code Variable: status_code Query: label_values(http_requests_total, status_code)
Enter fullscreen mode Exit fullscreen mode

Alerting: System Reactivity

Monitoring without alerts is like a car without brakes. Let's set up smart alerts.

Grafana Alerting

Alert 1: High Error Rate

( sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) * 100 > 1
Enter fullscreen mode Exit fullscreen mode
  • Condition: > 1 (more than 1% errors)
  • For: 5m (for 5 minutes)
  • Severity: Critical
  • Notification: Slack, Email, Telegram

Alert 2: High Latency

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
Enter fullscreen mode Exit fullscreen mode
  • Condition: P95 > 2 seconds
  • For: 10m
  • Severity: Warning

Alert 3: External API Down

sum(rate(external_api_errors_total[5m])) / sum(rate(external_api_calls_total[5m])) > 0.5
Enter fullscreen mode Exit fullscreen mode
  • Condition: More than 50% API errors
  • For: 2m
  • Severity: Critical

Alert 4: No Data

absent_over_time(http_requests_total[10m])
Enter fullscreen mode Exit fullscreen mode
  • Condition: No metrics for 10 minutes
  • Severity: Critical
  • Means: app crashed or Prometheus can't collect metrics

Best Practices: Battle-Tested Experience

1. Labels: Don't Overdo It

❌ Bad:

# Too detailed labels = cardinality explosion http_requests_total.labels( method=request.method, endpoint=request.url.path, # Every unique URL!  user_id=str(user.id), # Thousands of users!  timestamp=str(time.time()) # Infinite values! ).inc()
Enter fullscreen mode Exit fullscreen mode

✅ Good:

# Normalized endpoints + limited label set http_requests_total.labels( method=request.method, endpoint=normalize_path(request.url.path), # /api/users/{id}  status_code=str(response.status_code) ).inc()
Enter fullscreen mode Exit fullscreen mode

Rule: High-cardinality data (user_id, timestamps, unique IDs) should NOT be labels.

2. Naming Convention

Follow Prometheus naming conventions:

# Good names: http_requests_total # <namespace>_<name>_<unit> external_api_latency_seconds # Unit in name cache_hits_total # Clear it's a Counter  # Bad names: RequestCount # Not CamelCase api-latency # Don't use dashes request_time # Unit not specified
Enter fullscreen mode Exit fullscreen mode

3. Rate() Interval

Rate() interval should be minimum 4x larger than scrape_interval:

# If scrape_interval = 15s rate(http_requests_total[1m]) # 4x = 60s ✅ rate(http_requests_total[30s]) # 2x = poor accuracy ❌
Enter fullscreen mode Exit fullscreen mode

4. Histogram Buckets

Proper buckets are critical for accurate percentiles:

# Default (bad for latency): Histogram('latency_seconds', 'Latency') # [.005, .01, .025, .05, .1, ...]  # Custom buckets for web latency: Histogram( 'http_request_duration_seconds', 'Request latency', buckets=[.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10] )
Enter fullscreen mode Exit fullscreen mode

Principle: Buckets should cover the typical range of values.

5. Metrics Cost

Every metric costs memory. Let's calculate:

Memory = Series count × (~3KB per series) Series = Metric × Label combinations
Enter fullscreen mode Exit fullscreen mode

Example:

# 1 metric × 5 methods × 20 endpoints × 15 status codes = 1,500 series http_requests_total{method, endpoint, status_code} # 1,500 × 3KB = ~4.5MB for one metric!
Enter fullscreen mode Exit fullscreen mode

Tip: Regularly check cardinality:

# Top metrics by cardinality topk(10, count by (__name__)({__name__=~".+"}))
Enter fullscreen mode Exit fullscreen mode

Production Checklist

Before launching in production, check:

  • [ ] Retention policy configured (storage.tsdb.retention.time)
  • [ ] Disk space monitored (Prometheus can take a lot of space)
  • [ ] Backups configured for Grafana dashboards
  • [ ] Alerts tested (create artificial error)
  • [ ] Notification channels work (send test alert)
  • [ ] Access control configured (don't leave Grafana with admin/admin!)
  • [ ] HTTPS configured for Grafana (via nginx reverse proxy)
  • [ ] Cardinality checked (topk(10, count by (__name__)({__name__=~".+"})))
  • [ ] Documentation created (what metric is responsible for what)
  • [ ] On-call process defined (who gets alerts and what to do)

Real Case: Finding a Problem

Imagine: users complain about slow performance. Here's how monitoring helped find and fix the problem in minutes.

Step 1: Open Grafana → HTTP Performance Dashboard

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
Enter fullscreen mode Exit fullscreen mode

We see: P95 latency jumped from 0.2s to 3s.

Step 2: Check latency by endpoint

topk(5, histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) by (endpoint))
Enter fullscreen mode Exit fullscreen mode

Found: /api/activities — 5 seconds!

Step 3: Check external APIs

histogram_quantile(0.95, rate(external_api_latency_seconds_bucket[5m])) by (endpoint_type)
Enter fullscreen mode Exit fullscreen mode

External API athlete_activities — 4.8 seconds. There's the problem!

Step 4: Check error rate

rate(external_api_errors_total{endpoint_type="athlete_activities"}[5m])
Enter fullscreen mode Exit fullscreen mode

No errors, just slow. So the problem isn't on our side — external service is lagging.

Solution:

  • Add aggressive caching for external API (TTL 5 minutes)
  • Set up alert for latency > 2s
  • Add timeout to requests

Step 5: After deploy, verify

# Cache hit rate (cache_hits_total / (cache_hits_total + cache_misses_total)) * 100
Enter fullscreen mode Exit fullscreen mode

Hit rate 85% → latency dropped to 0.3s. Victory! 🎉

What's Next?

You've built a production-ready monitoring system. But this is just the beginning:

Next Steps:

  1. Distributed Tracing — add Jaeger/Tempo for request tracing
  2. Logging — integrate Loki for centralized logs
  3. Custom Dashboards — create dashboards for business (not just DevOps)
  4. SLO/SLI — define Service Level Objectives
  5. Anomaly Detection — use machine learning for anomaly detection
  6. Cost Monitoring — add cost metrics (AWS CloudWatch, etc.)

Useful Resources:

Conclusion

A monitoring system isn't "set it and forget it." It's a living organism that needs to evolve with your application. But the basic architecture we've built scales from startup to enterprise.

Key Takeaways:

  1. Three metric levels: HTTP (infrastructure) → API (dependencies) → Business (product)
  2. Middleware automates basic metrics collection
  3. PromQL is powerful — learn gradually
  4. Labels matter — but don't overdo cardinality
  5. Alerts are critical — monitoring without alerts is useless
  6. Document — in six months you'll forget what foo_bar_total means

Monitoring is a culture, not a tool. Start simple, iterate, improve. And your application will run stably, while you sleep peacefully 😴

About Peakline

This monitoring system was built for Peakline — a web application for Strava activity analysis. Peakline provides athletes with:

  • Detailed segment analysis with interactive maps
  • Historical weather data for every activity
  • Advanced FIT file generation for virtual races
  • Automatic GPX track error correction
  • Route planner

All these features require reliable monitoring to ensure quality user experience.

Questions? Leave them in the comments!

P.S. If you found this helpful — share with colleagues who might benefit!

About the Author

Solo developer building Peakline — tools for athletes. Athlete and enthusiast myself, believe in automation, observability, and quality code. Continuing to develop the project and share experience with the community in 2025.

Connect

  • 🌐 Peakline Website
  • 💬 Share your monitoring setup in comments
  • 📧 Questions? Drop a comment below!

Tags: #prometheus #grafana #monitoring #python #fastapi #devops #observability #sre #metrics #production

Top comments (0)