From Monitoring to Observability: Our Ultra-Marathon to a Cloud-Native Platform

Key Actionable Insights

1

Implement dynamic configuration management for monitoring agents instead of relying on static configuration files. By connecting your monitoring agents to a centralized inventory service and using mechanisms like Last-Modified timestamp checks, agents can automatically adapt to infrastructure changes without manual edits or redeployments.

This is especially critical for large-scale environments where devices are frequently added, removed, or reassigned across regions. Uber's approach uses ServiceNow as source of truth synced to Elasticsearch, with Telegraf agents polling for configuration updates.

2

Build an intelligent alert ingestion layer that de-duplicates, groups, and enriches alerts before routing them to communication channels. Rather than forwarding every alert directly to Slack or PagerDuty, process alerts asynchronously to filter noise, correlate related events, and update existing notifications as alert states change.

Alert fatigue is a major operational challenge. Uber uses FastAPI, Celery, and Redis for async alert processing, posting alerts once and editing them as states change rather than creating duplicate notifications.

3

Deploy monitoring infrastructure globally with probes geographically close to the devices being monitored. This ensures accurate, low-latency measurements and avoids misleading data caused by network distance between monitoring systems and their targets.

Uber deploys across USC, EMEA, and APAC regions, using regional parameters to filter which devices each Telegraf agent polls, enabling workload redistribution across regions.

4

Centralize all alert data—both firing and resolved—into a searchable index to create a historical dataset that enables AIOps capabilities. This historical context allows AI engines to identify patterns, correlate incidents, and suggest root causes based on past behavior.

Uber routes all alerts from Prometheus, MQTT sensors, and network APIs into a single Elasticsearch index, creating a continuous trail of operational awareness that engineers can query and that AI systems can analyze.

5

Choose an open-source observability stack over proprietary monolithic monitoring tools to gain flexibility, avoid vendor lock-in, and eliminate significant licensing costs. Prioritize tools that handle specific functions well (collection, storage, visualization, search) rather than a single tool that does everything.

Uber saved hundreds of thousands of dollars in recurring licensing fees by adopting Telegraf, Prometheus, Thanos, Grafana, Kibana, and Elasticsearch, while gaining better extensibility through API-driven integrations.

6

Use tag-based filtering (region, site, device type) in your monitoring configuration to enable fine-grained control over what each collector monitors. This allows you to roll out new collectors or plugin changes globally without redeploying agents.

Uber's Dynamic Config App uses URL parameters like /v1/snmp/cisco?region=EMEA to filter device polling assignments, enabling instant redistribution of workloads across their global infrastructure.