Prometheus at Scale: Surviving the Cardinality Cliff

The Day Prometheus Fell Over Prometheus memory usage spiked from 8GB to 32GB overnight. OOM-killed. Monitoring was down for 20 minutes while we scrambled. The cause? Someone added a request_path label to an HTTP metric. With 50,000 unique paths, that one label created 500,000 new time series. This