talk-data.com

Services should be tracking those anyway (e.g. how fast does a service respond, how many errors does it generate). When deploying a new version, we're currently (more or less) instantly swapping out the application for everyone. With the Canary Releases, we're starting to roll out a new version to a certain percentage of users and checking the SLOs to automatically.

Learn how Reddit uses a custom monitoring operator to manage Thanos and Prometheus to scale their metrics deployment beyond 45 million samples per second and 600 million active series. To achieve this they run thousands of Prometheus instances of varying sizes managed by their internally developed Kubernetes controller. They use Thanos for long-term storage and global single pane of glass querying across this massive deployment. Learn about the operator, other tools they've developed, and the challenges they've faced along the way.