Senior Site Reliability Engineer
Chainlink Labs
Apr 2024 - Current (2 years 5 months)
Drive ongoing modernization of the observability platform towards an OTEL based VictoriaMetrics/Traces/Logs stack, designing ingestion paths, retention strategy, query compatibility and alerting validation; while maintaining production telemetry continuity and minimizing work on product teams.
Own, scale and improve company-wide observability platform (Prometheus, Loki, Thanos, OTEL, Alertmanager), ingesting 1M+ metric samples/sec and 650K log lines/sec across 3 environments and 18 clusters, with 1.5PB+ historical data in S3; defined SLIs (metric ingestion success rate, query error rate, alert evaluation latency) and enforced >=99.99%
availability SLO.
Lead cloud cost savings initiatives for the entire Data Platform pillar in both AWS and