Senior Software Engineer
Datadog
Jun 2015 - Feb 2019 (3 years 9 months)
• Built and maintained data processing workflows using Apache Spark and Python to support Datadog's infrastructure monitoring platform, handling millions of metrics and events daily from customer environments. • Implemented monitoring and alerting for batch data pipelines using Luigi workflow management, ensuring reliable data delivery and reducing pipeline failures by 40% through proactive error detection. • Contributed to the development of log aggregation features following the Mortar Data acquisition (2015), writing Python scripts to parse and normalize log data from diverse customer sources into standardized formats. • Optimized data storage queries and indexing strategies in PostgreSQL database and Cassandra database, improving da