Venkata Sai Pavan Sr Data Engineer

Jobs 0% verified

H

Sr. Data Engineer

Hubbell, Inc.

Jun 2023 - Current (3 years 3 months)

• Spearheaded the implementation of Azure Data Factory and Databricks integration to engineer data pipelines, extracting data from 100+ blobs and seamlessly mounting it into Delta Lake bronze tables, driving a remarkable 30% reduction in data processing time. • Enhanced the existing multihop/medallion architecture by introducing an additional layer of data processing in Delta Lake, resulting in a significant improvement in data quality and data profiling accuracy, with a 25% reduction in data inconsistencies and a 40% increase in data profiling efficiency. • Implemented PySpark transformations on silver tables to optimize data transformations, effectively loading the processed data into gold tables for streamlined BI reporting. Resulting
Data Engineer Intern

Securonix

Jun 2022 - Aug 2022 (3 months)

• Automated Cron jobs (ETL) with Glue Data crawlers, using scheduled and conditional triggers, improving job efficiency by 25% and enabling timely data loading to S3. Utilized Athena to analyze data for retrieving EPS statistics. • Transformed and enriched raw data from Kafka brokers to depersonalized data using PySpark on AWS EMR, reducing job runtime by 75%. Stored transformed data in Hive tables for faster risk and threat model development. • Leveraged Spark SQL on Databricks integrated with AWS to perform data validation checks (CDC) for the files which are more than 1TB in size present in S3 after successful job completions. • Orchestrated AWS infrastructure with SQS, SNS, Lambda, and step functions in clients' VPC, to load producti
Data Engineer

Tata Consultancy Services

May 2019 - Jul 2021 (2 years 3 months)
T

Tata Consulatancy services

Education 0% verified

Venkata Sai Pavan

About

Detail

Timeline

Résumé