Chandra Thatiparthi Data Engineer

Data Engineer

TransUnion

Feb 2022 - Current (4 years 3 months)

• Engineered real-time data streaming solutions using Apache Kafka for ingestion and processing of streaming data using Amazon Kinesis. • Implemented PySpark transformations to process and manipulate streaming data. Configured Spark on Amazon EMR. • Engineered and orchestrated workflow with complex Airflow DAGs and automated the pipelines with Lambda and CloudFormation. • Integrated Airflow with SQS and SNS, leveraging Lambda functions to respond to real-time data events and triggers effectively. • Deployed and managed Docker containers encapsulating Spark Streaming applications, Python scripts, and microservices on ECS clusters. • Implemented a comprehensive metadata management system to capture data lineage, definitions, and usage ac

Data Engineer

British Petroleum

Jun 2021 - Feb 2022 (9 months)

• Developed a serverless data application from scratch that takes cross-country oil financial data from different regions and performs transformations based on region-specific business logic and loaded the data in parquet format to AWS S3. • Extracted data from multiple sources systems and created tables/schemas in the Glue Catalog by creating Glue Crawlers • Automated using AWS Step Functions, Cloudformation, Lambda, and CI/CD with Azure devops. Reduction of manual effort by at least 50%. • Built pyspark scripts and transformation using data frames and spark-sql for data aggregation, queries, and writing back into S3. • Utilized AWS serverless services like Step Function, Lambda, Glue, Redshift Spectrum, Athena, and Cloudwatch. • Clos

Data Engineer Intern

Virgin Pulse

May 2020 - Jan 2021 (9 months)

• Developed re-usable SQL scripts in PostgreSQL and AWS Redshift to generate complex financial reports. • Outlined analysis and insights of reports for around 80 million records by processing SQL scripts on Redshift for JIRA tickets. • Automated their reporting mechanism using Python and JSON to dynamically generate reports. • Responsible for Performance Tuning of Redshift SQL queries for generating reports using the automated framework RAF. • Reduced the amount of monthly JIRA tickets, by 4%, by frequently coordinating with Engineering teams, Business analysts, and CSMs.

Data Engineer

Tata Consultancy Services

May 2018 - Aug 2019 (1 year 4 months)

• Designed, Developed and Deployed reports in MS SQL Server environment using SSRS 2008. • Created many complex Stored Procedures/Functions and used them in Reports directly to generate reports on the fly. • Recognized performance bottlenecks in SSIS packages by in-depth analysis of ad-hoc T-SQL queries, stored procs, and functions. • Enhanced ETL processes for optimal performance by fine-tuning of ETL procedures resulting in reduction in processing times. • Implemented data quality checks within the data warehousing environment to ensure the consistency, accuracy, and integrity of data.

Chandra Thatiparthi

About

Detail

Timeline

Résumé