E-commerce Data Pipeline Optimization
Apr 2023 - Sep 2023 (6 months)
• Developed an end-to-end data engineering pipeline to process e-commerce data, including product details, transactions, and customer data, using Azure Data Factory, Databricks, and SQL.
• Designed and deployed a data pipeline that ingested and processed over 10 GB of e-commerce data daily, ensuring timely availability of data for analysis.
•
Fine-tuned data transformation jobs, reducing execution time by 40% and ensuring high data quality for downstream analytics.