Data Engineer & Cloud Consultant
CIPPlanner
Jul 2023 - Oct 2024 (1 year 4 months)
• Orchestrated complex ETL/ELT workflows as a technical consultant, enabling data integration across multiple cloud and enterprise sources.
• Architected Lakehouse environments on Databricks using Delta Lake and Apache Iceberg to support scalable and flexible data consumption.
• Developed high-performance transformation pipelines in Databricks using PySpark and Spark SQL for efficient large-scale processing.
• Automated end-to-end data pipelines across ADF and Databricks, implementing advanced scheduling, monitoring, and reliability controls.
• Implemented CI/CD pipelines with Azure DevOps to automate the deployment of Databricks notebooks and ADF components.
• Designed real-time ingestion pipelines using Azure Event Hubs and Spark Structur