Data Engineer
FourNest Talent
Jul 2023 - Current (3 years 2 months)
Designed and maintained end-to-end ETL/ELT pipelines using Azure Data Factory, ingesting data from SQL databases, APIs, flat files, and ERP systems into Azure Data Lake Storage (ADLS). Built scalable batch and streaming data transformations using Databricks (PySpark, Spark SQL) to clean, standardize, and enrich large datasets. Implemented Delta Lake tables following Medallion Architecture (Bronze, Silver, Gold) to ensure ACID compliance, versioning, and reliable downstream consumption. Developed incremental data loads and optimized Spark jobs using partitioning, caching, and parallel processing to improve performance and reduce compute costs. Enabled analytics, machine learning, and AI workflows by delivering curated, feature-ready datasets