Senior Data Engineer - Databricks Migration Specialist
Overview:
We are seeking an experienced Senior Data Engineer with deep expertise in Databricks and SQL to lead our platform migration initiative. This role demands exceptional proficiency in both the Databricks ecosystem and advanced SQL development to successfully modernize our data infrastructure.
Key Responsibilities:
- Lead end-to-end migration of SQL-based data platforms to Databricks, ensuring optimal performance and scalability.
- Design and architect enterprise-grade data solutions on Databricks using Delta Lake and Medallion Architecture.
- Develop complex ETL/ELT pipelines leveraging PySpark, SparkSQL, and Python on Databricks.
- Translate advanced SQL code (complex queries, stored procedures, functions, triggers, views) into highly optimized PySpark/SparkSQL.
- Perform SQL query optimization and refactoring for distributed computing on Databricks clusters.
- Implement Delta Lake advanced features including Change Data Capture, Time Travel, and Z-Ordering.
- Orchestrate data workflows using Azure Data Factory with Databricks integration.
- Tune and optimize Databricks cluster configurations, job performance, and cost management.
- Build automated CI/CD pipelines using Azure DevOps for Databricks deployments.
- Establish data engineering best practices, coding standards, and governance frameworks.
- Mentor team members on Databricks platform capabilities and SQL optimization techniques.
- Create comprehensive technical documentation and conduct knowledge transfer sessions.
Required Skills:
- 6+ years of expert-level SQL experience - advanced query writing, stored procedures, performance tuning, indexing strategies, execution plan analysis, database optimization.
- 3+ years of extensive Databricks experience - workspace management, cluster configuration, Delta Lake, Unity Catalog, Databricks SQL, notebooks, workflows.
- 3+ years of strong PySpark and Apache Spark hands-on development experience.
- Proven success delivering large-scale SQL to Databricks migration projects.
- Strong Python programming skills for data engineering and automation.
- 2+ years working with Azure Data Factory for orchestration and pipeline integration with Databricks.
- Experience with Azure DevOps for Git version control, CI/CD pipelines, and automated deployments.
- Deep understanding of data warehouse concepts, dimensional modeling, and data lake architectures.
- Expertise in Spark performance optimization (partitioning, bucketing, caching, broadcast joins).
- Strong experience with Azure cloud services (Azure SQL, ADLS, Key Vault, Azure Monitor).
- Excellent analytical, problem-solving, and troubleshooting abilities.
- Strong leadership, communication, and mentoring skills.
Preferred Qualifications:
- Databricks Certified Data Engineer Professional.
- Azure Data Engineer Associate certification.
- Experience with SQL Server, Oracle, or other enterprise RDBMS platforms.
- Knowledge of Databricks SQL Analytics and BI tool integrations.
- Experience with streaming data (Structured Streaming, Event Hubs, Kafka).
- Familiarity with Databricks Unity Catalog for data governance.
- Infrastructure-as-code experience (Terraform, ARM templates).
