Databricks Data Engineer at Vaspire Technologies Inc.

Job Overview:Databrick data engineer owns the end-to-end migration strategy, target architecture design and technical execution of moving legacy ETL. He will establish migration standard, optimize pyspark pipelines, orchestration complex data workflows and deploy proper tools to ensure a seamless, high performing transition from legacy systems.Job DescriptionResponsibilities:Design the target Databrick lake house architecture utilizing Delta Lake, Photon and Unity CatalogEstablish global code refactoring standard, optimization benchmarks and Pyspark best practices.Resolve highly complex dependency mappings and architect seamless, zero downtime dual ran strategies.Lead the technical deployment and integration of specialized migration accelerators.Hands on Engineering & Optimization:Review automated output from migration tool and manually refactor complex logic into high performing Pyspark NotebooksEliminate legacy anti pattern such as massive row by row processing and inefficient lookups.Optimize Pyspark code performance using advances Spark feature including Z-ordering, Partitioning and caching.Build robust Databrick Workflows and orchestrate complex DAGs based on comprehensive source lineage.Technical Skill and Competencies:Core Platforms: Databrick, Delta lake, Unity Catalog, Photon, DataStageLanguages & Frameworks: Pyspark, Python, SQL, Shell ScriptingCloud & DevOps: AWS Alongside, CI/CD deployment pipelines:Orchestration: Apache, Airflows, Databrick workflows.

Databricks Data Engineer

Emma

Responsibilities

Skills wanted:

Language(s) required:

Aamir Sid

Aamir Sid

About Vaspire Technologies Inc.:

vaspiretech.com/

Admin access needed