Role Overview: - Rebuild T-SQL logic as typed, tested PySpark pipelines. - Prove that new pipelines match legacy outputs. - Build FastAPI services that run and report on the pipeline jobs. - Add provider-neutral LLM features into the data workflows. Pipeline Responsibilities: - Build Spark and PySpark pipelines on Delta Lake. - Re-express T-SQL logic in Spark SQL. - Ensure schema evolution, time travel, and idempotent writes using Delta Lake. - Optimize partitioning and performance for production Spark jobs. API and Service Development: - Build REST APIs with FastAPI for pipeline jobs. - Implement request validation, authentication, and job-status semantics. - Ensure idempotency in service architecture. Code Quality and Standards: - Write type-hinted, tested Python code using Python 3.12, Pydantic, and static analysis tools like pyright or mypy. - Implement automated testing with pytest, including fixtures, markers, and tiered test suites. - Utilize modern packaging tools such as uv or Poetry. - Use Git and CI discipline with PR-driven workflows, including ruff and linting gates. - Work against design documents and Architecture Decision Records. - Maintain documentation alongside every change. Data Migration and Validation: - Validate migrated pipelines against legacy systems with parity evidence. - Read T-SQL and reproduce its semantics faithfully in set-based logic. - Manage migration of legacy SQL Server or Azure Synapse estates to Spark. Development Environment: - Use Docker-based development against Compose stacks. - Work autonomously from written requirements and specifications. Preferred Skills and Experience: - Integrate provider-neutral AI features and validate structured outputs. - Implement data governance and privacy engineering such as PII tokenization and GDPR-style data handling. - Maintain Lakehouse platform components like Hive Metastore, Trino, or Apache Ranger. - Manage Azure data estate components including ADLS Gen2, Synapse, or Key Vault. - Utilize observability tooling such as structured logging, OpenTelemetry, and OpenLineage.

Python/Spark/AI Developer

Emma

Responsibilities

Skills wanted:

Language(s) required:

Joeri Vertongen Verified

About Lingua Resources:

mission:

Admin access needed