For companies:
For professionals:
For everyone:
For developers:
Responsibilities
The Data Engineer will design, develop, and maintain data integration pipelines that ingest, transform, and load information into the platform. This role ensures reliable, efficient, and scalable data flows from multiple source systems into the data platform.
Key Responsibilities:
- Design and implement ETL/ELT processes for data integration in Azure environments.
- Develop data pipelines using Azure Data Factory, Databricks, and related tools.
- Build data connectors to various source systems.
- Implement data validation and quality control checks.
- Develop the business-required transformation logic.
- Optimize pipeline performance and reliability.
- Monitor and troubleshoot data integration processes.
- Collaborate with data architects on integration patterns.
- Implement incremental loads and change data capture (CDC) processes.
- Document pipeline processes and associated data lineage.
Required Skills:
- More than 4 years of experience in data engineering (Python, SQL).
- Strong experience with ETL/ELT tools and frameworks.
- Proficiency in SQL and data manipulation.
- Knowledge of Python, Scala, or other data processing languages.
- Experience with Azure data services (Data Factory, Databricks, etc.).
- Understanding of data integration patterns and best practices.
- Familiarity with API-based integrations and connector development.
- Experience with data quality validations and error handling.
- Knowledge of performance optimization for data pipelines.
- Ability to work with complex and diverse data sources.
This job post is closed.