Cloud Data Engineer
Indra
Sep 2022 - Apr 2023 (8 months)
• Developed processes in the DataLake system for data ingestion from the risk and CRM areas. In the cloud environment, processes and workflows were developed in Databricks, also taking advantage of the file upload to Azure Data Lake Storage, as well as adjustments in ETLs of Azure Data Factory and modified historical loads using AzCopy. On the other hand, in the on-premises environment, pipelines with Spark, Hadoop and DataStage were developed
• Developed quality processes for data quality analysis using SQL in (Oracle, SQL Server, MySQL, Hive), python, scala and Jira
• Used CD/CI methodologies for development deployment and testing, through tools such as DevOps (Job Jenkins pipeline, Bitbucket, Clear Case, Clear Quest, Remedy, Gi