Rajashekar reddy

• Senior Data Engineer with 5+ years of experience in IT, including expertise in Azure cloud services and Big Data technologies, as well as 3 years of experience in Data warehouse implementations. • Experienced in implementing Databricks Delta Lake Architecture (bronze, silver, and gold layers) and utilizing Delta Live Tables (DLT) to create pipelines for DLT. Proficient in using Auto Loader to process new data files incrementally and efficiently. • Skilled in building data pipelines using Azure Data Factory and Azure Databricks, and proficient in loading data into Azure Data Lake, Azure SQL Database, and Azure SQL Data Warehouse (Synapse Analytics). • Proficient in ensuring data quality and integrity through validation, cleansing, and transformation operations. • Strong expertise in optimizing Spark jobs and leveraging Azure Synapse Analytics for big data processing and analytics. • Implemented data visualization using Power BI and Power BI DAX. • Developed ETL transformations and validation using Spark SQL/Spark Data Frames with Azure Databricks and Azure Data Factory. • Experience in Apache Kafka, Apache NiFi, and Spark integration for real-time data processing. • Adept at designing cloud-based data warehouse solutions using Snowflake on Azure, optimizing schemas, tables, and views for efficient data storage and retrieval. • Deployed Data Factory for creating data pipelines to orchestrate the data into SQL databases. • Working on Snowflake modeling using data warehousing techniques, data cleansing, Slowly Changing Dimension phenomenon, surrogate key assignment, and change data capture. • Developed ELT/ETL pipelines to move data to and from Snowflake data store using a combination of Python and Snowflake SnowSQL. • Participated in the development, improvement, and maintenance of Snowflake database applications. • Built the Logical and Physical data model for Snowflake as per the required changes. • Defined roles and privileges required to access different database objects. • In-depth knowledge of Snowflake Database, Schema, and Table structures. • Implemented data pipelines using SnowSQL, Snowflake Integrated services, and SnowPipe. • Hands-on experience in implementing data pipeline solutions using Hadoop, Azure, ADF, Synapse, PySpark, MapReduce, Hive, Tez, Python, Scala, Azure functions, Azure Logic Apps, StreamSets, Azure Data Lake Storage Gen2, and Snowflake. • Strong expertise in optimizing Spark jobs and leveraging Azure Synapse Analytics for big data processing and analytics. • Proven track record in performance optimization and capacity planning to ensure scalability and efficiency. • Experienced in developing CI/CD frameworks for data pipelines and collaborating with DevOps teams for automated pipeline deployment. • Proficient in scripting languages such as Python and Scala. • Skilled in working with Hive, SparkSQL, Kafka, and Spark Streaming for ETL tasks and real-time data processing. • Have good working experience in Hadoop, HDFS, MapReduce, Hive, Tez, Python, and PySpark. • Hands-on experience in developing large-scale data pipelines using Spark and Hive. • Experience in using Apache Sqoop to import and export data from HDFS and Hive. • Hands-on experience in setting up workflows using the Apache Oozie workflow engine for managing and scheduling Hadoop jobs. • Experienced in importing and exporting data using Sqoop from HDFS to Relational Database systems. • Optimized query performance in Hive using bucketing and partitioning techniques and have extensive hands-on experience tuning Spark jobs. • Hands-on experience using version control tools such as GitHub, Azure DevOps, Bitbucket, GitLab, and ARM templates. • Implemented advanced data manipulation and analysis techniques using window functions, resulting in optimized and efficient data processing and analytics pipelines.