V
Vidya P
About
Detail
United States
Contact Vidya regarding:
work
Full-time jobs
Résumé
Jobs
verified_user
0% verified
Data EngineerAssurantMay 2023 - Current (3 years 6 months)• Orchestrated Azure Data Factory Pipelines for data ingestion, effectively migrated all tables from an on-premises SQL Server database to Azure Data Lake Storage Gen2. • Orchestrated workflows in Databricks and migrated data to delta tables and utilized delta live tables for real time data processing and loading. • Integrated Power BI with Azure SQL database to extract views, creating an engaging and interactive dashboard that effectively displayed and analyzed the data. • Integrated Azure Logic Apps and Azure Functions with Azure Monitor for real-time data pipeline monitoring and alerting, resulting in a 30% improvement in issue resolution time. • Scheduled workflows in Apache Airflow, ensuring seamless transformation and resolving 1- Data EngineerInfosysJan 2020 - Jan 2022 (2 years 1 month)• Created SQL Server stored procedures, functions, views, and triggers using both sql and T-sql improved . • Created ETL pipelines for data integration in both cloud and on-premises environments by leveraging the capabilities of Azure Data Factory and SSIS packages. • Built Spark applications using Pyspark and Spark-SQL programming with 30% improvement in data extraction, and aggregation from multiple file formats(JSON, Parquet, Avro, CSV, XML). • Enhanced Azure Data Factory pipeline deployment by implementing CI/CD processes via Azure DevOps, Kubernetes and Docker 50% betterment in production release efficiency. • Utilized Agile methodology and Azure Boards for sprint planning, ensuring complete alignment with sprint deadlines for a t
- Junior Data EngineerCognizantJan 2019 - Dec 2019 (1 year)• Created spark streaming applications for real time data processing by integrating with the AWS Kinesis, Kinesis Data Analytics and load the data to redshift. • Applied snowflake features like Snow Pipes and stages to process 10GB data everyday in real-time and ingest the data into snowflake tables. • Scheduled a Serverless Workflow with AWS Step Functions and Amazon EventBridge Scheduler. • Orchestrated ETL Pipelines in Glue Studio, applying diverse transformations, and ingesting data into S3, Redshift, and RDS reduced 40% completion time. • Proficiency in configuring, working with big data technologies to improve performance using Hadoop, Sqoop, Spark,Hive,MapReduce and HBase on AWS EMR clusters. • Architected RESTful APIs with Dja
Education
verified_user
0% verified
- MMICROSOFT CERTIFIED AZURE DATA ENGINEERASSOCIATE (DP203)
- CERTIFIED ON DATABASE MANAGEMENT SYSTEMSNptel
- Bachelors of Computer ScienceOsmania University
- Masters of Computer ScienceUniversity of Massachusetts Boston