shivani Shanigarapu Sr. Data Engineer at New Reach

Sr. Data Engineer

New Reach

Oct 2025 - Current (10 months)

Senior Data Engineer

Availity Clinical Solutions

Jan 2025 - Jan 2026 (1 year 1 month)

Engineered scalable data pipelines using Scala, Java, and Python to process complex healthcare datasets for attestation, claims, and provider operations. Designed and orchestrated DAGs using Apache Airflow to automate and monitor ETL workflows, ensuring consistent and timely data delivery across various internal stakeholders. Developed File-Based Attestation workflows to trigger automated email notifications and update client-facing systems, improving transparency and regulatory compliance. Leveraged AWS Glue for serverless ETL orchestration and AWS Lambda for event-driven transformations and lightweight service integration tasks. Utilized Amazon Redshift for large-scale analytics and reporting; optimized query performance and schema design

Senior Data Engineer

Apple

Mar 2024 - Jan 2025 (11 months)

• Design, develop, and maintain scalable and reliable data pipelines using PySpark and Scala. • Implement ETL processes to extract, transform, and load large volumes of data from various sources, including AWS services like S3, Redshift, or DynamoDB. • Integrate AWS data sources into Spark applications, ensuring efficient data ingestion and processing. • Manage data storage solutions on AWS, such as setting up and optimizing S3 buckets for data storage and access. • Design and optimize data architecture to support high-performance and scalable data processing using PySpark. • Ensure data quality and reliability by implementing data validation and monitoring strategies. • Write unit tests and integration tests using ScalaTest or other testin

Data Engineer

First American Bank

Apr 2022 - Feb 2024 (1 year 11 months)

• Utilized Spark streaming to capture real-time data from Kafka, employing Scala and NoSQL databases like HBase and Cassandra to store the streaming data into HDFS. • Configured Zookeeper to coordinate server clusters, ensuring data consistency crucial for decision-making processes. • Implemented distributed frameworks like Apache Spark and Presto in Amazon EMR, Redshift, interacting with data in AWS data stores like Amazon S3 and Amazon DynamoDB. • Automated data loading into Nifi and pre-processing with Pig using Oozie workflows. • Developed Sqoop jobs for incremental loads from RDBMS into HDFS, applying Spark transformations. • Designed, developed, and performed Data Modeling of Spark SQL Scripts based on Functional Specifications. • Cre

Senior Data Engineer

RealPage Inc

Dec 2020 - Mar 2022 (1 year 4 months)

Senior Data & DevOps Engineer

SAFECO Insurance

Nov 2017 - Nov 2020 (3 years 1 month)

shivani Shanigarapu

About

Detail

Timeline

Résumé