Abhishek Shapelly

Abhishek Shapelly

About

Detail

Data Engineer
Pennsylvania, United States
Contact Abhishek regarding: 
work
Full-time jobs
Starting at USD100k/year

Timeline

work
Job
school
Education

Résumé

Jobs verified_user 0% verified
  • A
    Senior Data Engineer
    Apollo Global Management Inc
    Apr 2023 - Aug 2023 (5 months)
    Responsibilities: • Experience designing and implementing Extract, Transform, Load (ETL) processes using ADF. • Proficient in building data pipelines for efficient data movement and transformation. • Scheduling all the batch jobs using Tidal monitoring tool. • Expertise in integrating data from diverse sources into a unified and coherent format. • Orchestrated complex data workflows using ADF to ensure seamless execution of tasks. • Implemented scheduling and monitoring of data processing jobs for optimal performance. • Proficient in monitoring ADF pipelines for performance and troubleshooting issues. • Implemented logging and alerting mechanisms for proactive issue resolution. • Collaborated with cross-functional teams to understand data r
  • Capital One
    Senior Data Engineer
    Capital One
    May 2022 - Apr 2023 (1 year)
    • Gather business requirements from clients and convert them to the technical specifications and generate partner deliverable reports. • Leveraging Big Data infrastructure for batch processing and responsible for building scalable data solutions using Spark. • Extract source feeds from AWS S3(One lake) location to read/modify/edit/update the PARQUET/CSV/Fixed Length data using Spark/SPARK-SQL and store it in HDFS or One Lake locations. • As a functional process, performed joins, aggregations, filters, and other transformations on the datasets using SPARK. Experience in handling appropriate features from Datasets in order to handle bad, null, zero, partial records in Spark-SQL. • Built ETL generic process, perform tasks like onboarding data
  • Wells Fargo
    Senior Hadoop Developer
    Wells Fargo
    Aug 2018 - Sep 2019 (1 year 2 months)
    • Designed and developed Hadoop-based Big Data analytic solutions and engaged clients in technical discussions. • Worked on multiple Azure platforms like Azure Data Factory, Azure Data Lake, Azure SQL Database, Azure SQL Data Warehouse, Azure Analysis Services, HDInsight. • Worked on the creation and implementation of custom Hadoop applications in the Azure environment. • Created ADF Pipelines to load data from an on-prem to Azure SQL Server database and Azure Data Lake storage. • Developed complicated Hive queries to extract data from various sources (Data Lake) and to store it in HDFS. • Used Azure Data Lake Analytics, HDInsight/Databricks to generate Ad Hoc analysis. • Developed custom ETL solutions, batch processing, and real-time d
  • Ness Digital Engineering
    Data Engineer
    Ness Digital Engineering
    Sep 2017 - Aug 2018 (1 year)
    Involved in architecture design, development, and implementation of Hadoop deployment, backup, and recovery systems. Developed MapReduce programs in Python using Hadoop to parse the raw data, populate staging tables, and store their fine data in partitioned HIVE tables. Enabled speedy reviews and first-mover advantages by using Oozie to automate data loading into the Hadoop Distributed File System and Pig to pre-process the data. Converted applications that were on MapReduce to PySpark which performed the business logic. Involved in creating Hive tables, loading with data, writing hive queries that will run internally in map reduce way. Implemented Spark using Scala and SparkSql for faster testing and processing of data. Imported Teradata d
  • Infosys
    Software Engineer
    Infosys
    Sep 2015 - Sep 2017 (2 years 1 month)
    -Analyzed the requirements provided by the client and developed a detailed design with the team. -Worked with the client team to confirm the design and modified based on the changes mentioned. -Involved in extracting and exporting data from DB2 into AWS for analysis, visualization, and report generation. -Created HBase tables and columns to store the user event data. -Used Hive and Impala to query the data in HBase. -Developed and implemented core API services using Scala and Spark. -Managed querying the data frames using Spark SQL. -Used Spark data frames to migrate data from AWS to MySQL. -Built continuous ETL pipeline by using Kafka, Spark streaming and HDFS. -Performed ETL on data from various file formats (JSON, Parquet and Database).
Education verified_user 0% verified
  • University of North Texas
    Master of Science - MS, Business Analytics
    University of North Texas
    Jan 2014 - Dec 2014 (1 year)
  • Osmania University
    Bachelor of Technology - BTech, Chemical Engineering
    Osmania University