DILIP KUMAR Sr Databricks Engineer at ShyftLabs

S

Engineering Technician

ShyftLabsPixel

Dec 2025 - Current (11 months)

Built end-to-end ML pipelines: preprocessing, feature engineering, model training and evaluation. Applied PCA, normalization, standardization to handle high-dimensional data at scale. Trained and tuned models (Random Forest, Gradient Boosting, Logistic Regression) with Grid/Random Search. Deployed ML models as REST APIs via Flask for real-time downstream integration. Integrated AI-driven solutions into data pipelines for intelligent business decision-making.

Sr Databricks Engineer

ShyftLabs

Mar 2025 - Current (1 year 8 months)

Built end-to-end AI/ML solutions by performing data preprocessing, feature engineering, dimensionality reduction (PCA), model training, evaluation, and selection using Python-based machine learning frameworks.

Optimized and validated machine learning models through feature scaling, hyperparameter tuning (Grid Search, Random Search), cross-validation, and performance metrics such as Accuracy, Precision, Recall, and F1-Score to ensure robust and scalable solutions.

Deployed production-ready ML models as Flask-based REST APIs and integrated AI-driven capabilities into enterprise data pipelines and applications, enabling real-time predictions and business decision-making.

Designed and developed scalable Data

S

Cloud Engineer

ShyftLabs Innotop Solutions

Mar 2025 - Dec 2025 (10 months)

Designed Databricks ETL pipelines with multi-job dependencies orchestrated via Databricks Workflows. Built config-driven reusable PySpark frameworks replacing hardcoded scripts, cutting runtime significantly. Optimised Delta Lake pipelines with partition pruning, window optimisations and intelligent filtering. Developed hybrid ingestion frameworks to flatten nested JSON/arrays into relational models. Performed data reconciliation and root-cause analysis on financial/fuel transaction tables.

Senior Data Engineer

Sopra Steria

Jul 2022 - Mar 2025 (2 years 9 months)

Designed end-to-end ADF + Azure Databricks pipelines loading ADLS for enterprise data needs. Implemented Medallion Architecture (Bronze/Silver/Gold) in Databricks for scalable processing. Built BigQuery solutions with Dataform; designed star schema and Data Vault models in BQ. Automated Cloud Composer DAGs and Pub/Sub workloads for end-to-end GCP ETL. Managed Terraform framework across Dev, UAT, and Prod; provisioned service accounts. Configured Azure Synapse Analytics + Power BI for real-time reporting dashboards.

Technical Lead

Sopra Steria

Jul 2022 - Mar 2025 (2 years 9 months)

Designed and implemented end-to-end data engineering solutions using Azure Data Factory (ADF), Azure Databricks, PySpark, ADLS, and Azure Synapse Analytics, processing structured and unstructured data at enterprise scale.

Built scalable Medallion Architecture (Bronze, Silver, Gold) with Databricks and PySpark, developing data cleansing, standardization, deduplication, and transformation frameworks to support analytics, reporting, and machine learning workloads.

Developed cloud-native GCP data platforms using BigQuery, Dataform, Cloud Composer (Airflow), Pub/Sub, and Terraform, automating ETL workflows, infrastructure provisioning, and data validation frameworks across multiple environments.

Created reu

Senior Data Engineer

Societe Generale

Nov 2021 - Jul 2022 (9 months)

Built Cloudera Hadoop ecosystem for versatile multi-source data integration. Developed Spark framework to read from Hive and produce files in multiple formats. Implemented pipelines with Control-M and Shell Scripts; configured failure email alerts.

Sr software engineer

Société Générale Global Solution Centre

Nov 2021 - Jul 2022 (9 months)

Contributed to the Big Data Hub platform for the Commercial Banking domain by building scalable data integration solutions to support multiple enterprise data and analytics requirements.

Designed and developed Hadoop and Spark-based frameworks on Cloudera, enabling data ingestion from Hive tables, data transformation, and generation of multiple output formats for downstream consumption.

Automated and optimized data operations using Control-M, Shell Scripting, and Spark performance tuning, ensuring reliable job scheduling, monitoring, housekeeping, and operational efficiency across the platform.

Software Engineer

OpenText

Feb 2019 - Dec 2021 (2 years 11 months)

Developed Big Data solutions using Hadoop, Hive, Sqoop, and Spark, building data ingestion pipelines, Hive tables, and optimized queries leveraging partitioning, bucketing, and efficient file formats such as ORC and Parquet.

Implemented enterprise data warehousing solutions using Data Vault 2.0 methodology, including Raw Vault, Business Vault, and Information Vault layers to support scalable and auditable data platforms.

Automated and optimized data processing workflows through Jenkins CI/CD pipelines, Control-M scheduling, Oozie workflows, Shell scripting, and Spark Scala performance tuning to improve reliability and operational efficiency.

O

Data Engineer

OpenText Technologies

Feb 2019 - Oct 2021 (2 years 9 months)

Applied Data Vault 2.0 for enterprise DWH (Raw Vault, Business Vault, Information Vault). Worked with Spark SQL/Scala, Hive ORC/Parquet, Sqoop, HDFS at scale. Configured Jenkins pipelines; optimised Spark Scala and Hive query performance.

Service Delivery Specialist

IBM

Aug 2016 - Feb 2019 (2 years 7 months)

Built and maintained Hadoop-based data processing solutions using HDFS, Hive, and Sqoop for large-scale data ingestion, transformation, and movement across enterprise systems.

Developed and optimized Hive data models and queries, utilizing partitioning, bucketing, and columnar storage formats (ORC and Parquet) to enhance query performance and storage efficiency.

Worked extensively with the Spark ecosystem, leveraging Spark SQL and Spark Scala to process large datasets and support enterprise analytics and reporting requirements.

Technical Specialist

IBM

Aug 2016 - Feb 2019 (2 years 7 months)

Hive table creation, partitioning/bucketing; Sqoop imports/exports with HDFS. Spark SQL and Scala development; Control-M scheduling and Oozie notifications.

DILIP KUMAR

About

Detail

Timeline

Résumé

DILIP KUMAR

About

Detail

Behavioral traits