Chaitanya O

Hello and welcome to my profile.I am a Senior Data Engineer with 9+ years of experience designing and implementing scalable data engineering, analytics, and cloud data platform solutions across AWS, Azure, and GCP. I specialize in building end-to-end data pipelines, modern data architectures, and governed data ecosystems that enable reliable, secure, and high-performance data processing for enterprise analytics and AI-driven initiatives. I have strong experience in enterprise data governance using Informatica Intelligent Data Management Cloud (IDMC), including Data Catalog, Data Quality, Data Lineage, and Data Marketplace. I have worked extensively on metadata-driven frameworks, data classification, business glossary alignment, and building trusted, reusable data products for enterprise consumption and self-service analytics. 🔶 AWS ExpertiseExperienced in building enterprise data solutions using AWS services including S3, Glue, Lambda, Step Functions, EMR, Kinesis, and Redshift. Skilled in PySpark and Spark SQL for large-scale data processing, and implementing data orchestration, monitoring, and security using IAM, KMS, and CloudWatch. Experienced in integrating data governance principles into AWS pipelines, including metadata tracking, lineage capture, and data quality enforcement.🔷 Azure ExpertiseHands-on experience in designing and implementing data pipelines using Azure Data Factory, Data Lake Storage, Azure Synapse Analytics, and Azure SQL. Skilled in Databricks and Spark-based processing for batch and streaming workloads, monitoring, and hybrid data integration solutions.☁️ GCP ExpertiseBuilt scalable data pipelines using BigQuery, Dataflow, Pub/Sub, and Cloud Composer for batch and real-time processing. Experienced in implementing data governance practices including data cataloging, classification, lineage tracking, and access control using IAM. Exposure to Vertex AI for machine learning integration and AI-ready data preparation.🛠️ Core SkillsPython, SQL, Spark (Batch & Streaming), PySpark, Kafka, ETL/ELT Development, Data Warehousing, Data Quality, Data Profiling, Schema Management, Data Governance, Metadata Management, Data Lineage, Data Products, CI/CD (Jenkins, GitLab), Terraform, Docker, Kubernetes, Cloud Security, and AI-ready data engineering.I am passionate about solving complex data challenges and transforming raw data into trusted, actionable insights for analytics and AI.Let’s connect to explore opportunities in Data Engineering, Cloud Data Platforms, Data Governance, and AI-driven data solutions.