Havila

• Overall, 5+ years of technical experience as a Big Data Engineer in business need of clients, developing effective and efficient solutions. • Experience developing a cloud-based Linux OS in AWS to develop scalable Python applications. • Extensive expertise and understanding of Hadoop ecosystem products such as HDFS, MapReduce, YARN, Spark, Kafka, Hive, Sqoop, Impala, HBase, Flume, Oozie, and Zookeeper. • Strong understanding/knowledge of Hadoop architecture and concepts such as HDFS, MapReduce, Job Tracker, Task Tracker, Name Node, and Data Node. • Working knowledge of Cloudera, Azure HDInsight, and the AWS cloud. • Converted SQL queries into Spark Transformations utilising Spark RDDs, Scala, and performed map-side joins on RDDs. • Worked on AWS Redshift and RDS for model and data implementation on RDS and Redshift. • Involved in the design and development of CSG enhancements leveraging AWS APIS. • End-to-end writing expertise Processing of Data analysis jobs utilising MapReduce, Spark, and Hive. • Extensive experience implementing production-ready Spark applications with Spark components like as Spark SQL, MLlib, Spark Streaming, and Graph X. • Experience with Hive partitioning, bucketing, and joins on Hive tables, as well as implementing Hive SerDes. • Strong familiarity with Amazon cloud web services such as EMR, Redshift, DynamoDB, Lambda, Athena, S3, RDS, and CloudWatch for effective large data processing. • Create ETL processes in AWS Glue to move Campaign data from external sources like as S3, ORC/Parquet/Text Files into AWS Redshift. • Experience extracting files from MongoDB using Sqoop, storing them in HDFS, and then processing them. • Worked with several ingestion services to handle batch and real-time data using Spark streaming, Kafka Confluent, Storm, Flume, and Sqoop. • Configured AWS CLI and used shell scripting to accomplish essential AWS service activities. • Hands-on expertise with NoSQL databases such as HBase, Cassandra, and MongoDB, as well as their interaction with Hadoop and Kubernetes clusters. • To launch