Software Engineer II
Studyportals
Oct 2024 - Mar 2025 (6 months)
• Architected and deployed scalable ETL pipelines using Python to ingest, process, and analyze over 500,000 web pages daily • Designed a scalable, distributed, and cost-efficient architecture leveraging AWS Spot Fleet, SQS, EC2 and S3 for parallelized data retrieval and storage • Developed a multi-threaded NLP pipeline featuring a binary classifier that leverages log probability scoring and keyword filtering, achieving 96% accuracy in reliable information processing • Built a custom RAG system, chunking webpages and using keyword filtering & embeddings to extract obscured or hidden information, enhancing LLM-powered analysis • Fine-tuned and tested the latest HuggingFace transformers using QLoRA for data extraction and augmentation