As a Senior Data Scientist at VIXPar, I work with large volumes of data from logistics operations, fleet management, and urban mobility, developing advanced AI solutions for process optimization.
Currently, I am building a RAG (Retrieval-Augmented Generation) Agent using the LangGraph framework, which integrates multiple corporate data sources through a complete processing pipeline. The project includes:
• Pre-processing of our data with Docling, extracting information from Word (.doc or .docx), Excel, and PDF files
• Intelligent web scraping with Crawl4ai for web data capture
• Conversion and standardization of all data to Markdown (.md) format
• Vector storage in Qdrant with hybrid search (dense and sparse) for optimized retrieval
• Integration with OpenAI models via Azure OpenAI Gateway
• Persistent memory system with contextual understanding capabilities, including greetings, farewells, and user intent
The goal is to create an intelligent assistant capable of answering complex questions based on corporate data that is in text, tables, etc., maintaining conversational context and offering accurate and relevant answers.
Thank God 🤓!