Responsibilities:
- Participate in all aspects of agile software development, including design, implementation, and deployment.
- Design, build and maintain data transformations efficiently and reliably for different purposes (e.g. indexing, reporting, growth analysis, multidimensional analysis).
- Design, implement and maintain reliable, scalable, robust and extensible big data systems that support core products and business.
- Design, develop and support data pipelines and workflows that integrate various sources of structured and unstructured data across our customers' landscape.
- Assess and implement any customizations and custom data wrangling needed to transform customer datasets in a way that is best suited for processing by our ML models.
- Monitor data processing and machine learning workflows to ensure customer data is successfully processed by our ML models, debugging and resolving any issues faced along the way.
- Generate necessary data artifacts and analytics needed to assess the quality of results generated by different parts of the AI workflow to ensure they meet desired quality standards.
- Collaborate closely with the engineering team to assess model performance and improve results as needed.
- Collaborate closely with Product Engineering as needed to resolve any issues faced during customer deployment and onboarding.
- Collaborate closely with Customer Success Managers and Account managers to maintain close customer relationships and ensure customer satisfaction.
- Develop tooling and playbooks to streamline and automate key technical delivery activities to be repeatable and scalable.
- Develop and maintain RAG & Vector embeddings for AI and semantic search.
- Establish solid design and best engineering practice for engineers as well as non-technical people.
- Develop and maintain scalable and high-performing software solutions using the specified tech stack.
- Architect and provide guidance on building data pipelines and lakes.
- Collaborate with cross-functional teams to design and implement new features.
- Ensure code quality through functional testing.
- Optimize applications for maximum speed and scalability.
- Troubleshoot and debug issues to ensure smooth functionality.
- Collaborate across time zones via Slack, GitHub comments, documents, and frequent videoconferences.
- Stay up to date with industry trends and best practices.
Required skills and qualifications:
- At least four years of experience in building large-scale software applications.
Experience in the automotive industry and automotive parts standards such as ACES and PIES.
- Experience in performing data analysis, data ingestion and ETL (Extraction, Transformation & Loading).
- Experience with Big Data technologies (Airflow, Hadoop, M/R, Hive, Spark, Metastore, Presto, Flume, Kafka, ClickHouse, Flink, etc.).
- Experience with Elasticsearch & OpenSearch.
- Experience with Vector Databases (Qdrant, Opensearch, Milvus).
- Experience with AWS Bedrock & Q.
- Excellent problem-solving and communication skills.
- Experience working in a small team startup environment.
- Experience working on or implementing e-commerce platforms or solutions.
- Experience with AWS and cloud computing services.
- Ability to write and maintain clear and concise technical documentation.
- Experience in search technologies such as full-text lexical search and/or semantic vector-based search.
- Strong understanding of Git and experience with CI/CD pipelines.
- Experience building SaaS dashboards with graphs to display analytics and performance KPIs.
- Experience working with RESTful and GraphQL APIs.
- Knowledge of Python, R, Rust.
- Excellent ability to write efficient SQL queries.
- Excellent debugging and optimization skills.
- Preferred skills and qualifications.
- Bachelor's degree in Computer Science, a related technical field involving software or systems engineering, or equivalent practical experience.
- Interest in learning new tools, languages, workflows, and philosophies.
- Professional certification.
If you are a talented Data Engineer passionate about creating innovative solutions and thriving in a fast-paced startup environment, we would love to hear from you.
Please submit your resume, cover letter, and portfolio or GitHub showcasing your previous work.
Note: This job description is for reference purposes only and is not an exhaustive list of responsibilities and qualifications.
