Role Details:
- Role: ML Architect.
- Start Date: ASAP.
- Duration: 3+ months.
- Allocation: 40 hrs per week.
- Location: Remote.
Key Responsibilities:
- Define the target architecture and roadmap for machine learning on AWS and Databricks.
- Design and implement scalable, secure, and production-ready ML solutions.
- Assess the current Databricks implementation and recommend improvements, integrations, and platform capabilities.
- Establish end-to-end ML workflows covering data preparation, feature engineering, experimentation, model training, validation, deployment, monitoring, and retraining.
- Implement MLOps practices using Databricks and MLflow, including experiment tracking, model management, automated deployment, governance, and monitoring.
- Design batch and real-time model-inference solutions based on business and technical requirements.
- Advise teams on the effective use of Databricks capabilities, including Unity Catalog, Feature Engineering, Model Serving, Lakehouse Monitoring, Delta Lake, Delta Live Tables/Lakeflow, and Databricks Workflows.
- Integrate Databricks with relevant AWS services, such as Amazon S3, IAM, KMS, SageMaker, Lambda, ECR, CloudWatch, and streaming services.
- Define reusable architecture patterns, development standards, and platform guardrails.
- Establish appropriate security, access-control, data-governance, model-governance, and compliance practices.
- Optimize workloads for scalability, reliability, performance, and cost.
- Partner with data scientists, data engineers, platform teams, security teams, and business stakeholders to move ML use cases from concept into production.
- Provide technical leadership, mentoring, documentation, and knowledge transfer to internal teams.
Required Qualifications:
- Extensive experience designing and delivering enterprise machine learning platforms and solutions.
- Strong hands-on experience with Databricks in a production environment.
- Deep knowledge of the Databricks Lakehouse architecture and its machine learning capabilities.
- Strong experience with MLflow, including experiment tracking, model lifecycle management, deployment, and governance.
- Extensive AWS architecture and implementation experience.
- Strong understanding of MLOps, CI/CD, infrastructure as code, automated testing, observability, and model monitoring.
- Experience deploying both batch and real-time inference workloads.
- Proficiency in Python and SQL, with experience using Apache Spark/PySpark.
- Experience with feature engineering, model evaluation, drift detection, explainability, retraining, and responsible AI practices.
- Strong knowledge of data security, IAM, encryption, networking, governance, and regulatory considerations in cloud environments.
- Ability to translate business requirements into practical architecture and implementation plans.
- Excellent communication skills and the ability to influence technical and non-technical stakeholders.
