Role Summary: - As our Machine Learning Engineer - Evals, you will take full ownership of evaluating Honcho end-to-end. - Build the core benchmarking harness that informs our research-driven team whether our entity representations are truly improving. - Plastic Labs develops the core memory and identity layer powering agentic systems, and our product is already live in production. - Because this involves complex multi-agent architectures with emergent black-box dynamics, your primary challenge is establishing rigorous standards for what constitutes quality in evolving representations without fixed ground truth, then architecting the underlying measurement infrastructure. - If you thrive on constructing evaluation metrics from scratch, analyzing deep execution traces to uncover root-cause issues, and directly engineering production-grade solutions, this team is an ideal fit. Responsibilities: - Architecting custom eval suites: establish objective scoring mechanisms and benchmark definitions for evolving entity representations, continually adapting them as system capabilities mature. - Building robust evaluation infrastructure: construct scalable pipelines, automated labeling pipelines, LLM-as-a-judge frameworks, and versioned regression runs to ensure rapid response cycles. - Extracting actionable insights: inspect execution traces and benchmark outputs to isolate systemic failures, delivering high-conviction system design updates that directly enhance output fidelity. - Scaling agent simulation environments: expand continuous multi-agent testing capabilities to unlock higher throughput for concurrent entity evaluation. - Driving complete execution loops: lead problem definition, data pipeline construction, automated benchmarking, and technical writeups with total autonomy and minimal administrative overhead. Requirements: - Must have: Track record of architecting, implementing, and running LLM evaluation frameworks end-to-end in real-world software environments. - Must have: Production-level Python expertise with emphasis on automated test suites, maintainability, and release engineering. - Required: 3+ years of experience designing and owning LLM evaluation platforms. - Required: Deep experience with PyTorch, Hugging Face ecosystem, and Weights & Biases; familiarity with data pipeline tools like Hydra, Kafka, or SQL is a strong plus. - Nice to have: Hands-on exposure to agentic evaluation, memory systems, user identity models, or personalization infrastructure. - Nice to have: Primary or co-authorship on peer-reviewed research or equivalent high-caliber open-source contributions. - Nice to have: Degree in Computer Science, Math, Stats, or relevant quantitative field from a recognized university.

Machine Learning Engineer

Emma

Responsibilities

Skills wanted:

Language(s) required:

Catherene Njeri Ng'ang'a Verified

About QVID Talent Solutions:

mission:

qvidtalentsolutions.com/

Admin access needed