About this role
We are looking for a Member of Technical Staff, AI Evaluation with 5+ years of experience, blending physical engineering (mechanical, electrical, aerospace, robotics) and AI evals, to own how we measure Archie, our AI engineer for the physical world. You'll build the evaluation framework that shows how Archie compares to human engineers and what it takes to grow him from junior to senior engineer, and eventually to engineering superintelligence.
What will you be doing?
Own Archie's evaluation framework from scratch, defining what "good" engineering design looks like when there's no single right answer
Work with engineering SMEs, customers, partners and internal teams to learn how top engineers work and where Archie succeeds or fails
Recreate real-world engineering tasks and failure modes in evaluation environments that the dev teams can rerun
Design and refine automated judges and human evaluation processes, and check that judges match expert human judgment
Turn eval results into clear priorities for improving Archie's engineering reasoning
Key Requirements
5+ years of experience across physical engineering and/or AI work. The ideal candidate is strong in both, but deep strength in one with working knowledge of the other is fine
Hands-on AI evaluation experience: building eval tasks, designing judges, and running human evaluation for agentic systems
Physical engineering background (mechanical, electrical, aerospace, robotics, power systems), enough to recognize strong engineering design
The presence and confidence to lead work across SMEs, customers and cross-functional teams in a high-ownership role
Experience in scrappy, early-stage environments building new things from scratch (physical AI or robotics companies a plus)
Role details
Salary
$200K - $300K
Remote work policy
US or Canada-based (remote), with on-site work one week every 6 weeks in San Mateo, CA. Relocation decided case by case.
Full-time position
Location
United States, Canada, San Mateo, CA
Equity
Competitive equity
Visa sponsorship details
Not open to any visas (e.g. US citizen, Green Card holders)
Hiring Count
Looking to hire 1 candidate for this role
Tech stack
Agentic AI Systems, AI Evaluation Frameworks, Automated Judges (LLM-as-a-Judge), Human-in-the-loop Evaluation, Evaluation Environments, SFT, RLVR, Engineering Design Tools
Requirements
By tier
By category
Hear the hiring manager detail the role requirements
00:49
Calibrate a candidate in Parabot. to see how well they fit the requirements.
Dealbreaker
Work experience
Experience with AI evaluations (creating evaluation tasks, designing judges, comparing automated evaluations to human judgment)
Hard skills
Understanding of agentic AI systems
Updated
Miscellaneous
Located in the United States or Canada (remote) or San Mateo, CA
Updated
Baseline
Seniority
5 - 15 years of experience as a Member of Technical Staff, AI engineer, or AI research scientist
Education
Technical background in a physical engineering domain such as mechanical engineering, aerospace, robotics, or power systems
Soft skills
Experience working directly with customers, subject matter experts, partners, and cross-functional technical teams
Updated
Miscellaneous
Ability to travel to the San Mateo, CA office one week every 6 weeks
Updated
Nice-to-have
Work experience
Experience at physical AI or robotics companies (e.g. Applied Intuition)
Updated
Experience doing zero-to-one work in early-stage startup environments
Updated
Traits to avoid
Pattern of short tenures (multiple consecutive stints of 12 to 18 months)
Updated
No obvious physical engineering or AI evals experience
Member of Technical Staff, AI Evaluation
Full-time
Remote
United States
$200K - $300K
How actively is this candidate exploring new opportunities?
