Senior AI / LLM Evaluation Engineer (Full-time, remote from LatAm) Ruzora places senior LatAm engineers full-time with US startups. We're hiring a Senior AI / LLM Evaluation Engineer to join a partner company shipping LLM products. Your job is to tell the team, with numbers, whether the model, prompt or agent got better or worse, and to build the tooling that makes that answer fast. What you'll do - Design evaluation sets and metrics for LLM features: RAG, agents, tool use, classification, generation - Build eval pipelines that run in CI and on every model or prompt change - Set up LLM-as-judge graders and check them against human labels - Run regression and A/B comparisons across models and providers - Find failure modes (hallucinations, refusals, prompt injection) and turn them into tests - Report results clearly to engineers and product owners What we're looking for - 4+ years of software or ML engineering in Python - 1+ years building or evaluating LLM applications in production - Experience with eval tooling (OpenAI Evals, promptfoo, LangSmith, Braintrust, Ragas or your own) - Solid statistics: sampling, confidence intervals, inter-rater agreement - Comfortable with the OpenAI, Anthropic or similar model APIs - Excellent written and spoken English (B2 or higher) Nice to have - Building labeled datasets and annotation workflows - Red-teaming or AI safety work - Fine-tuning or reward modeling - Observability for LLM apps (tracing, cost and latency dashboards) Stack: Python, LLM APIs, eval frameworks, SQL, CI/CD What you get - Full-time, 100% remote from anywhere in Latin America - Competitive salary in USD - Flexible hours with overlap with US time zones - Professional development budget, health insurance stipend, equipment allowance Full role details: https://www.ruzora.com/jobs/ai-llm-evaluation-engineer

Senior AI / LLM Evaluation Engineer

Emma

Responsibilities

Skills wanted:

Language(s) required:

Roberto Espinoza

About Ruzora:

mission:

ruzora.com

Admin access needed