AI Scenario Designer & Evaluator at Mercor | Torre

AI Scenario Designer & Evaluator

You will shape advanced AI reasoning by designing complex, real-world training scenarios.
Emma highlights
This highlight was written by Emma’s AI. Ask Emma to edit it.
Freelance
A project
Compensation
USD50 - 100/hour
Non-negotiable
location_on
Remote (anywhere)
skeleton-gauges
You have opted out of job matches in .
To undo this, go to the 'Skills and Interests' section of your preferences.
Review preferences
Posted 8 days ago

Responsibilities and deliverables


Responsibilities: - Design realistic, complex AI training scenarios that simulate real-world workflows and decision-making challenges. - Build detailed personas and digital environments (e.g., Gmail, Slack, Calendar, WhatsApp, Google Drive). - Create tasks that test AI reasoning, prioritization, and information filtering under constraints. - Run AI agents through scenarios and evaluate performance. - Provide structured feedback and hints to improve AI outputs. - Identify inconsistencies, edge cases, and gaps in AI reasoning. - Document scenarios, evaluations, and results clearly using tools like Airtable and Crucible. - Continuously refine scenarios to increase realism and difficulty. Requirements: Must Have: - Bachelor’s (Undergraduate) degree. - 2+ years of professional experience. - Strong written communication skills. - High attention to detail. - Ability to think critically and spot inconsistencies. - Ability to work independently and manage workload. - Comfort working with structured tools and digital platforms. Nice to Have: - Strong writing background (copywriting, UX writing, journalism, content creation, editing). - Experience in multi-stakeholder environments (PM, consulting, client-facing roles, coordination). - Experience in scenario design or world-building (game design, simulations, UX research, curriculum design). - Familiarity with technical/data tools (JSON, Airtable, Jira, Asana, SQL, Python).
Optionally, you can add more information later (benefits, pre-screening questions, etc.)
check_circle

Payment confirmed

A member of the Torre team will contact you shortly

In the meantime, continue adding information to your job opening.