For companies:
For professionals:
For everyone:
For developers:
Responsibilities and deliverables
About the Role
We are building a team of experienced software engineers to evaluate AI-generated code on real-world, production-grade tasks. This is a technical evaluation role - you will act as a senior engineer reviewer, assessing how well AI models handle complex software engineering problems drawn from real open-source repositories.
Your evaluations directly influence how these models are trained and improved.
Responsibilities
1. Set up local development environments from real open-source repositories
2. Read and understand a non-trivial software engineering task (based on a real codebase change)
3. Prompt an AI model with the task and observe its response across multiple turns
4. Review AI-generated code in detail — logic correctness, edge case handling, test coverage, code structure, and adherence to the task scope
5. Run existing tests and validate that changes work as expected
6. Write a structured, evidence-backed technical evaluation comparing two model outputs (Model A vs. Model B), with justification for every scoring decision referencing specific files, functions, or test output
Requirements
1. 2+ years of professional software engineering experience
2. Strong proficiency in one or more: Python, JavaScript/TypeScript, Go, Rust, Java, C++
3. Fluent with Git, GitHub, and CLI-based development workflows
4. Ability to read and reason about production-level code in unfamiliar codebases quickly
5. Strong written English — clear, precise, evidence-driven technical writing is core to the role
6. High attention to detail — ability to spot subtle bugs, missing edge cases, and incorrect self-reporting by models
Compensation & Terms
1. Per-task payment with performance bonuses based on your acceptance rate
2. Estimated time: ~2–3 hours per task
3. Fully remote and async — work on your own schedule
4. No upper limit on task volume
5. Referral bonus: earn a bonus for every qualified engineer you refer once they reach 5 accepted tasks
This job post is closed.