Senior Software Engineer, AI Evaluation and Coding Agents

Job Summary

We’re seeking an experienced, hands-on software engineer to evaluate and improve AI coding models. You’ll work with coding agents on real-world repositories, assess the correctness and quality of their work, identify failure modes, and develop evaluation criteria and feedback to improve model performance. This is a remote, independent contractor engagement for approximately three months, requiring 20–40 hours per week and at least six hours of daily overlap with Pacific Time. Candidates must be based in North America.

Key Responsibilities

• Design, refine, and iterate on rubrics and evaluation criteria for preference data.
• Review and label data to a high quality bar, identifying subtle technical issues.
• Evaluate AI-generated code and agent behavior for correctness, robustness, and maintainability.
• Build and maintain pipelines and infrastructure for data generation, collection, and evaluation workflows.
• Document findings and provide clear recommendations to the team.
• Collaborate with researchers and engineers to turn technical judgment into scalable evaluation processes.

Qualifications

• Skills: Strong proficiency in Python, TypeScript/JavaScript, Go, or another major production language; substantial code review experience; sound technical judgment; clear written communication; and familiarity with modern LLMs or AI coding tools. Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training is a plus.
• Education: A degree in computer science, software engineering, or a related field is useful; equivalent professional experience is also relevant.
• Experience: At least five years of hands-on software engineering experience, including work in substantial real-world codebases.
• Attributes: Analytical, detail-oriented, objective, collaborative, and able to explain technical decisions clearly.

Compensation & Benefits

• Compensation: $100–$150 per hour. Candidates should provide a specific hourly rate expectation.
• Remote work for candidates located in North America.
• Flexible commitment of 20–40 hours per week, with at least six hours of Pacific Time overlap per day.
• Independent contractor engagement for approximately three months, with an ASAP start.

How to Apply

Apply using https://work.turing.com/r/Y1tHX-_de0