Engineering Expert

Role Overview

We are seeking experienced AI Evaluation Engineers specializing in engineering simulation and design to author and validate complex, simulation-based engineering problems for training and evaluating AI agents. The role spans electrical, mechanical, control systems, aerospace, systems, and robotics engineering. You will create challenging, multi-constraint tasks that require agents to interpret requirements, navigate trade-offs, configure open-source simulation tools, diagnose failures, and iterate toward valid solutions. You will analyze agent execution logs, identify reasoning and tool-use gaps, and build objective automated graders.

Responsibilities

• Author self-contained engineering design tasks with competing constraints, explicit optimization targets, validated reference solutions, and objective autograders.
• Build, run, and validate problem environments using open-source simulation tools and custom Python test benches.
• Analyze agent outputs and execution logs across repeated trials to identify systemic failure modes, such as misinterpreting simulator feedback, premature convergence, or physically impossible geometries.
• Refine benchmark difficulty based on model performance data while avoiding ambiguity or missing information.
• Collaborate with AI researchers, pod leads, and domain experts to integrate rigorous benchmarks into model-evaluation pipelines.

Requirements

• A master's degree or PhD in electrical, mechanical, or aerospace engineering, and 10+ years of hands-on engineering design experience. A bachelor's degree or equivalent practical experience in any field is also listed as an education pathway; please clarify how this pathway relates to the advanced-degree and experience requirements.
• Proficiency in at least one relevant open-source simulation package, such as ngspice, PySpice, OpenFOAM, FEniCSx, CalculiX, python-control, CadQuery, build123d, OpenModelica, Cantera, or Gmsh, along with strong Python scripting skills.
• Hands-on experience with modern LLMs or coding agents and evaluation concepts such as pass@k, failure-mode analysis, and nondeterministic behavior; ability to audit trajectory logs and isolate reasoning or tool-use failures.
• Strong attention to physical plausibility, unit consistency, boundary conditions, convergence criteria, and technical documentation.
• A stable, high-speed internet connection and a personal computer suitable for remote work.
• Availability for 40 hours per week, four hours of overlap with PST, and weekend on-call availability. Part-time engagement is also noted as acceptable; please clarify the expected schedule.

Engagement Details

• Contractor engagement for up to 24 weeks; described as a short-term contract.
• Rate listed as $500 per task, with an estimated 30–40 hours of work per task. A partner flat fee of $70 is also listed; please clarify what each rate covers.
• Start date: immediately.
• The role description lists 210 positions.
• Candidates may be asked to complete an interest check form and a Google Form. Shortlisted candidates receive a Job Interest Form, followed by a delivery review.
• Eligible geographies listed: Bangladesh, India, Indonesia, Egypt, Ghana, Nigeria, Turkey, Brazil, Colombia, and approved states in the United States, subject to country-specific guidelines.