LLM Engineering Expert

Role overview

We are seeking experienced AI Evaluation Engineers specializing in engineering simulation and design to author and validate challenging, simulation-based engineering design problems for training and evaluating AI agents. The work spans electrical, mechanical, control systems, aerospace, systems, and robotics engineering. Tasks involve interpreting requirements, managing trade-offs, configuring open-source simulation tools, diagnosing failures, and iterating toward valid solutions. You will analyze agent execution logs, identify reasoning gaps, and build objective automated graders.

Responsibilities

• Create self-contained engineering design tasks with competing constraints, optimization targets, validated reference solutions, and objective autograders.
• Build, run, and validate problem environments using open-source simulation tools and custom Python test benches.
• Analyze agent outputs and execution logs across repeated trials to identify systemic failure modes.
• Refine task difficulty based on model performance while avoiding ambiguity or missing information.
• Collaborate with AI researchers, pod leads, and domain experts to integrate benchmarks into model evaluation pipelines.

Qualifications

• A master's degree or PhD in Electrical, Mechanical, or Aerospace Engineering, with 10+ years of hands-on engineering design experience. Another section of the listing mentions a bachelor's degree or equivalent practical experience in any field; please clarify which qualification applies.
• Proficiency in at least one relevant open-source simulation package, such as ngspice, PySpice, OpenFOAM, FEniCSx, CalculiX, python-control, CadQuery, build123d, OpenModelica, Cantera, or Gmsh, and strong Python scripting skills.
• Experience with modern LLMs or coding agents and evaluation concepts such as pass@k, failure-mode analysis, and nondeterministic behavior; ability to audit trajectory logs and identify reasoning or tool-use failures.
• Strong attention to physical plausibility, units, boundary conditions, convergence criteria, and technical documentation.
• AI evaluation, annotation, content review, or QA experience is preferred but not required.

Engagement and availability

• Contractor engagement for a short-term contract. The listing describes both a 12-week contract and an engagement of up to 24 weeks; please clarify the expected duration.
• 40 hours per week, with four hours of overlap with PST. Weekend on-call availability is required, although part-time engagement is also described as acceptable; please clarify the schedule.
• Remote setup with a personal computer and stable, high-speed internet.
• Immediate start; 60 positions are listed.

Compensation and selection

• The listing states a rate of $500 per task, capped AHT, an estimated 30–40 hours of AHT per task, and a partner flat fee of $80. Please clarify how these rates and estimates apply.
• Selection includes an interest-check form, Google Form, and delivery review.