AI Evaluation Engineer

About the Company

Responsibilities

• ● duct, and your work will directly determine what tens of thousands of field-service businesses experience when an agent answers on their behalf.
• ● Own release sign-off - gate every agent version on a green regression suite, judge-scored evaluations within agreed error bars, and a red-team pass - no promotion without them.
• ● Run error analysis - review sampled production traces across chat, email, WhatsApp, and voice every week, maintain the failure taxonomy, and turn new failure modes into dataset examples within the sprint.
• ● Build datasets and rubrics - design and maintain golden datasets and grading rubrics for multi-turn, tool-using agents, promoting interesting production runs into datasets in LangSmith.
• ● Build and calibrate LLM judges - align judges to human labels, re-label a held-out set each cycle, publish agreement per rubric, and retire or retrain judges that drift.
• ● Run offline and online evaluation - regression suites in CI alongside online scoring of sampled production traffic, with clear pass/fail thresholds and drift tracking.
• ● Triage regressions to root cause - attribute failures to prompt, tool, retrieval, model upgrade, or speech provider using paired statistics rather than aggregate deltas, and hand engineers concrete fixes.
• ● Own voice quality metrics - define and gate call-outcome metrics - task success, containment, barge-in recovery, word error rate under noise - alongside latency budgets.
• ● Evaluate classical ML models - set acceptance criteria, slice-level thresholds, calibration checks, and drift alerts for the vision, speech, and predictive models the platform depends on.
• ● Run adversarial testing - probe for prompt injection through inbound email and messaging content, jailbreaks, PII leakage, and tool misuse, and add regression coverage for every finding.
• ● Run the human evaluation programme - own annotation queues, reviewer guidelines, and inter-annotator agreement as an ongoing operation rather than a one-off study, and keep human and automated scores connected.
• ● Collaborate & ship - work in a cross-functional team using tools such as Jira and Slack, write clear documentation, and ship iteratively with a strong quality bar.

Requirements

• ● + years in roles where evaluating models was the core of the job - ML evaluation, ML quality, applied ML, or data science with an evaluation focus.
• ● Strong Python engineering skills, with experience building test harnesses and clean, well-tested code.
• ● Practical experience evaluating LLM-powered applications or AI agents: building datasets, defining heuristic and LLM-as-judge rubrics, running evaluations, and interpreting results to improve a system. This is a core requirement.
• ● Solid classical ML evaluation foundations: classification, regression, and ranking metrics, cross-validation, calibration, slice-based evaluation, and drift monitoring.
• ● Statistics for small evaluation sets: paired comparisons, confidence intervals, power analysis, and minimum detectable effect - you know roughly how many examples a claim needs before you make it.
• ● Hands-on experience with an LLM observability and evaluation platform (LangSmith, MLflow, or equivalent): datasets, experiments, custom evaluators, feedback, and wiring evaluations into release gates.
• ● Strong data analysis skills using Pandas, NumPy, and SQL to quantify behaviour and communicate findings.
• ● Working knowledge of how agents are built - prompts, tools, retrieval, and memory - and how they interact, sufficient to root-cause a failure rather than only report it.
• ● Awareness of AI safety and adversarial risk: prompt injection, jailbreaks, data leakage, tool misuse, and responsible-AI practice.
• ● Awareness of the compliance side of evaluation: handling conversation data under UK GDPR, keeping evaluation evidence auditable, and emerging record-keeping expectations for AI systems.
• ● Committed to continuous learning, proactive problem-solving, and timely issue identification, with a keen interest in staying current with a fast-moving field.
• ● Strong communicator, experienced in collaborating with cross-functional teams using tools such as Jira and Slack.
• ● Creative and innovative thinker, consistently contributing fresh ideas and solutions in alignment with current technological trends.
• ● Nice to Have
• ● Publications, open-source contributions, or public evaluation work.

Benefits

• ● Professional Working environment
• ● Market Competitive Salary
• ● Life Insurance & Medical Insurance (Including Family)
• ● Provident Fund
• ● Gym Facility
• ● Maximum 45 Weekly Hours (Monday-Friday)
• ● Remote Working (During Pandemic Situation)
• ● Company trip
• ● Annual Leaves
• ● Sick & uncapped Compassionate Leaves (As per Company Policy)
• ● Have a chance to work onsite with the UK team