Principal Research Scientist - Evaluations

About the Company

Responsibilities

• ● Company Description
• ● Hey, g'day, mabuhay, kia ora,你好, hallo, vítejte
• ● Thanks for stopping by. We know job hunting can be a little time consuming and you're probably keen to find out what's on offer, so we'll get straight to the point.
• ● Where and how you can work
• ● Our flagship campus is in Sydney. This role is preferred to be based out of our Sydney office, where the majority of this team is located.
• ● Job Description
• ● Canva's generative models are judged by millions of people who will never read a benchmark. They just know whether the design looks right. Turning that judgement into something measurable is the hardest problem in our research stack, and it gates everything else. If we cannot measure design quality reliably, we cannot train against it, we cannot tell a real improvement from noise, and we cannot decide what ships.
• ● This is a Canva-wide craft leadership role, setting direction across our research groups in Australia, Europe, the US and China. You will be the person others come to when the numbers and the eyes disagree.
• ● What you'll own
• ● The evaluation strategy for Canva Research. Define what great evaluation looks like across design, image, video, audio and agentic workflows, and set the long-term direction for how Canva measures generative quality as the space evolves. You will shape the principles teams use to trade off evaluation compute, human data spend and signal fidelity, and you will defend them.
• ● The science of measurement itself. Auto-raters and MLLM judges are only as good as their correlation with the thing they proxy for. You will treat that correlation as a research problem: validating metrics against human preference and downstream product outcomes, quantifying judge bias, and catching benchmark saturation and contamination before they quietly stop telling us anything.
• ● The link between evaluation and what actually matters. Evaluation should reflect user experience and product impact, not just perform well in isolation. You own closing that gap, including the fact that a good evaluator is not automatically a good reward model for RL. That extends to the full experience rather than the artefact alone, editability included.
• ● One standard across every region. Our teams in London, Vienna, San Francisco, Sydney and China all need to know they are measuring the same thing. You will build the shared evaluation layer that makes results comparable, and you will spot the collaboration opportunities nobody has been chartered to own yet. Taking that initiative is a core expectation of this role, not a bonus.
• ● Focus areas
• ● Human preference at scale. Rubric design, rater guidelines, inter-rater reliability, and calibration across markets and cultural contexts. Aesthetic judgement is not universal, and our evaluation systems need to hold that honestly rather than average it away.
• ● Learned quality models and reward signals. Reward modelling and preference learning that feed post-training (RLHF, RLAIF) and inference-time selection, and being explicit about where a good judge does not translate into a good reward model.
• ● Vision-Language Models for quality understanding. Novel architectures and training approaches for models that understand what makes a design effective, not just well-formed. These become the reward signal and feedback loop for our design generation models, so their failure modes are our failure modes.
• ● Agentic and automated evaluation. MLLM-as-a-judge systems, model-based grading, and the infrastructure to run hundreds of evaluations against live training checkpoints without the results becoming noise.