search Browse Jobs
Principal Research Scientist - Evaluations Canva · Sydney, NSW, au
Canva logo

Canva

company page ↗ Visa Sponsoring Company

Principal Research Scientist - Evaluations

✓ This company has sponsored visas before. The posting doesn't say so, but it's still worth applying.

🗓️ Posted 2026-08-26 Sydney, NSW, au Full-time Hybrid Science Technology
💰 Salary Not shared 📊 Market A$16,667–A$25,000/mo

About the Company

Empower the world to design. We're a global online visual communications platform on a mission to empower the world to design. Featuring a simple drag-and-drop user interface and a vast range of templates ranging from presentations, documents, websites, social media graphics, posters, apparel to videos, plus a huge library of fonts, stock photography, illustrations, video footage, and audio clips, anyone can take an idea and create something beautiful on Canva on any device, from anywhere in the world. Since our launch in 2013, we've had the crazy big goal of making design accessible to everyone. We were founded on the belief that people shouldn't need to understand complex software to unlock their creativity. We're leveling the playing field and democratizing access to design and visual communication by empowering 100% of the world to communicate in a way that was once limited to the 1%. We've always had a deeper mission surrounding Canva - which we talk about as our 'simple' two-step plan: to build one of the world's most valuable companies, and to do the most good we possibly can. We're committed to our core value of Being a Force for Good, so as the value of our company grows, so too does our ability to have a positive impact on the world. Software Development Specialties: Design, Graphic Design, Web app, Software engineering, Startup, Technology, Innovation, Publishing, product design

Responsibilities

  • ● Define and own the evaluation strategy for Canva Research across all modalities.
  • ● Set the long term direction for how Canva measures generative quality as the space evolves, defining what great evaluation looks like across design, image, video, audio, and agentic workflows.
  • ● Shape and defend the principles teams use to trade off evaluation compute, human data spend, and signal fidelity.
  • ● Treat correlation between auto raters/MLLM judges and human preference as a research problem, validating metrics against human preference and downstream product outcomes, quantifying judge bias, and detecting benchmark saturation and contamination.
  • ● Own the work to close the gap between evaluation and real user experience, ensuring evaluation reflects user experience and product impact, not just isolated performance.
  • ● Build a shared evaluation layer that makes results comparable across regions, and identify cross regional collaboration opportunities that have not yet been chartered.
  • ● Lead cross regional initiatives to standardize evaluation across teams in London, Vienna, San Francisco, Sydney, and China.
  • ● Design and calibrate human preference systems at scale, including rubric design, rater guidelines, inter rater reliability, and cross market cultural calibration.
  • ● Develop learned quality models and reward signals that feed into preference learning, RLHF, RLAIF, and inference time selection, while making explicit where a good judge does not translate into a good reward model.
  • ● Innovate novel architectures and training approaches for vision language models that understand what makes a design effective, providing the reward signal and feedback loop for design generation models.
  • ● Implement agentic and automated evaluation, including MLLM as a judge systems, to assess generative quality.