Senior Platform Engineer, AI Evaluation

Khan Academy

🇺🇸In-Person - CA$138K–$172K/yr5mo ago

K-12 Engineering Software Engineering AI Development Data Engineering Machine Learning Backend Development

Find Similar Open Jobs

Role Snapshot

Senior Platform Engineer at Khan Academy to develop and enhance an internal AI evaluation framework for assessing quality in AI-driven educational experiences. This 24-month fixed-term role focuses on building evaluation pipelines and driving an evaluation-driven development process across the organization.

Key Responsibilities: Develop evaluation pipelines, gather internal requirements from stakeholders, and create documentation and training materials for evaluation systems. Collaborate with ML data engineers and platform developers to help internal teams adopt offline benchmarking and online experimentation practices.

Skills & Tools: Strong programming expertise in Go, Python, and SQL with deep familiarity with AI systems evaluation. Knowledge of data pipeline frameworks, evaluation strategies (both offline and online), stratified datasets, ground truth labeling, and evaluation scoring schemes from human raters to LLM-as-judge approaches.

Qualifications: Bachelor's or Master's degree in Computer Science or related field with at least 5 years of software engineering experience, including 2+ years specifically in AI evaluation. Experience with evaluation systems and ability to gather requirements and communicate effectively with cross-functional teams.

Location: California

Compensation: $138K–$172K/yr

Prep for this role with AI tools →