Senior LLM Quality Analyst

MagicSchool AI • Full-time • Remote (Remote from USA) • $120,000 - $185,000 / year • 1w ago

Fast Facts

MagicSchool is seeking a dedicated Senior LLM Quality Analyst to join their Trust, Safety and Quality team, focused on enhancing AI outputs for educators and students. This role blends data analysis with educational insight, aimed at improving the quality of generative AI technologies in education.

Responsibilities: Design and execute experiments for prompt engineering, maintain dashboards to ensure response quality, and evaluate AI outputs from an educator's perspective to enhance the educational experience.

Skills: Strong analytical skills, experience with BI tools, basic statistics knowledge, and proficiency in SQL are required. Familiarity with analytical Python is a plus.

Qualifications: 5+ years in data analysis, experience with LLMs or AI products, and a background in K-12 education or EdTech is preferred. Team lead experience is advantageous.

Location: This position is based in the USA. Remote work is possible, providing flexibility for employees.

Compensation: $120000 - $185000 / Annually

WHO WE ARE: MagicSchool is the premier generative AI platform for teachers. We're just over 2 years old, and more than 6 million teachers from all over the world have joined our platform. Join a top team at a fast growing company that is working towards real social impact. Make an account and try us out at our website and connect with our passionate community on our Wall of Love.

Senior LLM Quality Analyst

Role Description

As a Senior LLM Quality Analyst, you'll work with the Trust, Safety and Quality team to ensure MagicSchool's AI outputs meet the highest standards for educators and students. This role combines data analysis, prompt engineering, and educational experience. You will design and execute experiments to test prompt strategies, evaluate model performance, and maintain dashboards. You will bring a critical eye to AI quality, strong analytical skills, and a deep understanding of educational needs.

What You'll Do

Design and Execute Prompt Engineering Experiments
Work with the product manager, data scientist and team lead to design prompt strategies and experiments to test them
Use our Evaluation Platform to improve and iterate on MagicSchool prompts from an educators perspective
Design experiments using our Evaluation Platform to compare models for different use cases
Analyze results and make data-driven recommendations for prompt improvements
Maintain Response Quality Reporting and Dashboards
Work with the team lead to design reports and dashboards for performance-over-time reporting
Maintain evaluation dashboards and reporting output using Metabase
Ensure stakeholders have visibility into response quality metrics
Evaluate and Improve AI Outputs
Work with the product manager and data scientist to design evaluators and test cases
Assess AI outputs from an educator's perspective to ensure they meet classroom needs
Identify quality issues and work cross-functionally with product managers, engineers, and the data scientist to address them
Stay Current on LLM Best Practices
Keep up with the latest research and techniques in prompt engineering and LLM evaluation
Bring passion for solving education problems with technology

Qualifications, Competencies & Skills

Strong analytical skills and comfort working in spreadsheets
Experience with BI tools (we use Metabase, but experience with any similar tool is valuable)
Basic statistics knowledge
SQL
Organized and detail-oriented: Maintains clear documentation and systematic approaches to testing
Critical thinker: Can design meaningful experiments and draw insights from data
Gets a lot done: Works hard, resourceful, does what it takes
Adaptable: Smart, learns quickly, curious
Builds relationships easily: Emotionally intelligent, warm communicator
Strong communication skills: Team-first mindset, highly collaborative, can articulate decisions within team's context
Nice to have: Analytical Python experience (huge plus - pandas, jupyter notebooks, data visualization)

Experience

5+ years of professional experience in data analysis
Team Lead Experience
Experience working with LLMs or AI products (prompt engineering, evaluation, quality analysis, or AI product work)
Strongly preferred: K-12 teaching experience or EdTech industry experience
Preferred: Startup experience
Passionate about solving education problems with technology

Application Notice:

Notice: Priority Deadline and Review Start Date

Please note that applications for this position will be accepted until 11/2/25 - applications received after this date will be reviewed on an intermittent basis. While we encourage early submissions, all applications received by the priority deadline will receive equal consideration. Thank you for your interest, and we look forward to reviewing your application.

Why Join Us?

Work on cutting-edge AI technology that directly impacts educators and students.
Join a mission-driven team passionate about making education more efficient and equitable.
Flexibility of working from home, while fostering a unique culture built on relationships, trust, communication, and collaboration with our team - no matter where they live.
Unlimited time off to empower our employees to manage their work-life balance. We work hard for our teachers and users, and encourage our employees to rest and take the time they need.
Choice of employer-paid health insurance plans so that you can take care of yourself and your family. Dental and vision are also offered at very low premiums.
Every employee is offered generous stock options, vested over 4 years.
Plus a 401k match & monthly wellness stipend

Our Values:

Educators are Magic: Educators are the most important ingredient in the educational process - they are the magic, not the AI. Trust them, empower them, and put them at the center of leading change in service of students and families.
Joy and Magic: Bring joy and magic into every learning experience - push the boundaries of what’s possible with AI.
Community: Foster community that supports one another during a time of rapid technological change. Listen to them and serve their needs.
Innovation: The education system is outdated and in need of innovation and change - AI is an opportunity to bring equity, access, and serve the individual needs of students better than we ever have before.
Responsibility: Put responsibility and safety at the forefront of the technological change that AI is bringing to education.
Diversity: Diversity of thought, perspectives, and backgrounds helps us serve the wide audience of educators and students around the world.
Excellence: Educators and students deserve the best - and we strive for the highest quality in everything we do.