LLM Quality Analyst

MagicSchool AI • Full-time • Remote (Remote from USA) • $80,000 - $130,000 / year • 1w ago

Fast Facts

MagicSchool is seeking a detail-oriented LLM Quality Analyst to support the quality of its generative AI outputs. Ideal for early-career professionals, this role offers the opportunity to impact education technology and explore various tech career paths.

Responsibilities: Manage feedback intake, replicate user issues, develop test suites, and support evaluator development for AI outputs.

Skills: Strong communication, analytical abilities, adaptability, and experience with LLMs or AI products.

Qualifications: 1-3 years of experience or relevant internship; education background preferred; passion for tech in education.

Location: Remote position within the USA.

Compensation: $80000 - $130000 / Annually

WHO WE ARE: MagicSchool is the premier generative AI platform for teachers. We're just over 2 years old, and more than 6 million teachers from all over the world have joined our platform. Join a top team at a fast growing company that is working towards real social impact. Make an account and try us out at our website and connect with our passionate community on our Wall of Love.

LLM Quality Analyst

Role Description

As a LLM Quality Analyst, you'll work with the Trust, Safety and Quality team to support the quality and reliability of MagicSchool's AI outputs. This is an excellent early career role who wants to work in tech and explore different career paths like product management or data analysis. You will work closely with the Senior LLM Quality Analyst, handling feedback intake, ground truthing LLM responses, and helping maintain evaluation test suites. You will be organized, self-motivated, and excited to get things done quickly in a fast-paced startup environment.

What You'll Do

Manage LLM Feedback Intake and Triage
Intake, triage, replicate, and prioritize user feedback on LLM outputs
Identify patterns in quality issues and escalate appropriately
Act as human-in-the-loop for quality checks
Manage and develop test cases
Review and label LLM-generated responses for evaluators
Generate ground truth datasets to validate evaluator accuracy and consistency
Support Evaluator and Judge Development
Help test and validate new judges as they are developed
Draft and iterate on judge prompts and scoring rubrics to improve coverage and quality
Assist with prompt adjustments and writing
Build and Maintain Evaluation Test Suites
Create and maintain test suites that allow for regular evaluation of LLM performance and evaluator rubrics
Compile collections of prompts, expected responses, and target evaluation scores
Ensure test coverage includes edge cases and diverse content
Support Evaluation Dashboards
Assist in building dashboards to visualize and monitor evaluator outputs, consistency, and regressions
Collect and prepare data for analysis

Qualifications/Competencies/Skills

Gets a lot done: Works hard, resourceful, does whatever it takes
Self-starter: Takes initiative, doesn't wait to be told what to do
Organized and detail-oriented: Can manage multiple workstreams and maintain clear documentation
Adaptable: Smart, learns fast, curious
Builds relationships easily: Emotionally intelligent, warm communicator
Strong communication skills: Team-first mindset, highly collaborative
Comfortable working in spreadsheets and organizing data
Basic analytical skills and ability to identify patterns in data
Strong written communication skills for drafting prompts and documentation
Experience with LLMs or AI products

Experience

Education background - worked in education or an education degree
1-3 years of professional experience (or recent graduate with relevant internship experience)
Passionate about solving education problems with technology
Excited to explore different areas of tech (product, data analysis, quality assurance)
Preferred: Startup or fast-paced environment experience

Application Notice:

Notice: Priority Deadline and Review Start Date

Please note that applications for this position will be accepted until 11/2/25 - applications received after this date will be reviewed on an intermittent basis. While we encourage early submissions, all applications received by the priority deadline will receive equal consideration. Thank you for your interest, and we look forward to reviewing your application.

Why Join Us?

Work on cutting-edge AI technology that directly impacts educators and students.
Join a mission-driven team passionate about making education more efficient and equitable.
Flexibility of working from home, while fostering a unique culture built on relationships, trust, communication, and collaboration with our team - no matter where they live.
Unlimited time off to empower our employees to manage their work-life balance. We work hard for our teachers and users, and encourage our employees to rest and take the time they need.
Choice of employer-paid health insurance plans so that you can take care of yourself and your family. Dental and vision are also offered at very low premiums.
Every employee is offered generous stock options, vested over 4 years.
Plus a 401k match & monthly wellness stipend

Our Values:

Educators are Magic: Educators are the most important ingredient in the educational process - they are the magic, not the AI. Trust them, empower them, and put them at the center of leading change in service of students and families.
Joy and Magic: Bring joy and magic into every learning experience - push the boundaries of what’s possible with AI.
Community: Foster community that supports one another during a time of rapid technological change. Listen to them and serve their needs.
Innovation: The education system is outdated and in need of innovation and change - AI is an opportunity to bring equity, access, and serve the individual needs of students better than we ever have before.
Responsibility: Put responsibility and safety at the forefront of the technological change that AI is bringing to education.
Diversity: Diversity of thought, perspectives, and backgrounds helps us serve the wide audience of educators and students around the world.
Excellence: Educators and students deserve the best - and we strive for the highest quality in everything we do.