MagicSchool AI is looking for an Associate LLM Quality Analyst to join our Trust, Safety and Quality team. In this early-career role, you will support the quality and reliability of our AI outputs, working on technology that directly impacts educators and students. This is a great opportunity for someone who wants to work in tech and explore different career paths like product management or data analysis.
What You'll Do
- Intake, triage, replicate, and prioritize user feedback on LLM outputs.
- Identify patterns in quality issues and escalate them appropriately.
- Act as a human-in-the-loop for quality checks.
- Manage and develop test cases.
- Review and label LLM-generated responses for evaluators.
- Generate ground truth datasets to validate evaluator accuracy and consistency.
- Help test and validate new judges (evaluators) as they are developed.
- Draft and iterate on judge prompts and scoring rubrics to improve coverage and quality.
- Assist with prompt adjustments and writing.
- Create and maintain test suites for regular evaluation of LLM performance and evaluator rubrics.
- Compile collections of prompts, expected responses, and target evaluation scores.
- Ensure test coverage includes edge cases and diverse content.
- Assist in building dashboards to visualize and monitor evaluator outputs, consistency, and regressions.
- Collect and prepare data for analysis.
What We're Looking For
- Gets a lot done: Works hard, is resourceful, and does whatever it takes.
- Self-starter: Takes initiative and doesn't wait to be told what to do.
- Organized and detail-oriented: Can manage multiple workstreams and maintain clear documentation.
- Adaptable: Smart, learns fast, and curious.
- Builds relationships easily: Emotionally intelligent and a warm communicator.
- Strong communication skills: Team-first mindset and highly collaborative.
- Comfortable working in spreadsheets and organizing data.
- Basic analytical skills and the ability to identify patterns in data.
- Strong written communication skills for drafting prompts and documentation.
- Experience with LLMs or AI products.
- Education background—either professional experience in education or an education degree.
- 1-3 years of professional experience (or a recent graduate with relevant internship experience).
- Passionate about solving education problems with technology.
- Excited to explore different areas of tech (product, data analysis, quality assurance).
Nice to Have
- Startup or fast-paced environment experience.
Team & Environment
You will work with the Trust, Safety and Quality team and report to a Senior LLM Quality Analyst.
Benefits & Compensation
- Work on AI technology that directly impacts educators and students.
- Join a mission-driven team passionate about making education more efficient and equitable.
- Flexibility of working from home.
- Unlimited time off.
- Choice of employer-paid health insurance plans.
- Dental and vision offered at very low premiums.
- Generous stock options, vested over 4 years.
- 401k match.
- Monthly wellness stipend.
Work Mode
This is a fully remote position.
MagicSchool AI is committed to building a diverse and inclusive team.
