Fast Facts

Handshake AI is seeking a Staff AI Research Scientist to lead pioneering research in evaluating large language models (LLMs) and advance methodologies for AI understanding and alignment.

Responsibilities: Guide research teams in developing benchmarks, frameworks, and methodologies for LLM evaluation; collaborate with engineers to implement scalable assessment systems; publish findings in leading conferences and journals.

Skills: Expertise in AI evaluation, LLM research, proficiency in Python and PyTorch, and experience in leading research programs.

Qualifications: PhD or equivalent experience in machine learning, cognitive science, or related fields with a strong research background and publication record.

Location: San Francisco, California, United States

Compensation: $350000 - $420000 / Annually

About Handshake AI

Handshake is building the career network for the AI economy. Our three-sided marketplace connects 18 million students and alumni, 1,500+ academic institutions across the U.S. and Europe, and 1 million employers to power how the next generation explores careers, builds skills, and gets hired.

Handshake AI is a human data labeling business that leverages the scale of the largest early career network.We work directly with the world’s leading AI research labs to build a new generation of human data products. From PhDs in physics to undergrads fluent in LLMs, Handshake AI is the trusted partner for domain-specific data and evaluation at scale.

This is a unique opportunity to join a fast-growing team shaping the future of AI through better data, better tools, and better systems—for experts, by experts.

Now’s a great time to join Handshake. Here’s why:

Leading the AI Career Revolution: Be part of the team redefining work in the AI economy for millions worldwide.
Proven Market Demand: Deep employer partnerships across Fortune 500s and the world’s leading AI research labs.
World-Class Team: Leadership from Scale AI, Meta, xAI, Notion, Coinbase, and Palantir, just to name a few.
Capitalized & Scaling: $3.5B valuation from top investors including Kleiner Perkins, True Ventures, Notable Capital, and more.

About the Role

As a Staff Research Scientist, you will drive frontier research on how we define intelligence of frontier models, i.e. develop benchmarks and measurements that help the research community to understand how large language models (LLMs) understand, reason, and interact with human knowledge. You will:

Lead teams of researchers to produce original research in LLM evaluation methodologies, interpretability, and human-AI knowledge alignment.
Develop novel frameworks and assessment techniques that reveal deep insights into model capabilities, limitations, and emergent behaviors.
Collaborate with engineers to translate research breakthroughs into scalable benchmarks, evaluation systems, and standards.
Pioneer new approaches to measuring reasoning, alignment, and trustworthiness in frontier AI systems.
Author high-quality code to enable large-scale experimentation, reproducible evaluation, and knowledge assessment workflows.
Publish in top-tier conferences and journals, establishing new directions in the science of AI evaluation.
Work cross-functionally with leadership, engineers, and external partners to set industry standards for responsible AI evaluation and alignment.

Desired Capabilities

PhD or equivalent research experience in machine learning, computer science, cognitive science, or related fields with focus on AI evaluation, interpretability, or model understanding.
6+ years of academic or industry experience post-doc in a research-first environment
Strong background in LLM research, evaluation methodologies, and/or foundational AI assessment techniques.
Proven ability to independently design, lead, and execute evaluation research programs with novel data types end-to-end.
Deep proficiency in Python and PyTorch for large-scale model analysis, benchmarking, and evaluation.
Experience building or leading novel benchmark development, systematic model assessment, or interpretability studies.
Strong publication record in post-training, evaluation, or interpretability that demonstrates field-defining contributions.
Ability to clearly communicate complex insights and influence both technical and non-technical stakeholders.

Extra Credit

Experience with RLHF, agent modeling, or AI alignment research.
Familiarity with data-centric AI approaches, synthetic data generation, or human-in-the-loop systems.
Understanding of challenges in scaling foundation models (training stability, safety, inference efficiency).
Contributions to open-source libraries or research tooling.
Interest in the societal impact, deployment ethics, and governance of frontier AI systems.

Perks

Handshake delivers benefits that help you feel supported—and thrive at work and in life.

The below benefits are for full-time US employees.

🎯 Ownership: Equity in a fast-growing company

💰 Financial Wellness: 401(k) match, competitive compensation, financial coaching

🍼 Family Support: Paid parental leave, fertility benefits, parental coaching

💝 Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend

📚 Growth: $2,000 learning stipend, ongoing development

💻 Remote & Office: Stipends for home office setup, internet, commuting, and free lunch/gym in our SF office

🏝 Time Off: Flexible PTO, 15 holidays + 2 flex days, winter #ShakeBreak where our whole office closes for a week!

🤝 Connection: Team outings & referral bonuses

Explore our mission, values, and comprehensive US benefits at joinhandshake.com/careers.

Evaluation Methodologies EdTech Jobs

Staff AI Research Scientist - Evaluation, Handshake AI