Shreya Shankar

AI Researcher and Educator at Independent

AI researcher and co-creator of the definitive online course on evals (the number one course on Maven), who has taught over 2,000 PMs and engineers across 500 companies including OpenAI and Anthropic, known for making evals approachable and actionable for product teams.

The Scientist The Growth Scientist

Dimension Profile

Strategic Vision 45%

Execution & Craft 60%

Data & Experimentation 95%

Growth & Distribution 20%

Team & Leadership 30%

User Empathy & Research 55%

Key Themes

evals as spectrum of quality measurement actionable product improvement over perfect evals evals beyond unit tests new user cohort discovery through data overcoming anti-eval sentiment practical eval methodology for product builders

Episode Summary

Shreya Shankar (alongside Hamel Husain) demystifies evals for AI product builders, explaining that evals are a broad spectrum of quality measurement far beyond simple unit tests. She emphasizes that the goal is actionable product improvement rather than perfect evaluation, explains how evals can discover new user cohorts you didn't know existed, and addresses the anti-eval sentiment that arises when teams have been burned by poorly implemented eval processes. The conversation provides a practical, accessible entry point for PMs and engineers new to building evals.

Leadership Principles

→ The goal is not to do evals perfectly, it's to actionably improve your product — perfection is the enemy of useful measurement
→ Evals are a big spectrum of ways to measure application quality — unit tests are a very small part of that very big puzzle
→ People have been burned by evals in the past and done them badly, which creates anti-eval sentiment — the solution is making the process approachable and practical

Notable Quotes

"The goal is not to do evals perfectly, it's to actionably improve your product."
— On the practical philosophy behind building evals

"People have been burned by evals in the past. People have done evals badly, so then they didn't trust it anymore, and then they're like, 'Oh, I'm anti evals.'"
— On why there is so much controversy around evals in the AI community

"Evals could also be a way of looking at your data regularly to find these new cohorts of people. You think, 'Oh, there's a different way you want to accommodate this new group of people.'"
— On evals as a discovery mechanism for new user segments

Want to know how you compare?

Take the Assessment