Ronny Kohavi

World Expert on A/B Testing at Former VP at Airbnb / Microsoft / Amazon

Ronny Kohavi is widely considered the world's foremost expert on A/B testing, having led experimentation at Airbnb, Microsoft (where he built the Experimentation Platform), and Amazon, now consulting and teaching experimentation.

The Scientist The Growth Scientist

Dimension Profile

Strategic Vision 50%

Execution & Craft 65%

Data & Experimentation 95%

Growth & Distribution 55%

Team & Leadership 40%

User Empathy & Research 40%

Key Themes

test everything philosophy experiment allocation strategy high risk high reward bets most experiments fail unexpected impact of small changes experimentation at scale

Episode Summary

Ronny Kohavi, the world's foremost A/B testing expert, makes the case for testing everything — including small bug fixes that can have surprising impact. Drawing on his experience building experimentation platforms at Amazon, Microsoft, and Airbnb, he explains why most experiments fail, how to allocate between safe and high-risk bets, and why it's impossible to experiment too much.

Leadership Principles

→ Test everything — any code change or feature must be in some experiment, because even small bug fixes can have surprising impact
→ It's not possible to experiment too much — the culture of testing should be pervasive
→ Allocate some experiments to high-risk, high-reward ideas that are most likely to fail but could be home runs

Notable Quotes

"Any code change that you make, any feature that you introduce has to be in some experiment. Even small bug fixes, even small changes can sometimes have surprising, unexpected impact."
— On his test-everything philosophy developed across Amazon, Microsoft, and Airbnb

"If you go for something big, try it out, but be ready to fail 80% of the time."
— On why teams must embrace failure as the norm when running high-risk experiments

Want to know how you compare?

Take the Assessment