Estimated reading time: 4 minutes
The key to creating a culture of experimentation is to make it as simple as possible for developers.
Driving engineering efficiency, enabling product innovation, and leveraging machine learning on a product all have one thing in common – the better strategies you have to measure the effect of changes on your product, the better your chances of success.
For example, without a proper experimentation strategy, it’s harder to understand the impact machine learning algorithms have on user, business, and product metrics. Similarly, if a product redesign is in the works – it’s better to evaluate different versions on a subset of users before potentially launching a redesign that degrades the entire product experience.
Creating a culture of experimentation requires a combination of tooling, statistical rigor, and process. The easier it is to run an experiment on the product, the less likely teams will find a reason not to do it.
That is how you build a culture of experimentation at your company; you make it so easy and seamless that teams have no reason not to measure the effect of the changes on user, business, and product metrics.
Breaking down the experimentation process
If the act of A/B testing is convoluted, teams will be less likely to use this methodology to evaluate changes. It will take time to build a frictionless process, and your focus should likely be spent on clearly illustrating the value proposition of running A/B tests on the product first. Once you have a few teams running A/B tests on your platform, then you can shift gears towards making it easier.
Consider the following to reduce the friction:
- Experimentation anatomy tooling: Build tools that automate setup tasks, such as sample size calculations, hypothesis templates, and basic statistical configurations, reducing the need for teams to handle statistical details manually.
- Experimentation documentation: Create step-by-step guides on designing, launching, and analyzing A/B tests. Adding case studies to illustrate practical applications of an A/B test at the company can help illustrate to teams that are more unfamiliar with the process.
- Automated experiment reviews: Build automated checks that flag common issues in experiment setup, such as underpowered sample sizes or overly short durations, providing feedback on experiment quality before launch.
If you’re further along in your experimentation journey yet still facing low adoption rates, consider conducting workshops and surveys to pinpoint bottlenecks in your current process. It’s often surprising how crucial the procedural side of A/B testing can be.
Even the most advanced experimentation strategies will go unused if they’re overly complex or challenging for teams to implement in their specific A/B testing scenarios.
Frontend tooling
The smartest engineering organizations prioritize tooling – especially if that tooling is used to configure, verify and monitor A/B tests on a product. The easier the user interface is to set up and manage experiments, the more likely adoption will increase.
Every step in the experimentation process should be supported by tools that simplify each task. For instance, if a team needs to configure sample size, there should be a tool that calculates it, eliminating the need for a deep understanding of statistical design. Similarly, if a team wants to implement a holdback, a step-by-step workflow should ensure it’s set up correctly.
The anatomy of an A/B test is crucial. Teams must thoughtfully define intent or hypotheses, success and guardrail metrics, sample size, duration, and other factors. By building tools that streamline these configurations, you reduce the complexity of setting up a well-designed A/B test.
Statistical rigor
A vital cog in this process is your trusty data scientist. Their expertise is essential for correctly designing experiments, from defining sample size and duration, to selecting the right metrics.
If you aim to foster a culture of experimentation at your company, involve data scientists in the process. Their skill in interpreting metrics and telling stories with data is invaluable, especially when highlighting the benefits and impact of A/B testing. Data scientists not only ensure experiments are designed accurately, but also help champion the value proposition of experimentation across teams.
Start with simplicity
By lowering the barrier to running A/B tests, you empower teams to experiment more frequently, have data insights to inform product decisions, and reduce risk before broad releases. Teams will be more likely to test, learn, and iterate quickly when they don’t face friction at every step of the process.
This is the foundation for a culture of experimentation: by making A/B testing accessible and seamless, you pave the way for continuous learning, innovation, and ultimately, more effective product development. In short, make A/B testing easy, and the culture of experimentation will follow.
To truly scale your A/B testing practices and foster a robust experimentation culture, consider broadening your strategy. The classic fixed-horizon A/B test is just one tool in the toolbox.
If you’re ready to explore more advanced strategies and take your experimentation to the next level, check out my latest book: Next Level A/B Testing. In it, you’ll find tips and techniques to help you design better experiments, measure more effectively, and create a culture where data-driven decisions guide your product strategy.
