A/B Testing How-To for Web Applications

Short Form


Multivariate Testing (also called A/B or Split Testing) lets you determine the best "choice" by showing each choice to a subset of your customers and measuring the results. Of course, you've already heard about it since Google has been doing these sorts of experiments for years, but how to begin?

I'll show you the tools you'll need and code you'll write, with plenty of examples. I'll share tips and tricks from the trenches to improving your customer experiments.


We’ve all attended meetings where the designer and the copy editor can’t agree on a critical part of your web application. They each argue their case before the boss, who ultimately chooses. However, since no customers attended the meeting, how would the boss know the best choice?

The solution? Perform an experiment on some of your customers as test subjects. Show one group Choice A, and another group Choice B. Easy, right? Just remember, the grouping needs to be well distributed as well as random, and you’ll need to measure and report the results.

If your company is anything like mine, you’ll have many concurrent tests, so make sure not to pick on some customers too much, or you’ll skew the results.

I’ll walk through algorithms on both client and server-side and demonstrate some of the tools you can use to make more scientific decisions in building your applications.

Speaking experience

Many years ago, after realizing that Google was experimenting on me personally, I got interested in the subject. Since then, I've produced and tracked many A/B tests while working as a software engineer for the Gilt Groupe and other companies.