A/B Testing Chaos? Avoid These Costly Mistakes

The Case of the Confused Conversion Rates: An A/B Testing Cautionary Tale

Sarah, the marketing director at “Bytes & Brews,” a local Atlanta coffee shop chain, was excited. She’d convinced the owner to invest in A/B testing technology to improve their online ordering system. The goal? Increase online orders. Her initial tests, however, showed wildly inconsistent results. Some weeks, Variant A (with a larger “Order Now” button) crushed Variant B (with a customer testimonial). Other weeks, the opposite happened. Sarah was pulling her hair out. Could A/B testing actually be hurting their business? Are you making the same mistakes in your A/B tests?

Sarah’s problem is more common than you might think. Many companies, eager to see immediate results, jump into A/B testing without fully understanding the nuances involved. They end up with data that’s either misleading or outright wrong.

Mistake #1: Insufficient Test Duration

One of the first things I tell clients is: patience. A/B tests need time to gather meaningful data. Sarah, in her eagerness, was only running her tests for a few days at a time. This is a huge problem. As Ronny Kohavi at Microsoft (now at Airbnb) has written extensively, statistical power depends on sample size and effect size, and short tests rarely give you either experimentationanalysis.com.

At Bytes & Brews, Monday mornings were always slammed with office workers ordering coffee for their teams. Weekends, on the other hand, saw a lot more individual orders for lattes and pastries. By ending her tests mid-week, Sarah was inadvertently biasing her results toward one type of customer or another.

Expert Analysis: A general rule of thumb is to run your tests for at least one full business cycle – typically a week. For some businesses, this might be longer. If you see significant day-to-day fluctuations in your traffic or conversion rates, you might need to run your tests for two weeks or even a month to get a clear picture of what’s really going on. I often advise clients to use a sample size calculator to estimate the necessary duration based on their expected conversion rates and desired level of statistical significance. There are plenty of free tools available online.

Mistake #2: Ignoring Audience Segmentation

Here’s what nobody tells you: your audience isn’t a monolith. Different groups of users behave differently. Sarah was looking at aggregate data, which masked important variations in behavior.

She wasn’t considering that Bytes & Brews had a loyalty program with a dedicated app. Users of the app, already familiar with the ordering system, might respond differently to changes than first-time visitors to the website. Similarly, customers ordering from the downtown location near the Georgia State Capitol might have different needs and preferences than those ordering from the suburban location near Northside Hospital.

Expert Analysis: Segmentation is key. Most A/B testing platforms, like Optimizely or VWO, allow you to segment your audience based on a variety of factors, including demographics, behavior, and technology. I recommend creating segments based on factors that are relevant to your business. For Bytes & Brews, this might include:

• New vs. Returning Customers
• Mobile vs. Desktop Users
• Loyalty Program Members vs. Non-Members
• Location (Downtown vs. Suburban)

By analyzing the results of her A/B tests for each segment, Sarah could have uncovered hidden patterns and insights. For example, she might have found that the larger “Order Now” button performed well for new mobile users but alienated loyal app users who preferred the streamlined interface of the original design.

I had a client last year who was testing different headlines on their homepage. The overall results were inconclusive. But when we segmented the audience by traffic source, we discovered that the new headline performed much better for users coming from social media, while the old headline resonated more with users coming from search engines. This allowed us to personalize the homepage content based on the user’s traffic source, leading to a significant increase in conversions.

Mistake #3: Making Changes Mid-Flight

This is a big one. Sarah, in her impatience, would often tweak her A/B tests while they were still running. “I just want to see if this makes a difference!” she’d say. This is a recipe for disaster.

One week, she decided to change the color of the “Order Now” button in Variant A from orange to green, thinking it would be more eye-catching. The next week, she added a limited-time discount code to Variant B. These changes completely invalidated her results. She was no longer testing the original hypothesis. She was testing a hodgepodge of different changes, making it impossible to determine which change (if any) was responsible for the observed results.

Expert Analysis: Once you launch an A/B test, resist the urge to make changes until the test has run its course. Changing variables mid-test introduces confounding factors that make it impossible to draw accurate conclusions. It’s like trying to bake a cake and changing the ingredients halfway through. You’ll end up with a mess. The only exception to this rule is if you discover a major bug or technical issue that is affecting the test. In that case, you should stop the test, fix the issue, and relaunch it.

We ran into this exact issue at my previous firm. A client was testing different pricing models for their software. They made a change to the pricing structure mid-test based on some anecdotal feedback they received. The results were all over the place. We had to explain to them that they had essentially wasted their time and money because the data was now unusable.

The Resolution

After a frustrating month, Sarah reached out to a local Atlanta digital marketing agency for help. The agency reviewed her A/B testing process and identified the mistakes she was making. They helped her design more rigorous tests, segment her audience effectively, and avoid making changes mid-flight.

Within a few weeks, Sarah started seeing much more consistent and reliable results. She discovered that the larger “Order Now” button did, in fact, increase online orders, but only for new mobile users. She also found that adding a customer testimonial to the website improved conversion rates for desktop users.

The key? Focus on a clear hypothesis, proper test duration, audience segmentation, and a commitment to letting the data speak for itself. Don’t let impatience and assumptions cloud your judgment. Remember, A/B testing is a science, not a guessing game.

Don’t let common mistakes derail your A/B testing efforts. Start small, focus on clear hypotheses, and be patient. By following these guidelines, you can unlock the power of A/B testing and drive meaningful improvements to your business. Start by identifying just ONE key metric to improve on your website this week, and design a test to address it. If you’re struggling with slow app performance, consider exploring how to fix slow apps.