Why Your A/B Test Results Might Be Hidden by the "Snowball Effect"

In the world of data analysis, we are often attracted to complex machine learning models and deep learning architectures, but we overlook a humble yet powerful tool: linear regression. Today, let’s start from a real scenario and see how it can change our understanding of A/B test results.

Scenario: Banner Testing on an E-commerce Platform

Imagine an online retailer launching a new webpage banner design, aiming to evaluate its impact on users’ average session duration. They conduct an experiment and collect data. Now, the question is: should we analyze these results using a T-test or linear regression?

The Answer from the T-test

Using the traditional T-test tool, we get numbers that look quite promising:

An estimated increase of 0.56 minutes (meaning users spend an average of 33 seconds more). This is the difference between the mean values of the control and treatment groups. It seems straightforward.

An Interesting Finding: Linear Regression Says the Same Thing

But what if we do the same thing with linear regression, treating whether the banner is shown as an independent variable and the average session duration as the output variable? What happens?

The result is surprising: the coefficient for the treatment variable is exactly 0.56—completely consistent with the T-test.

This is no coincidence. Both methods’ null hypotheses are identical, so when calculating the t-statistic and p-value, we get the same result.

But there’s an important caveat: the R² is only 0.008, meaning our model explains less than 1% of the variance. There are many things we haven’t captured.

The Hidden Power: Selection Bias and Covariates

Here’s the key turning point: Using only the treatment variable to explain user behavior might be too simplistic.

In real-world A/B testing, there could be selection bias—that is, systematic differences between the two groups not caused by randomization. For example:

• Returning users see the new banner more frequently than new users
• Certain user segments naturally tend to spend more time on the platform

While random assignment helps mitigate this, it’s hard to eliminate it completely.

Corrected Model: Adding Covariates

What if we add a covariate—say, the average session duration before the experiment?

The model’s performance suddenly improves. R² jumps to 0.86, explaining 86% of the variance. The estimated treatment effect becomes 0.47 minutes.

This difference is significant. In this specific simulated data, the true effect is 0.5 minutes. So, 0.47 (with covariates) is closer to the truth than 0.56 (simple model).

This phenomenon is sometimes called the “snowballing effect”—initial hidden variables can amplify or diminish the estimated effect layer by layer, causing the initial results to deviate from reality.

Why Choose Linear Regression?

So, between 0.47 and 0.56, which is the correct answer?

When we know the true effect, a linear regression model that includes appropriate covariates can usually provide a more accurate estimate. This is because it:

1. Provides a complete picture of model fit: R² tells us how much variance the model explains, which is crucial for assessing reliability
2. Allows control of confounding variables: By adding covariates, we can isolate the true treatment effect and reduce selection bias
3. Improves estimation accuracy: Especially in real-world scenarios with systematic differences

Further Reflection

This principle isn’t limited to T-tests. You can extend the linear regression framework to Welch’s T-test, chi-square tests, and other statistical methods—though each case may require some technical adjustments.

The key insight is: Don’t be blinded by seemingly simple results. Dive into the data, find those hidden variables that might cause the “snowballing effect,” and you’ll discover a more accurate truth.