What is the right way to A/B test product recommendations?
Updated October 6, 2026 ยท KilterMuse answers
Short answer: The right way to A/B test product recommendations is to randomize at the visitor level, hold the test for full business cycles, and measure incremental revenue per visitor rather than click-through rate. The most common failure is testing the algorithm against a weak control: beating a random or popularity-based baseline proves the recommendations are better than nothing, not that they are good. A meaningful test compares the new algorithm against the current production system, with the same placements, the same traffic split, and a primary metric of revenue per visitor that captures both the clicks and what happens after them. Run for at least two full weekly cycles, segment by new versus returning visitors, and do not peek at results until the pre-registered sample size is reached.
Why most recommendation tests lie
The lies start with the control. Teams test their new collaborative filtering model against a most popular baseline and celebrate a forty percent click lift, but most popular was never the alternative; the production system was. The honest control is the current recommender, running unchanged, because that is what the business would actually keep if the test fails. Anything weaker inflates the apparent win and leads to shipping algorithms that are better than a strawman and worse than what you had.
The second lie is the metric. Click-through rate on recommendations is easy to game: sensational products, low prices, and curiosity-bait images all earn clicks that never convert. An algorithm that maximizes clicks can actively reduce revenue by steering shoppers toward clickbait instead of purchasable products. Revenue per visitor is the metric that keeps the test honest, because it captures the full journey from recommendation click through purchase, including the effects on products the shopper finds afterward. Tests optimized for clicks ship clickbait; tests optimized for revenue ship revenue.
Designing a test that isolates the algorithm
Randomize at the visitor level, not the session or pageview level. A visitor who sees algorithm A on one page and algorithm B on the next is contaminated; their behavior cannot be attributed cleanly. Visitor-level randomization with sticky assignment ensures each shopper experiences one consistent recommendation strategy for the whole test. This matters especially for recommendations, where the effect compounds across a session: the first recommendation influences what the shopper views, which influences the next recommendation.
Hold placements and presentation constant. If the new algorithm also gets a bigger widget or a better position, the test measures the redesign, not the algorithm. The only thing that should differ between variants is the ranking logic producing the product order. This discipline feels constraining, but it is what makes the result interpretable: when revenue per visitor moves, you know the algorithm did it. Document the exact configuration of both variants before launch, because the post-test argument about what was actually tested is where bad decisions get made.
How long to run and when to stop
Run for full business cycles, with two weeks as the practical minimum for most stores. Recommendation effects have weekly seasonality: weekend browsers behave differently from weekday buyers, and a test that covers only weekdays mismeasures the weekend audience. Shorter tests also fall prey to novelty effects, where shoppers click the new recommendations out of curiosity rather than preference, inflating early results that fade. If the test needs a fast read, pre-register a smaller minimum detectable effect and accept the uncertainty explicitly rather than stopping early on a peek.
Pre-register the stopping rule and honor it. Decide before launch how long the test runs and what effect size counts as a win, then do not check the results until the sample size is reached. Peeking and stopping at the first significant moment is the surest way to ship false positives, because random fluctuations guarantee that some early peek looks significant. The teams with the best recommendation systems are not the ones with the cleverest algorithms; they are the ones with the most disciplined testing, because discipline is what compounds true improvements while discarding illusions.
Segmenting the results honestly
Always segment by new versus returning visitors. Recommendations behave completely differently for these audiences: new visitors have no history, so the algorithm is really being tested on its cold-start strategy, while returning visitors test its personalization depth. An algorithm that wins overall by crushing it with new visitors while hurting returning ones is a bad trade, since returning visitors are the higher-value segment. The blended result hides this; the segments reveal it.
Segment by placement too. Homepage, product page, cart, and post-purchase recommendations serve different intents and have different baselines; an algorithm can win in one placement and lose in another. Roll out placement by placement based on the segmented evidence rather than shipping globally on a blended win. And keep a holdout: reserve a small permanent control group on the old algorithm to measure long-term effects and to have a comparison ready when the next challenger arrives. The holdout is cheap insurance against the slow drift that makes every algorithm look good against its own past.
Can we test with 50/50 traffic splits?
Yes, for the test duration, though many teams use smaller challenger allocations to limit risk. The split size affects how fast you reach significance, not the validity. What matters is that the split is random, sticky at the visitor level, and held for full business cycles.
What if the new algorithm needs user history to work?
Then test it where history exists: run the test on returning visitors, or run a separate cold-start test for new visitors with an algorithm designed for sparse data. Do not test a history-dependent algorithm on new visitors and conclude personalization does not work; you tested the wrong thing.
How do we handle multiple recommendation tests running at once?
With mutual exclusion or factorial design, not hope. Overlapping tests on the same placements contaminate each other. Either exclude visitors in one test from the other, or design the tests to measure interaction effects explicitly. The testing roadmap needs a traffic plan, not just a list of ideas.