How long should you run a personalization test before calling a winner?

October 7, 2026

Short answer: Run the test until it has seen at least two full business cycles, usually two to four weeks, and enough conversions to detect the effect size you care about. Shorter than that and weekly patterns distort the result; longer than that and you are usually burning opportunity waiting for a precision you do not need. Pre-commit to the duration and the decision rule before the test starts.

Why the fixed-day rule fails

The folk wisdom says run every test for two weeks. That rule is wrong in both directions. A high-traffic homepage test can reach a reliable read in days, while a low-traffic personalization test on a segment of email subscribers might need two months. Duration should come from the math of your traffic and the size of the effect you need to see, not from a calendar superstition.

Weekly cycles are the hidden variable. Ecommerce traffic has a strong day-of-week pattern: different visitors, different intent, different conversion rates on weekends versus weekdays. A test that runs five weekdays has not seen the weekend shopper at all. Any duration shorter than one full weekly cycle is measuring a slice of your audience and calling it the whole.

Effect size sets the sample requirement. Detecting a 20 percent lift takes far fewer conversions than detecting a 2 percent lift. Before starting, decide the smallest effect that would change your decision. If only a 10 percent lift would justify the engineering cost of the personalization, size the test for 10 percent and stop when the answer is clear.

The two-cycle minimum

Two full business cycles is the practical floor for most personalization tests. For a typical store that means two weeks: enough to see each weekday twice and to smooth out the noise from one unusual day. Tests that must cover monthly patterns, like payday effects or subscription renewals, need two months instead.

The reason for two rather than one is variance estimation. A single cycle tells you what happened; two cycles tell you how much it varies. If week one favors the variant and week two favors the control, you have learned something important: the effect is not stable, and a longer run or a segmented analysis is needed before deciding.

Novelty effects are the other reason. Personalization changes are noticeable, and noticeable changes get a curiosity bump that fades. The first cycle often overstates the effect as visitors react to the newness. By the second cycle the novelty has worn off and you are measuring the durable response, which is what you will actually live with.

Knowing when to stop early

Stop early for harm, never for victory. If the variant is clearly degrading the experience, like a personalization bug showing wrong products or a layout breaking on mobile, kill the test immediately. Peeking at results to stop a losing test is fine; peeking to declare an early winner is how teams ship false positives.

Use a pre-committed decision rule. Before launch, write down the duration, the primary metric, and the threshold for a decision. This is the single highest-leverage testing discipline: it prevents the endless maybe-one-more-week drift that keeps inconclusive tests running and it prevents victory-peeking. The rule makes the call, not the analyst's optimism.

Watch the segment breakdowns as the test runs, not to decide early but to catch implementation problems. If the variant wins on desktop and collapses on mobile, that is a bug to fix, not a result to call. Monitoring for breakage is legitimate; monitoring for an excuse to stop early is not.

After the decision

A winner is a hypothesis until it is rolled out and held. Roll the winning variant to 100 percent and keep measuring for at least one more cycle. Rollout effects are real: the full-traffic behavior sometimes differs from the test behavior, especially for personalization that interacts with caching, audience composition, or downstream systems.

Document the context with the result. The test ran during a specific season, with a specific audience mix, against a specific control. That context is part of the finding. A personalization that won in November may not win in February, and the documentation is what stops the team from treating a seasonal result as a universal truth.

Bank the learning, not just the lift. Even inconclusive tests teach you about your audience: which segments respond, which metrics move together, what effect sizes are realistic. The teams that improve fastest are the ones whose test archive is searchable, not the ones with the highest win rate.