What decades of replicated field experiments say about persuasion, which famous effects are real, which are folklore, and how to run a test your traffic can actually hear.

Every marketing team has a meeting where somebody senior says, “I just think the green button feels more trustworthy.” Then the room nods, because the person who said it approves the budget. This article is for the next meeting. It collects what large, pre-registered replications and millions of real A/B tests have found, with the numbers attached, so you can walk in with evidence instead of vibes. It is written in question-and-answer form so you (and any AI summarizing it) can grab exactly the piece you need.

Q1. Why should a test outrank the smartest person in the room?

Because the smartest person in the room is wrong most of the time, and so are you, and so am I. The companies with the biggest experimentation programs have published their batting averages, and they are humbling.

1. At Microsoft, Ronny Kohavi found about one third of experiments win, one third do nothing, and one third actually hurt the metric they were meant to improve. In mature products like Bing the win rate fell to roughly 10% to 20% (Kohavi interview, DiggInTravel).
2. Booking.com, which runs more than 1,000 concurrent tests, reports that about 9 out of 10 ideas fail to improve metrics (Thomke, Harvard Business Review, 2020, via O’Reilly Radar).
3. At Airbnb, only 20 of 250 search ideas moved key metrics. Those 20 still added about 6% to booking conversion, worth hundreds of millions (AB Tasty, 1,000 Experiments Club).
4. The Obama 2008 campaign tested 4 buttons against 6 images and videos (24 combinations). The staff favorite, a video, finished dead last. A family photo plus a meek “Learn More” button lifted sign-ups 40.6%, which Dan Siroker estimates meant 2.88 million extra emails and about $60 million in donations (Siroker, A/B Testing).
5. Bing’s biggest revenue win in its history, a tweak to ad titles that raised revenue 12% (over $100M a year), sat in the backlog for more than six months because it seemed unimportant (Nish Bhana summary of Kohavi).

Q2. Which behavioral science effects actually hold up in the field?

A handful of effects keep showing up in huge samples, in different countries, run by people with no stake in the outcome. These are the ones worth building on.

1. Defaults. A meta-analysis of 58 studies (73,675 people) found a pooled effect of d = 0.68, roughly a 27% shift in binary choices (Jachimowicz et al., Behavioural Public Policy, 2019). The classic case: automatic enrollment raised 401(k) participation among new hires from 37% to 86% (Madrian & Shea, 2001). In copy terms, the pre-checked box and the pre-selected plan do real work.
2. Social norms, when true and local. The UK tax authority added “9 out of 10 people pay their tax on time” to reminder letters sent to over 100,000 taxpayers. Payment rose 5.1 percentage points (Hallsworth, List, Metcalfe & Vlaev, 2017). In the earlier pilot of 140,000 letters, payment went from 67.5% (plain letter) to 83% (hometown norm) (UCD summary). Hotel signs saying most guests reuse towels raised reuse from 35.1% to 44.1% (Goldstein, Cialdini & Griskevicius, 2008).
3. Scarcity, social proof and urgency in e-commerce. Qubit pooled 6,700 experiments across 2 billion user journeys, audited by PwC. Low-stock messages averaged +2.9% revenue per visitor, social proof +2.3%, countdown urgency +1.5% (Qubit; CMSWire). Small numbers, yes. On $50 million of revenue, 2.9% is $1.45 million.
4. Anchoring. It was one of the most robust results in Many Labs 1, where 10 of 13 classic effects replicated across 36 labs. Showing a higher reference price first still makes the real price feel smaller.
5. One warning label. A crawl of about 11,000 shopping sites found dark patterns (fake timers, invented “only 2 left”) on roughly 11% of them (Mathur et al., 2019). Fake scarcity may win the week and lose the customer, and regulators read those papers too.

Q3. Which famous effects fall apart when you look closely?

Plenty. The pattern is consistent: a small, exciting lab study, a TED talk, a bestseller, and then a big replication that finds a fraction of the effect or none at all.

1. Published nudges shrink about sixfold at scale. Stefano DellaVigna and Elizabeth Linos compared academic nudge papers with 126 RCTs covering 23 million people run by two US Nudge Units. Journals showed an average 8.7 percentage point lift (+33.4%). At scale it was 1.4 points (+8.0%). About 70% of the gap came from selective publication plus low power (Econometrica, 2022).
2. The average nudge may be close to zero. A PNAS meta-analysis of 447 effect sizes from 212 papers reported d = 0.43 (Mertens et al., 2022). A re-analysis that corrected for publication bias concluded there was “no evidence remains that nudges are effective” on average, while noting big variation between nudges (Maier et al., 2022). Translation: some nudges work, the average one is oversold.
3. Ego depletion. The original meta-analysis said d = 0.62. A 23-lab replication with 2,141 people found d = 0.04, with a confidence interval that includes zero (Hagger et al., 2016). So much for “tired shoppers buy more.”
4. Choice overload. The famous jam study found 30% bought from 6 jars versus 3% from 24. A meta-analysis of 50 experiments (5,036 people) found a mean effect of “virtually zero” (Scheibehenne et al., 2010). Cutting options can help; it is situational, so test it.
5. Psychology in general. Many Labs 2 re-ran 28 findings in 125 samples across 36 countries. Only 15 (54%) replicated, and the median effect size fell from d = 0.60 to d = 0.15 (Klein et al., 2018). The Open Science Collaboration replicated just 36% of 97 studies (Science, 2015).
6. Button colors. In the Qubit data, color changes averaged 0.0%, button restyles -0.2%, and generic call-to-action wording -0.3%. Google famously tested 41 shades of blue. Google also has a few billion visitors to spare.

Table 1. The replication scorecard

Effect	Original claim	What large replications found	Verdict for copywriters
Defaults / pre-selection	401(k): 37% to 86%	d = 0.68 across 58 studies	Holds up. Use it honestly.
Local social norms	Tax letters +5.1 pts	Replicated in 200,000+ taxpayers	Holds up when the claim is true.
Scarcity / social proof	Big lab effects	+2.9% / +2.3% RPV in 6,700 tests	Holds up, modest size.
Anchoring	Strong in lab	Replicated in Many Labs 1	Holds up.
Nudges (average)	+8.7 pts in journals	+1.4 pts at scale	Real but ~6x smaller.
Choice overload	Jam: 30% vs 3%	Mean effect “virtually zero”	Context-dependent. Test it.
Ego depletion	d = 0.62	d = 0.04 in 23 labs	Does not hold up.
Button color	Folk wisdom	0.0% mean uplift	Rarely worth your traffic.

Q4. How big a win should I realistically expect?

Smaller than your agency deck says. Across the 6,700 Qubit tests the distribution of conversion uplift was centered near zero, with the 5th and 95th percentiles at roughly plus or minus 1.2% (Browne & Swarbrick Jones, summarized in arXiv 1909.03457). At Bing, successful changes typically moved key metrics 0.1% to 1%.

1. Twyman’s Law applies: “Any figure that looks interesting or different is usually wrong.” A +40% result on a button change deserves an audit before a press release.
2. The winner’s curse is brutal on small tests. I simulated 20,000 tests with a true lift of 5%. With 5,000 visitors per arm only 6% of tests reached significance, and those “winners” reported an average lift of +31%, more than six times reality. At 300,000 per arm, winners reported +5.2%. Underpowered tests crown exaggerated winners that then “regress” after launch.

Q5. How do I size a test my traffic can actually detect?

Work backward from the smallest lift worth caring about (the minimum detectable effect, or MDE). A handy rule of thumb: visitors per arm is about 16 times p(1 minus p), divided by the squared absolute lift, for 80% power at 5% significance. The table below does the math for you.

Table 2. Visitors needed per arm (80% power, two-sided alpha 0.05)

Baseline conversion	Detect +5% relative	Detect +10%	Detect +20%	Detect +30%
1%	637,000	163,000	42,700	19,800
2%	315,000	80,700	21,100	9,800
3%	208,000	53,200	13,900	6,450
5%	122,000	31,200	8,160	3,780
10%	57,800	14,700	3,840	1,770

1. Worked example. A store converting at 3% with 20,000 visitors a week puts 10,000 in each arm. Detecting a +10% lift needs 53,200 per arm, so about 5.3 weeks. Detecting +5% needs 208,000 per arm, which is 21 weeks. That test is a wish, and you should skip it.
2. Low traffic? Swing bigger. Test a new offer, a new headline promise or a new page structure, since those can plausibly move 20% or more. Save the microcopy tweaks for when you have Bing-sized traffic.
3. Move the metric up the funnel (add-to-cart or click-through have higher baselines), then confirm revenue with a longer holdout.
4. Cut the noise. CUPED, which adjusts for each user’s pre-test behavior, cut variance by about 50% in Microsoft’s tests (Deng et al., 2013), which roughly halves the sample you need.
5. Run whole weeks (minimum two) so weekday and weekend shoppers both count, and check for sample ratio mismatch. Microsoft researchers reported SRMs in about 6% of experiments (Fabijan et al., 2019); a 50/50 split that lands at 52/48 means the data is broken.

Q6. What goes wrong when people “peek” at results?

Peeking turns a 5% false alarm rate into a coin flip with better branding.

1. An analysis of 2,101 commercial Optimizely experiments found about 57% of experimenters stopped tests right when a positive effect crossed 90% confidence. That habit raised the false discovery rate from 33% to 42%, and about 70% of all tested effects were truly null (Berman, Pekelis, Scott & Van den Bulte, MSI 2018).
2. A follow-up on 4,964 effects estimated that at the standard 5% threshold, about 18% of “significant” wins were false, rising to 28% at a 10% threshold (Berman & Van den Bulte, Management Science, 2021).
3. My own simulation of 4,000 A/A tests (identical pages, 1,000 visitors per arm per day) checked the dashboard daily. After 28 days, 27% of identical pages had been declared “winners.” You were promised 5%.
4. Base rates matter too. If only 10% of ideas truly work, then even a clean, fully powered test at p < 0.05 has about a 22% chance that its winner is a dud (GrowthBook, citing Kohavi). If you must peek, use a sequential method (Optimizely’s Stats Engine, or group-sequential boundaries) built for it.

Q7. When do I stop testing and just decide?

Testing is a tool for buying certainty. Stop buying when the certainty costs more than the mistake.

Table 3. Stop-and-decide rules

Situation	What to do
Reached the pre-planned sample and duration	Decide now. Ship if the result clears your threshold; keep control if not. No extensions “to see if it gets there.”
A guardrail metric (refunds, unsubscribes, load time) is clearly hurt	Stop early. Harm is a valid reason to peek.
Sample ratio mismatch or tracking bug	Stop, fix, restart. Results from broken plumbing are fiction.
Flat after full sample	Ship whichever version is cheaper or simpler to maintain. Flat is an answer.
Traffic cannot detect less than ~30% lift	Skip the A/B test. Use proven principles (Q2), user interviews, and a before/after holdout.
Change is cheap, reversible and low-risk	Berman & Van den Bulte show an expected-gain maximizer can ship any positive result; demand less proof.
Legal, ethical or brand-critical change	Just do it. Honesty is not a test variant.

1. Write the plan before launch: hypothesis, primary metric, MDE, sample size, run length, guardrails. Five lines in a shared doc stops most p-hacking cold.
2. Keep a small long-term holdout (say 5% of traffic) that sees none of the quarter’s winners. It tells you whether 12 “wins” added up to anything real.
3. Retest big winners. If a result looks too good, a replication costs a few weeks and can save a bad rollout.

Q8. What is the one-paragraph takeaway?

Build on the effects that survived big replications (defaults, true social proof, real scarcity, anchoring), and treat the TED-talk classics as hypotheses. Expect wins of 1% to 3%, which compound nicely. Size tests from your traffic before you write a word of copy. Decide your stopping rule in advance and stick to it. And the next time someone says the green button “feels more trustworthy,” smile, nod, and ask how many visitors per arm they had in mind.

Sources

1. DellaVigna, S. & Linos, E. (2022). RCTs to Scale. Econometrica 90(1). nber.org/papers/w27594
2. Klein, R. et al. (2018). Many Labs 2. AMPPS. journals.sagepub.com; COS summary
3. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. science.org
4. Camerer, C. et al. (2018). Replicating social science experiments in Nature and Science. nature.com
5. Mertens, S. et al. (2022). The effectiveness of nudging. PNAS. pnas.org
6. Maier, M. et al. (2022). No evidence for nudging after adjusting for publication bias. PNAS. pnas.org
7. Jachimowicz, J. et al. (2019). A meta-analysis of default effects. Behavioural Public Policy. cambridge.org
8. Madrian, B. & Shea, D. (2001). The Power of Suggestion. QJE. nber.org/papers/w7682
9. Hallsworth, M. et al. (2017). The behavioralist as tax collector. J. Public Economics. wrap.warwick.ac.uk
10. Hagger, M. et al. (2016). Multilab replication of ego depletion. Perspectives on Psych. Science. journals.sagepub.com
11. Scheibehenne, B. et al. (2010). Can there ever be too many options? J. Consumer Research. academic.oup.com
12. Qubit / Browne & Swarbrick Jones (2017). What works in e-commerce: 6,700 experiments. qubit.com
13. Berman, R. et al. (2018). p-Hacking and False Discovery in A/B Testing. SSRN
14. Berman, R. & Van den Bulte, C. (2021). False Discovery in A/B Testing. Management Science. ron-berman.com
15. Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge University Press; Kohavi interviews at DiggInTravel and AB Tasty
16. Siroker, D. & Koomen, P. (2013). A/B Testing. Wiley. perlego.com
17. Mathur, A. et al. (2019). Dark Patterns at Scale. arxiv.org
18. Miller, E. How Not to Run an A/B Test. evanmiller.org
19. Charts marked “author simulation” were computed in Python for this article (4,000 A/A tests; 20,000 runs per sample size).