A Q&A guide to the findings that hold up, the ones that fell apart, and how to tell the difference

Marketing has plenty of studies. What it lacks is the habit of checking which ones survive a second look. A cute lab result becomes a conference slide, the slide becomes a LinkedIn carousel, and by the time it reaches your budget meeting the confidence interval has been left behind at the airport. Here are the findings with real weight behind them, the famous ones that crumbled, and a quick test for spotting a sales deck wearing a lab coat. Sources are named in the text and linked at the end.

Q1. How much published marketing research has actually been re-tested?

• Very little. Hubbard and Armstrong checked 1,120 papers from three top marketing journals and found zero replications; extensions made up just 1.8% of papers. A 2007 update found the rate had fallen to 1.2% and warned that “practitioners should be skeptical about using the results published in marketing journals.”
• When someone finally reruns the experiments, the scorecard is rough. The Open MKT Replication Tracker counts 5 successes out of 45 high-powered direct replications of marketing studies, about 11%. Most failures are lab studies.
• For comparison, psychology’s Open Science Collaboration (2015) replicated 36% of 97 studies, Many Labs 2 got 54%, and Many Labs 1 got 10 of 13. Big field experiments do far better, which is the thread running through this piece.

Sources: Open MKT tracker; Open Science Collaboration (2015); Klein et al. (2014, 2018); DellaVigna & Linos (2022), Econometrica.

Q2. Which dozen findings have the strongest evidence behind them?

Table 1. Twelve findings that hold up

#	Finding	The number	Evidence base
1	Price moves sales far more than ads do	Avg. price elasticity -2.62	1,851 estimates, 81 studies (Bijmolt et al. 2005)
2	Ad elasticities are small and have shrunk	0.12 short-term; 0.24 long-term	751 + 402 estimates, 56 studies (Sethuraman et al. 2011)
3	Most brands overspend on TV at the margin	Median elasticity about 0.014; negative marginal ROI for 80%+ of brands	288 brands, Nielsen data (Shapiro, Hitsch & Tuchman 2021)
4	Ad returns are brutally hard to measure	Median ROI confidence interval over 100 points wide	25 RCTs, $2.8M in spend (Lewis & Rao 2015)
5	Non-experimental attribution overstates lift	Off by 3x or more in half the studies	15 Facebook RCTs, 1.6B impressions (Gordon et al. 2019)
6	Brand search ads for a famous brand are mostly wasted	99.5% of clicks recaptured by organic; ROI of -63%	eBay randomized experiments (Blake, Nosko & Tadelis 2015)
7	Defaults change choices	d = 0.68, about a 27% shift on yes/no outcomes	58 studies, n = 73,675 (Jachimowicz et al. 2019)
8	Nudges work, but small at scale	+1.4 percentage points (8%)	126 RCTs, 23M people (DellaVigna & Linos 2022)
9	Neighbor-comparison letters cut energy use	About 2.0% (range 1.4 to 3.3%)	17, then 111 RCTs, 8.6M homes (Allcott 2011, 2015)
10	Left-digit pricing bias is real	1 cent above $X.99 acts like 20+ cents	3,500 products, 25 chains (Strulov-Shlain 2023)
11	Double jeopardy: small brands get fewer and slightly less loyal buyers	Loyalty tracks brand size	Decades of panel data, many categories (Ehrenberg et al. 1990)
12	Anchoring on arbitrary numbers	Replicated in the aggregate	36 samples, 6,344 people (Many Labs 1, 2014)

Q3. So does advertising work or not?

• It works, in the way a gym membership works: real effect, smaller than the brochure. Sethuraman, Tellis and Briesch pooled 751 short-term estimates and found an average elasticity of 0.12, down from the 0.22 reported in 1984. Long-term: 0.24, versus 0.41 before. Only about 53% of estimates were significantly above zero.
• Shapiro, Hitsch and Tuchman then did something unusual. They picked 288 brands by a fixed rule and reported every result, nulls included. The median TV elasticity was roughly 0.014, so doubling your TV weight buys you about 1% more sales. Marginal ROI was negative for more than 80% of brands, and the full ad schedule paid off for only about a third. Their interactive site lets you browse the results yourself (see sources).
• Now compare price: an elasticity of -2.62 means a 10% price cut lifts volume about 26%. Margin disappears too, so that is no argument for discounting. It does tell you which lever the customer notices first.

Sources: Assmus, Farley & Lehmann (1984) via Sethuraman et al. (2011); Shapiro et al. (2021); Chan et al. (2011), Google; Blake, Nosko & Tadelis (2015).

Q4. Is paid search really incremental?

• Google’s own Search Ads Pause Studies (400+ cases, all authors at Google) reported that 89% of paid clicks vanished when ads were paused, with organic results making up only the rest. An update of 5,300 cases put it at 85%. The catch: advertisers chose when to pause, so it is observational.
• eBay ran the randomized version. When it switched off brand-keyword ads, 99.5% of the traffic came right back through organic results. On other keywords, new and infrequent users responded, while loyal users who would have bought anyway ate most of the budget. Estimated ROI: -63%.
• Both can be true. eBay is about as famous as brands get; smaller brands with rivals bidding on their name likely gain more. Run your own holdout instead of borrowing an answer.
• How wrong can attribution be? Across 15 Facebook RCTs, Gordon and colleagues found standard observational estimates off by 3x in half the studies; a 2022 follow-up found matching methods overstated median lift by roughly 7x to 9.5x.

Q5. What do real pricing experiments show?

• Anderson and Simester randomized prices across versions of a women’s clothing catalog. One dress sold 16 units at $34, 21 at $39, and 17 at $44. Charging five dollars more sold about 31% more dresses. Across three experiments, $9 endings raised demand every time, especially on new items.
• Strulov-Shlain scaled that up to 3,500 products across 25 U.S. chains. Shoppers react to a one-cent move above a .99 price as if it were a 20-cent jump, and retailers under-use the effect, leaving an estimated 1 to 4% of gross profit on the table.

Sources: Anderson & Simester (2003); Iyengar & Lepper (2000) as reported in Chernev et al. (2015); Scheibehenne, Greifeneder & Todd (2010).

Q6. Which famous findings fell apart?

Table 2. Headline findings that shrank, stalled, or were retracted

The claim	What happened on the second look
Too much choice kills sales (the jam study)	Original: 30% bought at a 6-jam display vs 3% at 24 jams. A meta-analysis of 50 experiments (N = 5,036) found an average effect near zero; it appears only under narrow conditions (Chernev et al. 2015).
Decoy options reliably steer choice	Yang & Lynn: 91 attempts, 11 reliable effects, and chance results when options were pictures or words. Defenders say it holds in numeric setups.
Signing at the top makes people honest	Reruns with 4,559 and 1,235 people found nothing. The 2012 PNAS paper was retracted in 2021 after Data Colada showed its field data were fabricated.
Smaller plates and bowls make you eat less (Wansink lab)	Cornell found academic misconduct in 2018. Retraction Watch counted at least 18 retracted papers by the time TIME reported his resignation.
Pleasant ad music conditions brand choice (Gorn 1982)	A high-powered direct replication in 2014 failed.
Sensory marketing cues (color, touch, scent)	Of 10 influential studies retested in 2022, 2 replicated (20%), with effect sizes about half the originals.

Q7. Why do effects shrink when you scale them up?

• Publication bias. DellaVigna and Linos compared nudges in academic journals (average +8.7 points, a 33.4% lift) with every trial run by two U.S. Nudge Units (+1.4 points, 8.0%). Selective publication, made worse by small samples, explains about 70% of the gap. A good working rule: expect around a sixth of the published effect.
• Site selection. Allcott showed Opower’s first 10 sites over-predicted results at the next 101 by 0.41 to 0.66 points, worth $560 to $920 million in electricity at national scale. Early adopters were greener and more responsive, and your pilot market probably is too.
• Noise. Lewis and Rao found individual sales are so volatile (coefficient of variation around 10) that a clean ad test can need more than 10 million person-weeks.

Q8. How do I tell a field experiment from a vendor white paper?

Table 3. The five-minute credibility check

Ask this	A real field experiment	A typical vendor white paper
Who got the treatment?	Randomly assigned, with a holdout group	Customers who chose the product
What was measured?	Sales, sign-ups, dollars	Clicks, engagement, “attributed” revenue
How big, how sure?	Thousands to millions; confidence intervals shown	“Our clients”; one lift number, no range
Was the plan set in advance?	Pre-registered or rule-based sample	Analysis chosen after seeing the data
Can anyone check it?	Peer review, public data or code	Proprietary
Who paid?	Conflicts disclosed	The seller is the author

• Insider research can still be good. The eBay paper came from eBay’s own staff, yet it was randomized and published in Econometrica. Design beats letterhead.

Q9. What should a marketer actually do on Monday?

1. Keep a randomized holdout for every major channel so you can see your own incrementality.
2. Test price before creative. Customers respond to price roughly 20 times more.
3. Treat single lab studies as hypotheses, and check the Open MKT tracker before quoting one.
4. Discount any published lift by a lot, and discount attribution-based lift even more.
5. Chase penetration over loyalty schemes. Double jeopardy says loyalty mostly follows size.

Honest marketing science is less flashy than the conference circuit. It is also what keeps working after the applause stops.

Sources

1. Bijmolt, van Heerde & Pieters (2005), Journal of Marketing Research. journals.sagepub.com2. Sethuraman, Tellis & Briesch (2011), Journal of Marketing Research. journals.sagepub.com3. Shapiro, Hitsch & Tuchman (2021), Econometrica, 288 brands. econometricsociety.org4. Shapiro, Hitsch & Tuchman, interactive results by brand (Chicago Booth). advertising-effects.chicagobooth.edu5. Lewis & Rao (2015), Quarterly Journal of Economics. academic.oup.com6. Gordon, Zettelmeyer, Bhargava & Chapsky (2019), Marketing Science. pubsonline.informs.org7. Gordon, Moakler & Zettelmeyer, MSI Report 22-106 (2022). msi.org8. Blake, Nosko & Tadelis (2015), Econometrica / NBER w20171. nber.org9. Chan et al. (2011), Google Search Ads Pause Studies + 2012 update. research.google

10. Jachimowicz, Duncan, Weber & Johnson (2019), Behavioural Public Policy. cambridge.org11. DellaVigna & Linos (2022), Econometrica, RCTs to Scale. econometricsociety.org12. Allcott (2011), Opower results (J-PAL summary). povertyactionlab.org13. Allcott (2015), Site Selection Bias, QJE. nber.org14. Anderson & Simester (2003), Quant. Marketing & Economics. link.springer.com15. Strulov-Shlain (2023), Review of Economic Studies. academic.oup.com16. Ehrenberg, Goodhardt & Barwise (1990), J. of Marketing. journals.sagepub.com17. Klein et al. (2014), Many Labs 1, Social Psychology. econtent.hogrefe.com18. Scheibehenne, Greifeneder & Todd (2010), Journal of Consumer Research. academic.oup.com

19. Chernev, Bockenholt & Goodman (2015), Choice Overload meta-analysis, JCP. chernev.com20. Yang & Lynn (2014), Journal of Marketing Research. journals.sagepub.com21. Kristal et al. (2020), PNAS; Shu et al. (2012) retraction notice, PNAS (2021). pnas.org22. Data Colada [98], Evidence of Fraud in an Influential Field Experiment. datacolada.org23. TIME (2018) on Brian Wansink’s resignation and retractions. time.com24. Open MKT Replication Tracker (Aaron Charlton). openmkt.org25. Motoki & Iseki (2022), sensory marketing replications, Frontiers in Communication. frontiersin.org26. Evanschitzky et al. (2007), Replication Research’s Disturbing Trend, J. Business Research. faculty.wharton.upenn.edu