A field guide to sampling frames, recruitment, question wording, response bias, and the disclosures that let a 200-person study outrank a 20,000-person mystery.
Most industries have a “State of Whatever” report built on a survey of “over 1,000 professionals” that turns out to be 1,000 people who clicked a LinkedIn link during lunch. You can do better with 200. A small survey gets taken seriously when a reader can see exactly who answered, how they were found, what they were asked, and how much wiggle room the numbers carry. Here are the real questions, answered with receipts.
Q1. Why should anyone trust 200 people over 2 million?
- Size does not fix a bad list. In 1936 The Literary Digest mailed 10 million ballots, got 2.38 million back, and predicted Alf Landon would win with 57.08% of the vote. Franklin Roosevelt took 60.8% and 523 electoral votes. George Gallup, working with about 50,000 interviews, called Roosevelt the winner at 55.7% (Wikipedia, The Literary Digest). Its list leaned on phone books and car registrations, a wealthier crowd in 1936.
- A skewed sample can be rescued if you know the skew. Andrew Gelman (of the Statistical Modeling blog) and colleagues polled 345,858 unique Xbox users in 2012. The sample was 93% men (the electorate was 47%) and 65% ages 18 to 29 (the exit poll had 19%). Adjusted with multilevel regression and poststratification, the final estimate missed by just 0.6 points (Wang, Rothschild, Goel & Gelman).
Table 1. Big Samples vs. Good Samples
| Study | Respondents | Known skew | Result |
|---|---|---|---|
| Literary Digest, 1936 | 2,380,000 | Wealthier owners of phones and cars | Overstated Landon by ~20 pts (57% vs. 36.5%) |
| Gallup, 1936 | ~50,000 | Quota sample by demographics | Correct winner (55.7% FDR) |
| Xbox poll, 2012 | 345,858 | 93% male, 65% aged 18 to 29 | Off by 0.6 pts after adjustment |
Q2. What is a sampling frame, and why should I write it down first?
- Your sampling frame is the actual list you draw from. “Independent insurance agencies” is a population. “Agencies on the Ohio, Indiana, and Kentucky license rolls with 2 to 50 employees” is a frame. Write that sentence before you write a single question.
- Your own newsletter list is a frame of fans. Scott Alexander of Astral Codex Ten puts it neatly: “Selection bias is fatal for polls, but only sometimes a problem for correlations.” Translation: a fan list is shaky for “62% of agencies use X” and much sturdier for “agencies using X report faster quote turnaround.”
- Even giant studies say this out loud. The Reuters Institute Digital News Report, fielded online, warns that “online samples will tend to under-represent the news consumption habits of people who are older and less affluent.”
- Small markets help you. If only about 2,000 firms exist in your niche, 200 responses is 10% of the universe, and the finite population correction trims the margin of error from roughly ยฑ6.9 to ยฑ6.6 points.
Q3. How do I recruit 200 real humans and keep the bots out?
- Expect a low response rate and say so. Pew’s telephone response rate fell from 36% in 1997 to 9% in 2016 and 6% in 2018 (NBC News on Pew data). A 12% completion rate from a clean list is respectable.
- Open links attract professionals of the wrong kind. In a Pew opt-in experiment, 12% of adults under 30 claimed a license to operate a class SSGN nuclear submarine; the true share rounds to 0% (Pew Research Center, 2024). That is a lot of submarine captains filling out marketing surveys.
- Attention checks catch less than you think. Across 62,639 respondents, 84% of bogus cases passed Pew’s attention check and 76% passed both the attention and speeding checks (Pew Research Center). One autism researcher got nearly 2,000 responses overnight and kept three (The Transmitter).
- Pay fairly, then verify. User Interviews analyzed nearly 20,000 projects and found $100 an hour to be the B2B baseline with the lowest no-show rates; B2C unmoderated studies at $60 an hour saw nearly 30% no-shows versus 8.2% at $160. B2B panel firm Wynter says its respondents are checked against a corporate email and LinkedIn. Use unique invite links, work emails, and one open-ended question you read by hand.
Table 2. Where Fake Respondents Come From
| Sample source (Pew study) | Bogus respondents | What it means for you |
|---|---|---|
| Crowdsourced (gig platforms) | 7% | About 14 junk replies in every 200 |
| Opt-in panels (avg. of 3) | 5% | About 10 in 200; budget for cleaning |
| Address-recruited panels (avg. of 2) | 1% | Closest analog: a verified list you invite |
Q4. How do I word questions so they measure opinion instead of manufacturing it?
- Single words swing results by 20+ points. In Donald Rugg’s classic split-ballot test, 54% would “forbid” speeches against democracy, yet 75% would “not allow” them (CBS News). In the 2010 General Social Survey, about one quarter said we spend too little on “welfare” while about two thirds said too little on “assistance to the poor” (Shanker Institute).
- Agree/disagree formats invite yes-saying. An opt-in poll found 20% of adults under 30 agreeing that “the Holocaust is a myth.” Pew’s probability panel of 5,140 adults found 3%, the same as every other age group. Pew blamed acquiescence bias.
- Practical rules: ask about behavior (“in the last 90 days, did you…”) before attitudes; offer “don’t know”; randomize answer order; and pilot with five people who will be blunt. Keep it short: in SurveyMonkey’s analysis of 100,000 surveys, drop-off rose fastest with each added question up to 15.
Table 3. Same Topic, Different Words
| Wording A vs. Wording B | Result A vs. B | Gap |
|---|---|---|
| “Forbid” vs. “not allow” speeches | 54% vs. 75% | 21 points |
| “Welfare” vs. “assistance to the poor” | ~25% vs. ~67% “too little” | ~40 points |
| “Holocaust is a myth” (opt-in vs. Pew) | 20% vs. 3% of under-30s | 17 points |
Q5. What can 200 responses actually tell me?
- The math is short: margin of error = 1.96 ร โ(p(1 โ p)/n). Statistics By Jim notes that “the largest margin of error for a given sample size occurs when the percentage is 50%,” which is why the table below uses 50%.
- Subgroups are where small surveys get into trouble. Split 200 into 70 agencies and 130 in-house teams and the margins grow to ยฑ11.7 and ยฑ8.6. A 10-point gap between them is a coin flip with a press release.
- Real error runs bigger than reported error. G. Elliott Morris of Strength In Numbers gives a blunt rule: “Take the margin of error and double it.” The margin “only accounts for random sampling error” (Statistics By Jim).
Table 4. 95% Margin of Error (in points) by Sample Size
| n = 50 | n = 100 | n = 200 | n = 300 | n = 500 | n = 1,000 |
|---|---|---|---|---|---|
| ยฑ13.9 | ยฑ9.8 | ยฑ6.9 | ยฑ5.7 | ยฑ4.4 | ยฑ3.1 |
Notice the curve. Going from 100 to 200 cuts error by about three points. Going from 500 to 1,000 buys only 1.3 points at double the cost. For most industry studies, 200 to 300 clean responses is the sweet spot.
Q6. How do I report a modest sample without overselling it?
- Show counts beside percentages: “31 of 200 (16%).” Skip decimals; “15.5%” from 200 people implies precision you do not have.
- Draw the uncertainty. Datawrapper’s blog warns that “people judge values inside the bar to be more likely than values outside the bar,” so use dots with range lines or lighter bars.
- Name the crowd precisely. Write “200 operations leaders at U.S. agencies, recruited from state license lists” instead of “the industry.”
- Flag correlation. SaaS Capital’s 2026 survey of 1,000+ private SaaS firms reports median selling spend at 15% of ARR and still notes the funding/growth link “may not be a causal relationship.”
Q7. Which disclosures make a small study beat a big vague one?
The AAPOR Transparency Initiative lists 11 items to disclose at release. Transparency even earns points: Silver Bulletin’s pollster ratings, built on 12,300+ polls, give firms a bonus for AAPOR Transparency Initiative membership or sharing data with the Roper Center. Here is how they translate to a 200-person survey.
Table 5. The Disclosure Checklist, Translated
| Disclose | What it looks like in your report |
|---|---|
| Sponsor and funder | “Funded and run by Acme Software, which sells agency tools.” |
| Population and frame | “U.S. agencies with 2 to 50 staff, drawn from three state license lists (about 4,100 firms).” |
| Recruitment and incentive | “1,600 emailed invites; $15 gift card; probability sample of the list.” |
| Dates and mode | “Online, March 3 to 24; English only; median completion 7 minutes.” |
| Exact wording | Full questionnaire in an appendix, with answer order and skip logic. |
| Sample size and precision | “n = 204; ยฑ6.9 points at 95% for the full sample.” |
| Weighting and cleaning | “Weighted by agency size; 19 removed for duplicate or nonsense answers.” |
| Limitations | “Self-reported; excludes captive agencies; subgroup gaps under 12 points are not reliable.” |
- Post the anonymized data. Data Colada caught fabricated data in a famous 13,488-customer insurance study because the file was public: miles driven were spread evenly from 0 to 50,000 with no rounding, which real odometers do not do. Openness earns trust and catches your own mistakes.
Q8. Is original research worth the trouble for demand and distribution?
- The numbers say yes. Orbit Media’s annual blogger survey of 1,042 marketers finds only 13.9% report “strong results,” an all-time low, yet publishing original research raises the odds of strong results by about 50%. And 92.4% now use AI for blogging, so original data is one of the few things a chatbot cannot fake for your competitors.
- Repetition builds the brand. Edison Research’s Infinite Dial, a probability phone survey of Americans 12+, produces headlines like “Over 100 Million Americans Listen To A Podcast Each Week” because it shows up every year with the same method.
- Built for AI answers. Answer engines lift short, sourced, numbered facts. Give each stat its n, date, and a link to the methods page.
Q9. What does the whole thing look like, start to finish?
- Write the frame sentence and the three findings you hope to test (Week 1).
- Draft 12 to 15 questions, behavior first; pilot with five blunt people (Week 2).
- Send about 1,600 unique invites; at a 12% to 13% completion rate you land near 200 (Weeks 3 to 5).
- Clean: drop duplicates, straight-liners, and nonsense open-ends; log every removal (Week 6).
- Publish the report, the questionnaire, the disclosure box, and the data; release one stat per week (Week 7 onward).
Two hundred honest answers, plainly labeled, will out-argue a thousand anonymous clicks. Readers who can check your work tend to quote it.
Sources
* From the provided list of independent sources (13 of 22).
Discover more from 1000 Startups
Subscribe to get the latest posts sent to your email.
Claude Penland builds the marketing and business strategy for companies that are good at what they do and hard to find. Thirty years operating, one exit, eight of them as a practicing casualty actuary.
The free two-page read is genuinely free. Email claude@1000startups.com and I'll send back what I can see from the outside. Or see the work samples and how to work with me.