Survivorship Bias: Why Reading Too Many Stories of Success is Bad For You

What is Survivorship bias?

• A reasoning error that happens when you focus only on stories of success, leading to a flawed understanding of what caused that success
• When you draw conclusions from a data set that is incomplete and biased — without being aware that it’s incomplete and biased
• A cognitive bias that ignores the role of luck and randomness in producing outcomes

Examples in Business and Marketing

Drop out of college, become a billionaire

Bill Gates and Mark Zuckerberg dropped out of college and went on to become industry legends. Therefore, if you’re in college and working on a startup idea, you should drop out and pursue your idea full-time.

How is this Survivorship Bias?

• For every 1 Zuckerberg who drops out of college, there are tens of millions of college dropouts you’ll never hear about. Zuckerberg and Gates are extreme outliers. Quite literally, they are once-in-a-generation people.
• Ignores starting conditions and context. For Gates and Zuck, dropping out of college was a low-risk move, since they came from wealthy families. Most people do not have this luxury.

“How One Weird Marketing Tactic Led To 300% More Revenue”

You read a case study that presents incredible results based on A/B testing. Therefore, you conclude that you must also be A/B testing.

How is this Survivorship Bias?

• Again, no accounting for failure cases. Not every A/B test is successful, or will produce meaningful results. Companies that run failed A/B tests typically do not write case studies about this failure. (aka the “file drawer problem“)
• Case studies are biased because they only show outliers — the most remarkable stories of success. When you don’t hear about negative outcomes, you are looking at a biased sample.

The Pet Rock

An inventor comes on Shark Tank with a weird idea. Mark Cuban tells him that it’s a terrible idea and it will have no market. The inventor replies, “Well, the Pet Rock was successful and everyone said that was terrible, therefore I should ignore you and dedicate my life to make this successful.”

How is this Survivorship Bias?

• Ignores the role of luck, chance and randomness. The inventor has overweighted the probability that The Pet Rock was intentional and purposeful. Sometimes a confluence of factors comes together such that something unexpected occurs; we have to take this into account whenever we try to learn from the past.
• Extrapolating from an unrelated sample. The Pet Rock has little in common with anything, other than it was a fluke product. If you’re going to use a fluke as a reference for why you will be successful, you have to draw extremely similar parallels to your situation.

Why Survivorship Bias happens

WE LOVE TO focus on outliers.

Our brains are trained to spot novelty. Extremes are interesting. The middle of the bell curve is not interesting, since that’s where most of us are, and what we observe on a daily basis.

Society rewards stories of success.

Most articles and content highlighting stories of success are written by journalists or marketers who are incentivized to get clicks and conversions. Failure cases of someone following a “winning” playbook are ignored. Nobody wants to read that.

Randomness is scary.

Humans are narrative-driven. It’s hard for us to accept that luck is a real factor. We need to feel in control all the time. We want someone to just tell us what to do — it’s scary to realize “oh shit, I have to figure it out myself” and it’s comforting and reassuring to think there’s a “golden sequence” that we just have to follow.

How to deal with Survivorship Bias

Here are a few mental bugfixes you can use to avoid falling into the Survivorship Bias trap:

Take the starting conditions, then project forward.

If you re-played the scenario, what are your chances achieving of the same outcome? How many people took the same advice, took the prescribed action, but failed anyway? Which outcome is likeliest?

Ask Why, not How.

Your job as a marketer is to be good at identifying causal factors. The wrong way to do this is to obsess about HOW things happened: the video that went viral, the Facebook Ads campaign that drove results. Not many people obsess about the WHY, which is more important for driving future causality.

Question the author’s incentives.

All case studies, all company features, all celebrity biographies are biased and edited to show them in the best light. They have all been written with an agenda, either (a) to lionize the person/company or (b) sell something to you. There may be useful gems found in the work, but it’s rarely the tempting narrative they dangle in front of you.

Cookbooks, not playbooks.

No two people have the exact same kitchen. Therefore, if you are trying to achieve similar results, you have to study the directions being given to you, see whether you have enough of the same ingredients, then adapt it thoughtfully to your situation, accepting that the outcome is not going to be the same.

Once lucky, twice good, thrice skilled

You can overthink Survivorship Bias.

Just remember that there ARE differences between consistent winners and consistent losers. The key is to study the track record.

Bill Belichick, who has kept the New England Patriots perennial Superbowl competitors for 2 decades, is clearly doing something right.

Your job is to identify the things that are responsible for producing the outcomes. Don’t blindly follow something just because successful person X did it. Does the narrative make sense?

If Bill says “I win because I kiss a Rabbit’s foot every night”, you would be silly to take that as the reason for his success.

But if Bill says “I win because I maintain an unnaturally high standard of performance, even during practice”, you’ll want to dig further.