How to Get Lucky - The Elements of Serendipity

Luck favors the prepared mind
-Louis Pasteur

In the summer of 2019 I entered the annual 2.5 day company hackathon with a project that had been brewing in the back of my mind for months.

Hackathons are very short, very intense, single or multi-day projects where people from various backgrounds collaborate to create new products or try out new things. They're used in tech companies as a source of new ideas for innovation. They also happen to be excellent sources of serendipity.

That is exactly what happened to me after I presented my project in front of the three judges in the first round. Unexpectedly I made it to the second round, presented in front of a much bigger group of colleagues and judges and found out that the company was already working on a similar project.

I kept following these subtle pings (more later) into even more interesting meetings with people I would have never met otherwise, and which culminated with me collaborating with a group of MIT MBAn students as a consultant on their project. The whole thing was surreal as I never expected or planned for any of it to happen. Serendipity however is not that uncommon.

So the natural question I had was "how do I cultivate more serendipity in my life?" I dove into the research and kept reading articles and books. I took copious notes, fervently looking for patterns and trying to distill the ideas into a structure. Finally the elements of serendipity made themselves known.

So what is serendipity?

The word origins come from a fairy tale called "The Three Princes of Serendip.” These princes “were always making discoveries by accident or sagacity, of things which they were not in quest of.” It was coined by English writer Horace Walpole  in the 1700 in a letter to his friend Horace Mann.

That definition doesn't do justice to the power that serendipity has in laying claim to discoveries in science and technology. Most of the things you see and use every day were discovered while the scientists were looking for something else. The examples are vast: penicillin, Viagra, rubber, microwave ovens, laser, vaccines, etc.

• When Alexander Fleming returned from a two-week vacation he found that a mold had developed on an accidentally contaminated staphylococcus culture plate. Upon examination of the mold, he noticed that the culture prevented the growth of staphylococci.
• Viagra was developed as a heart medication to treat hypertension, but while that failed, during the trials they noticed that none of the male patients returned their samples.
• Percy Spencer discovered that microwaves meant for radar melted a chocolate bar in his pocket. After many attempts he managed to build the microwave oven we all use and love today.
• Charles Goodyear discovered the vulcanization process for rubber when Sulfur accidentally spilled onto a sample of boiling rubber on his stove and hardened. His quest to discover this process nearly bankrupted him, yet the discovery was accidental.

Elements of serendipity

When I looked at the research on serendipitous scientific discoveries, I noticed three distinct elements that were always present:

1. Curious Tinkering
2. Casual Collisions
3. Subtle pings

Curious Tinkering
It all starts with curious tinkering, which is the pursuit of ideas you find intrinsically very interesting. Nassim Taleb calls this “stochastic tinkering,” where stochastic means random, but I don’t believe it’s random. I think there's a lot more to following your curiosity and seeing where interesting ideas take you. I plan to write more about this in the future.

Science would advance a lot faster if scientists were allowed more time to tinker with things and ideas they find interesting. Instead they're forced to submit plans for what they intend to study and describe what they are trying to find as if central planning ever worked for discovery.

Nassim Taleb, in his book The Black Swan talks about a company that actually did this:

"This was my first encounter with a firm that lived off Black Swans of the positive kind. I was told that a scientist managed the company and that he had the instinct, as a scientist, to just let scientists look wherever their instinct took them. Commercialization came later"

The same thing happens in entrepreneurship. Good luck getting the bank to lend you money if your business idea is to tinker in your garage until you build a successful product. Yet we know from history most successful companies were founded by tinkering in the garage or in some dark basement (see Apple). Why doesn't curious tinkering get more respect? I suspect it's due to it's presumed random nature, which is not at all random.

But following your curiosity is actually not random. Experiments in artificial intelligence have shown that when an algorithm has been trained to follow its curiosity and maximize interestingness and novelty, it solves complicated mazes much faster than when given explicit objectives. This algorithm is called novelty search

Casual Collisions
Share your ideas with others even if they're not fully fleshed out. Tell people you meet randomly what you're working on. Companies try to design shared spaces, like kitchens, so people can randomly run into each other, talk about what they're working on and see if any new ideas bloom.

Richard Hamming, the Alan Turing award winning mathematician, called it the "open door" phenomenon:

working with one's door closed lets you get more work done per year than if you had an open door, but I have observed repeatedly later those with the closed doors, while working just as hard as others, seem to work on slightly the wrong problems, while those who have let their door stay open get less work done but tend to work on the right problems! I cannot prove the cause and effect relationship, I only observed the correlation
Richard Hamming - You and Your Research

Having an open door policy leads to casual collisions which can change the course of your project in many beneficial ways. In my case my casual collision was the presentation I did in front of the judges and my peers. The number of comments I got from that led me in a number of seemingly random but interesting paths which culminated with my involvement with the MIT students' project.

Subtle pings
Subtle pings could be anything from an off-hand comment about someone doing a similar project as you, an invitation to an event or party that doesn't seem particularly interesting on the surface, a random like of your tweet or a seemingly random connection on LinkedIn. I've met some of my best friends through this. You have to recognize and follow up on many subtle pings and collect opportunities. Not all of them will pay off, but the ones who do could make up for it.

"The great scientists, when an opportunity opens up, get after it and they pursue it. They drop all other things. They get rid of other things and they get after an idea because they had already thought the thing through. Their minds are prepared; they see the opportunity and they go after it."
Richard Hamming - You and Your Research

When you think of seemingly random occurrences as subtle pings it helps you to recognize their potential as an opportunity to pursue further, but noticing them in the first place requires, as Hamming puts it, an open mind. In fact, psychological research on luck puts a premium on openness.

Richard Wiseman in his book The Luck Factor found that people who consider themselves "lucky" tend to score high on extroversion (an orientation of one’s interests and energies toward the outer world of people and things rather than the inner world of subjective experience) and openness (to new experiences) and low on neuroticism (the tendency to experience negative emotions like anxiety, anger, guilt and depression).

Derren Brown, the famed British mentalist, explored these ideas in a highly recommended documentary about luck where he attempted to deliberately make people of the town of Todmorden more lucky.

So noticing subtle pings takes a prepared mind, taking us back to that opening Pasteur quote. But how can we have a prepared mind?

The prepared mind
The prepared mind is the overarching pattern that runs across all the elements and illuminates every single one of them. It has several characteristics:

The dominant characteristic of the prepared mind is curiosity. When something unexpected happens, curious people say "hmmm, that's funny" and dig further.

The second characteristic is a certain cognitive flexibility, for example you would perceive failure as feedback or as information and not discard it. You also have multiple explanations for why an event has occurred instead of jumping to conclusions quickly.

The third characteristic of a prepared mind is a broadly read mind. Someone who has read more broadly is more likely to notice connections among seemingly disconnected ideas that narrow focused readers easily miss. And finally, set an overarching goal but be flexible on how you get there.

Serendipity is everywhere. If you're not tinkering with interesting ideas, talking about them with anyone who will listen and following up on subtle pings, you're missing out.