In Part 1 of this series, I discussed the importance of asking good questions at the beginning of the innovation process. Once good questions have been asked and the problem framed, serious work needs to be done to answer those questions. Although this post focuses on why experimentation and prototyping are important for the innovation process, they are only some of the tools available to answer questions. I agree with Tim Kastelle, who asserts, “I am always suspicious of one-size-fits-all solutions. They are very easy to sell in a book or a blog post, but they rarely work in the real world. There’s too much variation.” [“There Must Be Forty Ways to Innovate,” Innovation for Growth, 5 November 2012] Too prove his point, Kastelle offers a list containing forty ways to innovate:
Idea Generation
- get to the edge
- scratch your own itch
- be a genius
- blue sky R&D
- applied R&D
- ask your customers
- watch your customers
- ask your people
- brainstorm
- gamestorm
- think outside the box
- think inside the box
- co-create
- scenario planning
Selection and Implementation
- experiment!!
- R&D
- stage/gate
- innovation team
- innovation coach
- expert panel
- minimum viable product
- iteration
- gut instinct
- does it fit with what we’ve always done?
- do whatever the CEO wants
- focus groups
- market testing
- A/B testing
- team consensus
Spreading Ideas
- network
- traditional distribution
- viral marketing
- advertising
- influentials
- small seeds
- word of mouth
- lead users
- co-creation
- pull strategies
- partnerships
Notice that only one of his forty ways has exclamation points associated with it — experimentation. Kastelle isn’t the only innovation guru who is keen on experimentation. Jim Manzi, chairman of Applied Predictive Technologies, is another proponent of experimentation. He wrote his thoughts in a book entitled Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. In a review of the book, Trevor Butterworth writes:
“Imagine that you are the chief executive for a chain of 10,000 convenience stores, 8,000 of them called QwikMart, 2,000 of them called FastMart. Strangely, the FastMarts are bringing in 10% more in sales on average than the QwikMarts, and your instinct is that it may have something to do with customer preference for the FastMart name. How do you find out if your hunch is correct? Jim Manzi … was once asked to address such a question by the owner of a convenience-store chain—a story he relates in [in his book]. Finding the answers was not easy. There were hundreds of variables that could account for the QwikMart-FastMart revenue gap, including distance to nearby highways, number of cash registers, cleanliness of stores and the ‘exact position of each product on each shelf.’ Worse, all these variables could mate with each other to produce even more variables. Nevertheless, after studying mathematics at the Massachusetts Institute of Technology, Mr. Manzi found his dream job as a strategic consultant trying to solve such puzzles for various companies. His approach began with: What makes experiments in science so good at producing reliable knowledge—and could the same principles and methods be applied to business and even social policy? The answer, according to Mr. Manzi, was a qualified yes.”
Butterworth claims that “the hero” of Manzi’s book is “the randomized controlled trial or, as Mr. Manzi prefers, the ‘randomized field trial’ (RFT).” In other words, experimentation. Butterworth writes:
“It should come as no surprise that the most successful companies in information technology—Google, Amazon and eBay—are relentless experimenters. Millions of consumers, for example, can be tested at little cost to find out whether pop-up ads are more effective on the left side or the right side of a computer screen. Google alone, says Mr. Manzi, ‘ran about 12,000 randomized experiments in 2009, with about 10% leading to business changes.'”
Butterworth goes on to write, “It’s one thing, for instance, to conduct experiments, but it’s another to learn from them.” Manzi believes that the best experiments are conducted by someone who is not emotionally attached to the innovation. “A company, Mr. Manzi says, ‘is an alliance of individuals, and there are always competing theories, power centers, and knowledge silos within any firm.’ Amid all the ‘jockeying for control,’ the most successful experiments are performed when the experimenters don’t have a dog in any strategic fight.”
Another proponent of experimentation is marketing technologist Scott Brinker. He wrote that “big testing” is critical in today’s marketplace. [“The big data bubble in marketing — but a bigger future,” Chief Marketing Technologist, 21 January 2013] He writes:
“A few months ago, I wrote a post, “We want to test bold, new ideas that always work.” It highlighted recent research by the Corporate Executive Board showing that the far majority of Fortune 1000 marketers think their organizations are not effective at test-and-learn experiments. One reason why: at least half didn’t think an experiment should ever fail. I believe this is the single biggest obstacle most organizations face: their culture and politics dissuade people from trying experiments because failure of an experiment implies failure of the tester. Who wants to stick their neck out in that environment? So either people don’t test anything, or they test in a non-controlled manner so that outcomes are comfortably subject to caveats and interpretations.”
Brinker asserts that even companies claiming to embrace testing often conduct only superficial tests that involve little risk. “Perhaps the greatest damage they do,” he writes, “is that they give people the illusion of engaging in real experimentation. … Big testing is qualitatively different.” When I think of the term “big testing,” two things comes to mind: high risk and high reward.
Face it, if you are not failing you are not experimenting. Jon Custer notes that one of America’s most famous innovators was also famous for the number of failures he experiences during his experiments. [“The Power of Failure,” CIPE Development Blog, 30 January 2013] He writes:
“Thomas Edison … had to conduct thousands of failed experiments before hitting on a successful design for the electric light bulb, and then had to build the infrastructure to power them. In the 21st century, the corporate descendant of the company that Edison founded in 1880 ensures that today’s entrepreneurs in and around New York City have reliable electricity to run their businesses. Edison famously said, ‘I have not failed. I have just found 10,000 ways that won’t work.'”
Scott Anthony, a managing partner at Innosight, writes, “You don’t have to be Thomas Edison to be an active experimenter. Think about … ‘everyday experiments’ you could run. Change the way in which you commute to work. Alter the order in which you do things in the day. Try eating different portion sizes or different foods at different times of the day. If the professors [Clayton Christensen Jeffrey Dyer, and Hal Gregersen] are right — and I think they are — the process will wire your brain in a way that makes it better at innovation.” [“Innovators: Become Active Experimenters,” HBR Blog Network, 29 March 2010]
Tim Kastelle believes that anytime you feel a bit stuck in the innovation process experimenting is a good idea to get the juices flowing again. [“Experiments – the Key to Innovation,” Innovation for Growth, 28 March 2010] He writes:
“There must be something to this idea, because I’ve run across three different people saying basically the same thing in the past three days. The first was Dan Ariely:
‘They asked me what I thought the best approach was. I told them that I was willing to share my intuition but that intuition is a remarkably bad thing to rely on. Only an experiment gives you the evidence you need. … Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups—a dozen people riffing on something they know little about—to set strategies. And yet, companies won’t experiment to find evidence of the right way forward.’
“Unsurprisingly, he goes on to make a case for the value of experimenting. Part of this reluctance is that experimenting leads to short-term losses – if you try several things to find out what works best, you have wasted resources by trying the ideas that end up not working. Or do you? Rita McGrath doesn’t think so:
‘If your organization can approach uncertain decisions as experiments and adopt the idea of intelligently failing, so much more can be learned (so much more quickly) than if failures or disappointments are covered up. So ask yourself: are we genuinely reaping the benefit of the investments we’ve made in learning under uncertain conditions? Do we have mechanisms in place to benefit from our intelligent failures? And, if not, who might be taking advantage of the knowledge we are depriving ourselves of?’
“She includes a list of conditions that can lead to what she’s calling ‘intelligent failures’, the approach that she outlines is both good and practical. Then I ran across this by Bob Sutton:
‘The final point that Jeff Pfeffer and I make in Hard Facts is about failure. We emphasize that is impossible to run an organization without making a lot of mistakes. Innovation always entails failure. Most new products and companies don’t survive. And if you want creativity without failure, you are living in a fool’s paradise. It is also impossible to learn something new without making mistakes. … Failure will never be eliminated, and so the best we can hope for from human beings and organizations is that they learn from their mistakes, that rather than making the same mistakes over and over again, they make new and different mistakes.’
“To be innovative, we have to try out new ideas. Some of these will fail. If we’re smart, we’ll set up our experiments so that we can learn as much as possible from the ideas that don’t work. We face an environment that is filled with uncertainty. This makes planning dangerous. The best possible way to meet this uncertainty is not with intuition and guesswork, but with experimentation.”
One type of experimentation that doesn’t immediately pop into most people’s minds is gaming. Marla M. Capozzi, John Horn, and Ari Kellen, analysts at McKinsey & Company, report that some companies have improved their innovation process “by integrating war games into their innovation activities. By simulating the thoughts, plans, and actions of competitors, these companies are improving their products and services, while gaining a deeper understanding of how their innovation assets compare with those of rivals—insights that help them better identify, shape, and seize opportunities.” [“Battle-test your innovation strategy,” McKinsey Quarterly, December 2012] They conclude, “War games are a tried-and-true strategic tool, yet relatively few companies use them to innovate. Those that do so effectively can not only avoid the problem of overlooking what the competition might do but also determine how likely their new products and services are to survive in the crucible of the marketplace.”
There are all kinds of experimentation that can be done. Don’t be locked into any one method. The bottom line is that experimentation and innovation go hand in hand. The better you are at the one the better you will be at the other.
