Experimenting Your Way to Bolder Innovation

Colin Palombo, Managing Partner at Innovation Framework Technologies, explains how experimentation and innovation should work hand in hand to produce groundbreaking results.

Amazon is one of the most innovative companies on the planet – launching successful, bold new products and services on a regular basis. From dominating online retail, Amazon has extended into areas as diverse as AWS cloud computing services, drone package delivery, Kindle e-book readers, Prime video streaming and Echo home artificial intelligence.

Innovation Experimentation - jeff bezos
Jeff Holden – “Build a team that has an experimental ethos”

Jeff Bezos, Founder and CEO of Amazon, states: “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”

Peter Diamandis, visionary founder of organizations like XPRIZE foundation, Human Longevity Inc and Planetary Resources Inc, agrees: “Experimentation is a crucial mechanism for driving breakthroughs in any organization. If you want to create a successful, hyper-growth company, you’ve got to focus on empowering your teams to rapidly experiment.”

Most organizations don’t test their innovation ideas with quick experiments to generate real-world evidence. Instead they rely on in-house opinions, or conversations with lead customers to vet ideas. When data-driven experiments are run (like market surveys), they often take too long, cost too much or deliver little new insight to develop the idea.

For example, at Amazon in the early days, many of the experiments conducted were found to be useless. Jeff Holden, Amazon’s former Chief Product Officer (and now Uber’s CPO) explains: “The experiments had no chance of yielding any value. There wasn’t any point to them. We were just kind of curious. We were just running a lot of experiments, which have a cost by the way, and were taking up slots so others couldn’t experiment.”

So how can companies use experiments to accelerate and de-risk great ideas, while killing or pivoting away from the bad ones? Here are four steps to include experimentation in your front end of innovation process: 1. Encourage bold/crazy ideas 2. Select the best idea for experimentation 3. Use a rapid ‘sprint’ experiment process 4. Interpret results, accelerate action.

Innovation Experimentation
X marks the spot – Google’s moonshot factory

Encourage Bold/Crazy Ideas 

Bold experimentation and innovation does not start with ‘ho-hum’ ideas. Running experiments on ideas for minor tweaks to your current offerings will be costly and demoralizing. Instead, start with a bold ‘ideation challenge’ – a statement that clearly identifies a big problem for your current customers, or an opportunity to serve new markets with unique, highly differentiated solutions.

The challenge also specifies constraints to the problem and invites a diverse group of people to submit unconventional, unusual and potentially unfeasible ideas for solutions. The goal here is to explore the boundaries of the ‘proximate future’, from as many different perspectives as possible. Bold ideas could be based on using new technologies, business models, processes, services, partnerships or some other approach to create new value for customers.

To get bolder ideas, you also must give license for people to think different, as Steve Jobs put it. Astro Teller, CEO of Google’s moonshot factory, does this by running a ‘Get Weirder Award’ – challenging people to ask ‘weird’ questions, put forth crazy ideas around framing problems differently and to design experiments that really push the limits. The award is given out every two weeks to the people who ran the best experiment in that time.

Select the Best Idea for Experimentation and Innovation.

Once you have a hopper of bold/crazy ideas for your Challenge, you next must select one or a cluster of them to perform the experiment on. Experiments require time commitment from employees, so you need to be selective in picking experiments that test critical questions for potential bold new solutions.

Astro Teller uses the following three principles to help identify good experiments:

  • Principle 1: Any experiment where you already know the outcome, is a BAD experiment.
  • Principle 2: Any experiment when the outcome will not change what you are doing, is also a BAD experiment.
  • Principle 3: Everything else, especially where the input and output are quantifiable, is a GOOD experiment.

Innovation Experimentation - experimentJeff Holden adds: “Build a team inside your organization that has an experimental ethos, and make sure that the experiment, value proposition, and hypothesis are really thought through before you invest the time and energy to actually do them.”

Jeff continues: “At Uber, if you can’t articulate your hypothesis crisply, or your hypothesis doesn’t matter to the company, then you must not do that experiment. Oftentimes you’ll send folks back to the drawing board or ask them to recast the experiment. Our company learned, and we got much better.”

To select good experiments, set up a ‘screening gate’ in your ideation challenge, where an evaluation team uses a scorecard to evaluate and prioritize experiment candidates. The evaluation team selects the best one, and authorizes an experiment team to dedicate their time to completing it.

To use a “Rapid Sprint Experiment Process” to run effective experiments, you need a reliable method to get valuable information in the minimum time and cost possible. A great example of this is the one-week ‘sprint experiment’, invented by Jake Knapp and John Zeratsky of Google Ventures, described in their book ‘Sprint’ and on this page.

Here’s a quick summary of the process: 

First, assemble a sprint team of four to seven people who can dedicate an uninterrupted week to the experiment. This is quite a time commitment, so the people assigned must be willing, able and co-located to participate. The sprint team consists of relevant cross-functional experts – from marketing, sales, technology, product design, supply chain, IT, finance and so on. A facilitator ‘sprint coach’ is added to coordinate the week.

Innovation Experimentation - steps

  • Monday – Understand & Map. The sprint team spends the first day developing a common understanding of long-term goals for jobs to be done and outcomes for the customer. They also identify assumptions, uncertainties and constraints to achieving those outcomes. Team members consolidate this knowledge onto a white-boarded value-chain map, graphically showing activities and related issues to deliver desired customer outcomes. The sprint team augments this model by reaching out to other company experts to gather their feedback.
  • Tuesday – Explore & Sketch. The sprint team explores potential solutions to the problems identified. Each team member generates their own solution insights, capturing them in short notes and sketches. The goal is to identify as diverse an insight set as possible – probing the boundaries of feasibility. All insights are then shared anonymously, and each team member creates potential solutions by combining insights in different ways to achieve the desired outcome. The team also starts to recruit five existing or potential customers who represent the targeted customer segment, to participate in Friday’s test.
  • Wednesday – Decide & Focus. By now you have a set of proposed solutions, so the best one needs to be selected to move to the prototyping step. The team critiques each solution, eliminates currently unfeasible ones, and then uses crowd voting tools such as token voting to select the best one to be prototyped and tested. In the afternoon, team members develop a prototype storyboard – a workflow of 10 to 15 actions that the prototype will support, covering the most critical, most uncertain part of the selected solution.
  • Thursday – Prototype. The sprint team builds a disposable, very low cost prototype to test the specific unknowns and assumptions in the prototype storyboard. Instead of taking weeks or months to do this, the sprint team has to produce the best possible prototype in a single day. It could be a mock-up website, physical rendering of the product (possibly 3D printed), paper-based brochure, video, computer simulation, play-acted service and so on. The team chooses the approach that will be ready in a day, which provides an illusion that enables target customers to react naturally and honestly to the experience.
  • Friday – Test. On the final day of the sprint, your five target customers are scheduled for individual interviews throughout the day. One sprint team member invites each customer to try to complete the desired job using the prototype, and asks the questions about the experience. The other team members observe and takes notes together via a live video stream. During the day, the team will see behavioral patterns that can be further probed in the following customer interview. By the close of Friday, the sprint team has learned a lot, gathered useful empirical data, and is much clearer on how to adapt the proposed solution or pivot to an alternate approach.

Interpret Results, Accelerate Action

Having completed a week-long sprint experiment, Astro Teller’s 2nd principle applies – the team must be able to interpret the results correctly and take action.

Jeff Holden advises: “You have to be able to interpret the experimental results really well. It’s statistics. Know the difference between statistically significant and insignificant results.”

Taking action means learning from the results and fast-tracking the tested solution through your front end of innovation process. This may shave off weeks or months from the idea gestation period. Next, your tested solution needs to be commercialized. It should flow into your portfolio of active product development projects – either as an addition to an existing project, or as a project in its own right. Ideally one or more of the sprint team join the new project team, but at the very least, walk the project team through the entire experiment and their conclusions.

Sprint experiments are not only useful in the front end of innovation. They can also be used to de-risk projects in their scoping and business case stages. In fact, there is a natural progression from using sprint experiments for ideation to using them in an agile stage and gate process for new product development.

Peter Diamandis concludes: “Sprint experiments offer a path to solve big problems, test new ideas, and accelerate the decision-making process. Today’s most successful companies, the ones that are crushing it, started as a series of crazy ideas, followed by experiments to test just how viable those ideas might be. I have leveraged the sprint process across all of my companies.”

These four steps, including the one-week sprint experiment method, can all be supported using Qmarkets software. To discover how Qmarkets can help you generate more ROI from experimentation and innovation, get in contact today!


About Colin Palombo

Innovation Experimentation - colin palomboColin Palombo is the Americas Managing Partner of Innovation Framework Technologies, based out of Fort Lauderdale. He is also a visiting Professor of Innovation at Strathclyde Business School, the UK’s ‘2016 Business School of the Year’ and a member of Abundance 360 – a select group of entrepreneurs collaborating with Peter Diamandis. This coming April, IFT and Qmarkets will be hosting an enterprise innovation leader breakfast, which will explore a variety of topics around the themes of innovation challenges and collective intelligence. Find out more about the Paris Innovation Leader Breakfast Tour here.

You Might Also Like...

Corporate Innovation
Article Best Practices Innovation Management
Discover how to excel in corporate innovation with effective strategy alignment, clear processes, and the right innovation management...
by Charlie Lloyd
6 min
technology 4256272 1280
Article Idea Management Innovation Management Trend Management
Discover the top innovation models for corporate success. Learn how innovation management software can help you implement each model...
by Charlie Lloyd
6 min
An image representing an innovation ecosystem
Article Innovation Management
Discover the importance of leveraging an innovation ecosystem for enterprises. Learn how integrated innovation activities can drive...
by Charlie Lloyd
5 min