Reducing risk through experimentation.
How we saved our client from investing in the wrong product idea.
Our recent clients have been new solo entrepreneurs or small, early-stage startups with super-tight budgets. When they reach us seeking design help, they say things like “I want to get the user experience right” or “I want the product to have the right look & feel”.
Regardless their motivation, they are all eager to build and launch their product idea to market asap so they can fastly reach users and/or investors.
We then ask them “Why do you want to build this?”
Their answer helps us understand what problem they’re trying to tackle and how they think their product will solve for it in better ways for people. Furthermore, it let us evaluate quickly if their line of thinking is the result of deep investigation in the problem space or merely fuelled by a gut-feeling approach.
The problem with the latter is that creates false positives. Clients feel entitled to move forward because they’ve gathered some market research that justifies/confirms their beliefs and biases (conscious or unconscious). They are confident they know their users, because they see themselves as part of the audience they’re trying to address.
So they end up with a product strategy and roadmap informed by unconfirmed hypothesis and assumptions about people’s behaviours and motivations. This is a risky and expensive move that (pretty much always) ends in failure and with little margin to reconduct.
That’s why we discourage these entrepreneurs to “move forward” and advise them instead to invest a fraction of their modest budget to “hold back” (aka validate your product idea and identify its requirements based on evidence).
If they’re onboard, we help them evaluating where they are and what assumptions need testing to validate their direction. The outcome of the research is a win-win situation: If their hypothesis are confirmed, they’re proving upfront that they can deliver specific value to a specific audience.
This helps to create a product that people want, as well as a sustainable business model, which will attract investors and partners even before they have a product to market. On the other hand, if they’re proven to be wrong, they end up saving invaluable money and time for something else.
This is what we advised Jean (baker, entrepreneur and recent client), when he approached us looking for a designer for his product idea: a mobile app to connect local businesses in his town (a Spanish holiday destination), so they could promote their businesses among the tourists, share and gain new customers with other local businesses and hopefully increase all of their sales. For tourists, the app would be the place to discover good local businesses and benefit from special perks.
Before moving forward, we agreed with him to plan, design and conduct a contained experiment during Easter (tourists high-season) to test his assumptions. This would be a perfect scenario for us to validate Jean’s idea, as it brings 10,000s of tourists to the small seaside town.
We would fake the app experience for both customers and businesses by leveraging and connecting existing online services, removing the need to start designing the interface or opening a code editor.
We wanted Jean to be involved, more than just being a business owner in the experiment. If we took him on the journey of exploring, testing and validating his idea, the learnings would be greater than a just R&D presentation at the end with a fancy slidedeck.
Planning the experiment.
We started by defining the goals:
• Whether customers of one business would be motivated to go to other business and become their customers too
• Whether local businesses saw an increase of sales during experiment
• Whether local businesses participating in the experiment received new online reviews / ratings
• Whether local businesses were committed to consistently track redeemed perks
From here, we started brainstorming ideas of how we could quickly build an app-like experience, to test these goals.
Building the experiment
For the customer-facing ‘app’, we utilised Carrd.io, as it allowed us to build simple, responsive websites, really fast. We made a template site which listed the local businesses’ info, customer reward, location, customer ratings and links to leave reviews on Google and Tripadvisor. We then duplicated and localised the template for Spanish and French.
For the business-facing ‘app’, we utilised Coda.io as it allowed us to quickly build out the functionality we needed, works across devices and allows multiple users to interact with the same ‘Doc’, at the same time.
Our Doc allowed the businesses to quickly track the customers promo codes when they redeemed a reward (🇪🇸Canjear), and to activate and onboard new customers to the experiment (🇪🇸Activar); and for us to track the experiment (🇪🇸Análisis).
To activate a customer in the experiment, all the business would do is select the language of the customer and show them their unique QR code. The customer then scans it with their phone’s camera, taking them to their unique webpage.
To allow for further insights, we tracked which business was redeeming a reward, to which customer, and who added a customer to the experiment. Thanks to some complex Formulas, we could automate the tracking and summarising of the experiment data into a dashboard view to track important metrics, at both experiment and business level.
To make it feel more app-like the Doc was dynamic, based on who was logged in. So a business didn’t need to select their business, when adding a customer or redeeming a reward. This also helped the data integrity and prevent user errors.
As the common language of all the local business owners and employees is Spanish, once we had built the Doc, we then translated to Español.
We empowered Jean to take responsibility of recruiting the businesses for the experiment. This would give him the opportunity to speak to other business owners about his idea, gaining valuable feedback and insights.
We were ready. All that was left was to onboard the businesses and start the experiment.
While this might sound like a lot of work, it wasn’t. We (Matt & Judit) planned, designed and built this in 1 week. This included having to learn how to use Coda on-the-fly... Big thanks to Coda Support team and community... especially Mallika, who helped us understand how to construct, connect and automate our Doc and its data.
Learning from the experiment.
Unfortunately, the experiment didn’t start during Easter.
Despite much effort, Jean failed to recruit any businesses in time. He believed the other local businesses would be interested and would actively want to participate. However, he found they didn’t have enough time or interest, often skeptical about the gains of participating, and concerned about the amount of hassle it would involve to be part of it.
We decided to postpone the experiment and take a step back to reflect and learn.
The fact that this experiment failed to even start, is not a negative. Actually, this is a valid and valuable outcome. It has provided us with some interesting learnings, particularly around the needs and experience requirements for business users in this idea.
It also gave Jean time to speak to many of his target users, including business owners, employees and customers. This allowed him to refine his vision and understand the problems his idea would solve for.
We think this is really powerful. We could have put in a proposal to design and build his idea. We could have wasted 6 months, only to learn that we made something no-one needed. However, within a week, we have possibly saved Jean all of his budget.
What happens next?
Well that’s up to Jean, with what we have learnt, we could modify or create a new experiment to try validate a different approach to this idea. Or he can choose to explore and validate a different one.
While we were building our Coda Doc for this experiment, Product Hunt and Coda were running Makers Festival 2019: No-code edition. Although we hadn’t finished, we submitted this project. Since which, Coda reached out to us and ask to turn it into a template for their library.
You can demo, make a copy of the template to see how it’s was constructed or copy elements to your own projects: https://coda.io/t/LOCAL_tlOWitgDsE9.