What is a limited release?
A limited release consists of having a small sample of customers using a whole product, not just one feature, over an extended period and outside of a lab environment. It is one of the most expensive ways of testing a product because it has little to no defects and nearly finished.
Limited release is used in numerous industries; as an example, car manufacturers also use limited releases to test their cars on the road. You might have seen cars with a “3D camouflage”, it allows car manufacturers to safely test their new cars without taking the risk of revealing too much to the public.
A limited release doesn’t replace a usability testing or any other form of user research; it is another way minimising risks. Be aware that using a limited release as the only way to get customer feedback is risky because you have developed the product so the cost of fixing design issues will be high. As a rule of thumb, always run regular usability testing through the development lifecycle, so during the limited release, you should have a fair idea of what feedback you will get.
Benefits of doing a limited release for a digital product
Get feedback on the final product
The goal of a limited release is to get feedback on the product you have created. The more data from the participants, the more successful the limited release. The quality of the feedback is an essential factor, and it will depend on the participant engagement over an extended period.
Also of the participant’s feedback, you can also monitor insightful metrics during the limited release, for example; user experience metrics, business metrics, engineering metrics, health metrics and feature specific metrics. Those metrics can help to shape the direction of the product.
Use the participants for your design sprints
Having participants in the limited release makes it an excellent opportunity also to involve them in the development lifecycle. Your participants are not only part of a limited release, but they become a valuable source of information while continuously pushing new feature in the limited release. If you need participants for a usability testing, you can simply invite someone from the limited release as a side activity.
Measure the end-to-end customer experience
To access the product during the limited release, your participants will go through different touch points outside of your product, for example, the initial phone call from your sales department or the interaction with support. Consider asking questions to the participants about these touch points; it will give you valuable data about the overall customer journey also of the product.
Continuously classify the overall customer sentiment
Jeff Patton summarised very well the types of sentiment you can get: “Awesome”, “Meh”, “Awful”. If things are “Awesome”, you are doing well, and you are likely to have a successful launch. If things are “Awful”, not a problem you learn what is wrong with the product and start improving. The real danger zone is the “Meh”; you are in what Jeff calls the “dud zone” and this is very dangerous because it means that you haven’t met the basics expectations of the participants and you haven’t excited the participants.
Continuously measure the overall participant’s sentiment and share it with the internal team on a regular basis.
A contained environment
The first time I used a limited release, it was in the context of B2B for an established market. I recall the product manager stating that even if the limited release fails, we will be fine because it is in a contained environment; therefore, the impact on the brand is already minimised. A limited release is a sandbox for your product team and learning safely in real life condition is what the limited release will also give you.
Challenges of a limited release
A limited release can span over several months, and the team engagement is as critical as the participant engagement. A successful limited release involved stakeholders management across discipline and areas of the business. You will need time with all the stakeholders to get organised and refine the process, this can be challenging, especially in a large company.
Collecting the feedback from the participant will also be challenging, especially in a B2B environment. I have been surprised to see participants paying to be part of a limited release and not engage with the product. Remember that the feedback will not automatically come you so be prepared to call them, email them and visit them to get some feedback. Later in this post, we will explore ways to simplify the feedback collection.
Charge the participants
For a limited release, you want to make it a bit difficult for the participants, yes you have read this correctly. If you are about to collect feedback at such a late stage of the product development, you want your best friend to help you, friends, that are willing to go through small difficulties and still help you. Your best friends are participants that are engaged and committed to telling you what they think even if they have to pay a small price to access the product. I would push as hard as charging the participant to take part in the release. This might seems counter-intuitive, and I have been questioning this too until I saw the quality of feedback coming from participants willing to pay you to see what you have created.
As soon as you put a price tag on the limited release, finding participants will become a bit harder. It is important to let your internal know because it is an unusual process especially for a salesperson having the mission to find the participants.
It is the most expensive way to test a product
A limited release is the most powerful way to check the final product, and it involved spending time and money to develop the product. In a limited release, there are no prototypes, no fake doors; it is a nearly finished product.
On top of spending money developing a nearly finished product, you will have to mobilise a set of stakeholders to get the limited release off the ground. As mentioned before, you can charge the participants to take part in the limited release but don’t expect to make a profit; this is not the goal at this stage you are releasing to learn no to earn!
Establish processes from the very beginning
Setup an internal team
Your internal team is as valuable as your participants because they will make it happen. Start by engaging with everyone and explain what you want to get out of the limited release. Also, take the time to explain that you will charge participants not to make a profit but to get quality feedback.
Finally, make it clear that your team we will probably make mistakes and reassure them that it is the goal, learning quickly to improve before the official launch.
Create a process map
Running a limited release is a collaborative process involving numerous stakeholders in the background, a typical setup looks like this; the sales team, the support team, the training team (if you provide training), the legal team, the finance team, the engineering team and the product team.
It is essential to communicate how your team will impact each other in the customer journey so you will need a process map to clarify everyone role. On the process map, create a set of horizontal swim-lanes, one lane per team showing the different touch points and action that each stakeholder is required to do. To make the process map more clear, I also recommend adding the systems that each team will need to use during the limited release.
There is a good chance that the teams will need to refine their processes along the way, especially for limited release spanning over several months. Organising regular meetings will help to keep the team coordinated during the limited release, it will also enable everyone to learn from each other. Not having internal catch ups during a limited release will ultimately lead to a lack of interest so it is essential to keep the communication flowing and address any issues that the team might have along the way.
Your team members are probably the first one that will hear about the participants challenges before it even reaches the product team, pays attention to what the team is saying they are essential to your success.
Continuously recruit participants
Participant fatigue is inevitable, although you are solving a real-life problem your participants might not continually give you feedback. There is also the risk of building the product with the same group of participants. These two factors combined and you have the perfect recipe for low feedback.
As a general rule of thumb try to engage with a group of 5 participants for a limited time and then use another group of 5. You will notice that you will be rotating between groups of participants. Adding new participants will help you having quality feedback and make up for participants that lose interest over time.
Protect the business with an NDA
The worst thing that can happen to you is having participants in the limited release leaking information to the public without your consent. To avoid this situation, you will need to ask your participants to sign a non-disclosure agreement to protect the product and the company throughout the limited release. Signing an NDA is an extra step that can potentially deter some participants, but remember, if your participants stick with you despite all these little hurdles, you have someone committed to helping you, and this is what you need.
Your commitment to the participants
Set the participants expectations
The participants will need to know the kind of feedback you want, sending them a few instructions before they start using the product will increase your chances to get what you need. The instructions can state that you are after everything that makes the participants happy or frustrated, screenshots of the products and any additional comments they might have. Also, explain that any feedback the participants will write will help to improve the product.
Being flexible on the feedback channel
Collecting feedback can be challenging, the key is to remain flexible. One of the most efficient ways to collect feedback is to use Intercom. Megan Dell has explained very well how to repurpose Intercom to collect user experience feedback.
If you can’t get a third party application in your product, you can try Slack to collect feedback. Finally, following up via phone is very efficient but time-consuming. Regardless of the channel you end up using, take the time to centralise any feedback you receive.
Inform the participants about new features
As time goes, you will probably add or updates features, so it is essential that you inform your participants about these changes. A monthly release note will help to keep your participants engaged.
Track and socialise the participant’s feedback
Create a conversation guide
A digital product can be complex; sometimes the product team might want to get some information about a particular feature of the product, and there will also be times when the product team will need feedback on the overall product. To make sure you have all the feedback you need, preparing a conversation guide can help you structure the feedback and also track what each participant have been saying to you.
For example, during a four weeks limited release, you can ask questions about the onboarding in week one, questions about the customer experience in week two, questions about a specific set of features in week three and finally questions about the product overall in week four. Don’t expect the participants to cover every single aspect of your questionnaire; it is just a guide to help structure the data collection.
Track the participant’s feedback
To keep track of the participant’s feedback, I use a shared Excel sheet on Google Drive. In the Excel sheet, you can find the participants name, the contact details, the date of the provided feedback, the channel they used to give feedback and a summary of the feedback at a different point in time.
For example, if a participant has been giving feedback on Slack on Monday, do a quick analysis and create a summary of the feedback in your Excel sheet. You will likely have to develop multiple summary entries because the participant is likely to provide feedback on numerous occasions through the week.
Engage the internal team
Keeping the client feedback in one central location is ideal, depending on your setup, make sure that the team can access the participants feedback easily. To increase understanding of the participant’s challenges and frustrations, I also suggest inviting the product team to take part in the feedback collection. Let your stakeholders calling the participants, having the team talking to the participants makes a huge difference.
When to stop the limited release
You have been perfecting the customer journey and the participants have been telling you that the product is awesome, these are the signs when you should stop the limited release and start selling the product to earn revenue. Some products might be in beta for as long as a couple of years; it depends on your product, your market and the business you are into.
That it! You know everything I know about creating and running a limited release. I hope my learnings will help you improve your product and make it a success. Thank you for reading.