Lean UR: How Firefox Lite finds user insights
The 4 steps our product team applies to find valuable insights that support us to make the right decisions.
Firefox Lite is a lightweight browser made for emerging market users with storage/ data constraints on their mobile devices. Last week, Firefox Lite reached to 1.6M monthly active users, which was the highest number that it has had so far. 🙌
To keep improving Firefox Lite to provide a better user experience for our users, we did some A/B testing on the top sites, which is one of the most used features in Firefox Lite. We looked into the telemetry data, listed assumptions, ran A/B tests…which was taxing on the entire team, including PM, UX, devs, QA, and data analysts, but unfortunately, the test failed.
“Why did the test fail?” we asked ourselves during our sprint retrospective.
It ended up that our A/B test was too complex for a small team. 6 groups with different top site variants were rolled out at the same time, which answered the 3 assumptions we listed. We were too ambitious to find all the answers that could influence the UX design, without thinking about how we could narrow down the questions by making our own decisions based on the learnings from the previous user research. Eventually, the test scope was too aggressive that it didn’t allow any human errors to happen when team members passed on work from one to another.
We need to level up and transform how we do research: to have a lean approach which is leveraging different methodologies from qualitative user research and quantitative data analysis on the same group of users, at the same time. And most importantly, ask the right questions to make the right design decisions.
The Firefox Lite team
Our product team consists of dedicated team members and shared resources. The product team is small but flexible on supporting each other due to the research resource shared. Product managers can dig into telemetry data, and designers can run user tests.
Just like most fast-paced software companies, things are changing fast. Sometimes we swap priorities based on the environmental changes or the new goals that the organization sets. Therefore, moving fast to adopt changes and efficiently find answers to the research questions is key. Mixing qualitative and quantitative methods in user research can help us reduce the time and effort.
Our recent research projects for Firefox Lite are mostly initiated with the insights from the telemetry data, then dig deeper by qualitative user research and following-up on “why”. We’re lucky to have a user researcher and data scientists all talented, collaborating closely as a team while running research projects. Both qualitative and quantitative mindsets are complementary to the research they conducted.
If you put user needs in mind and orient yourself by asking “why” questions to the user scenarios, you will be able to translate the quantitative data with qualitative insights.
- Lany, senior data scientist in Firefox Lite team
How to Lean UR?
The spirit of the Lean UR is to move fast and respect the shared research resources. The following is the best practices that we developed from our lesson learned:
Step 1: Frame research objectives and questions that match the project goals.
🧑💻 Who is involved: PM and UX
Product managers are usually the people who initiate a project and set goals for it based on the initial findings from the telemetry data. To make sure the research objectives are matching the project goals, we found it more efficient if the PM and UX draft research objectives and questions before inviting researchers to kick-start it.
Step 2: Prioritize the research questions.
🧑💻 Who is involved: PM and UX
As I mentioned the failed test in the beginning, we designed an A/B test with a large scope which was too big and complicated for a small team. I would suggest to ask the team some questions when making decisions on running a research project or not:
- Is it really something that you need to know before making the right design decisions?
Any previous research that can provide some pieces of knowledge which can guide you to make good decisions? - Is the usage rate of the feature high enough to impact the entire product significantly if you find the best solution by doing user tests or A/B tests?
Let’s say the feature only has 4% of the usage rate. How can it help with the overall retention rate if you enhance it to 100%?
Here’s another story: In the project “Home Customization”, we were considering running an A/B test on the feature button (which is next to the search bar on Firefox Lite Home) to see if the Smart Shopping Search or Private Mode is the best bet for app retention.
After looking at the usage data and the previous user testing results, we decided to make the decision without spending time and effort on running a test. We’re confident to have the Private Mode next to the search bar as it makes more sense to the UI layout and the usage of the Private Mode is much higher than Smart Shopping Search.
The decision was taken immediately, and our research resource was saved for the next big project, wonderful! 🙌
Step 3: Sit together with user researchers and data scientists to identify the right methods to answer research questions.
🧑💻 Who is involved: PM, UX, UR, and Data scientists
After prioritizing the research questions, now it’s clear for our user researcher and data scientists to move forward on the important ones to find the best combination of research methods. The following tips are to make the test lean and effective on research resources:
Quantitative research
Tip #1: Make a small number of A/B test groups and iterate multiple times.
Tip #2: Collect the data only when it’s essential to the research questions. Explore ways to collect data that is not sacrificing users’ privacy.
Consciously collecting telemetry data can not only protect your users’ privacy but also reduce the effort of developers from setting unnecessary telemetry. Moreover, it helps you build trust with users and reduce operational risk in your organization.
If you don’t need a piece of data, don’t collect it.
If you need a piece of data, keep it for only as long as necessary and anonymize the data before you store it.
Qualitative research
Tip #1: Evaluate the research scope and invite designers to support small and medium design validation.
Not all research has to be conducted rigorously to find valuable insights. We categorize user research with 3 levels in terms of scope:
- [Small scope] Internal design validation
This is the test that can be conducted by designers. Sometimes a quick-and-dirty user testing in the office reveals the majority of the insights for an iteration. Examples like icon testing for a new feature, usability testing for the new menu panel can be frequently conducted in any stage of the design process. - [Medium scope] External design validation
If the project includes some big changes on a high-visibility feature or some user flows that are not commonly used in the industry, we’ll consider running a remote user test or survey to validate the design with global participants. Unmoderated usability tests on Usertesting.com saves time on conducting user tests with numbers of participants. However, it has a lower tolerance for the comprehension of the test script. Some pilot tests for checking the wording and task flow are essential to the success of the test. We found it easier to make the test scripts unbiased and direct when we have more than one person working on it. Therefore, sometimes we’ll have designers making a draft script and then get reviewed and shipped by our user researcher. - [Large scope] Fundamental user research
Firefox Lite has some large-scope fundamental user research that helps us understand our users more. Research projects like Persona, Push factors, and Pull factors require the expertise of our user researcher and highly rely on the collaboration with our data scientists. That’s why we have designers supporting the design validations so that our researchers can contribute more time and effort on building fundamental knowledge for Firefox Lite.
Tip #2: User testing with no more than 5 participants.
3~5 is the magic number for recruiting participants because you’ll find more and more overlapping insights if you test with more participants in the same category. The article that Jakob Nielsen wrote 20 years ago is still a good reference for today’s user researchers. Highly recommended.
Step 4: Review results simultaneously to generate comprehensive insights.
🧑💻 Who is involved: UR, and Data scientists
It’s natural to have comprehensive insights from both ends because our user researcher and data scientists have worked as a team since the beginning of the planning stage. Take Project Persona as an example: deep-diving into telemetry data can help us categorize users into several groups of personas, and user interviews can reveal the “why” under their behavioral patterns.
Interested in the Project Persona? Click here to read more stories.
Research insights are one of the tools that help us get greater confidence in making product decisions.
There are always insights from tests, but you may not need all pieces of evidence to avoid making wrong product decisions. Our lean UR approach can lower the time and effort of running unnecessary tests, focus more on important features, and test & iterate them more.
We’re continuously exploring and polishing our lean UR process. Feel free to share your thoughts with me on how your organization runs research projects :)
Special thanks to our user researcher, Ivonne Chen, and Data scientist, Lany Liu for generously sharing their work experiences and thoughts on research that were summarized in this post.
References:
- Simultaneous Triangulation: Mixing User Research & Data Science Methods — Colette Kolenda
- Why You Only Need to Test with 5 Users — Jakob Nielsen
