Case Study:
Formative's Onboarding
Discover how I helped Formative Increase activation by 754% in a 2 week design sprint!
INTRODUCTION
_________________
PREFACE
A number of business and product decisions led to this specific design challenge becoming an urgent priority. It's my hope to be fully transparent and forthcoming in this case study, explaining the miscalculations and successes along the way, that led to the final outcome.
MY ROLE
Sole designer (end-to-end). Strategy, planning, ideation, research, user testing, wireframes, prototypes, high fidelity.
TEAM
Strategy & brainstorming sessions with the co-founders (CEO & CFO). Pair design sessions with the customer facing team, sales & developers.
TOOLS
Pen & paper, Adobe XD, Asana, ProductBoard, Airtable, Intercom, Maze, Metabase, Slack, Google (Forms, Meet, Optimize, Analytics).
TIME
2 week design sprint & 6 weeks of development.
THE CONTEXT
_________________
PAST BUSINESS STRATEGY:
When I first joined Formative, two years earlier, we had one main business and strategy objective: focus on selling large premium deals to school districts. The idea was that we could maximize revenue by charging not only for every teacher at the school or district but also per student. Additionally, we were able to lock in longer-term contracts (often multi-year deals), generate additional revenue through services such as training and provide access to admin tools that weren't available in an individual teacher plan.
While there is revenue to be made with individual teacher upgrades, almost no teacher would be willing to pay for a classroom tool that required an additional monthly fee per student.
As the founders, Craig & Kevin, can confirm, early on during our strategy sessions I made a strong push to focus more attention on individual premium teacher upgrades, as a short-term win. I felt it would be faster and easier revenue, that could bolster our runway. However, the founders determined that the long-term math favored school and district sales. So, we set course, full sails ahead in that direction.
PAST PRODUCT OBJECTIVES:
To align with the business strategy, we chose 3 main design or product objectives:
1) Completely redesign the FREE product & features offered for individual users. At its core, it's a free tool for teachers to create an interactive digital classroom that tracks student progress & data. The reason for this focus is that often times school sales hinged on their teachers trying out our free product and communicating with decision makers whether or not they loved the product.
2) Create robust tools for our premium admin level users from schools & school districts. Covering needs such as collaboration features for co-teachers or professional learning communities. As well as, enabling the ability to track data & overall performance on standards, benchmarks and common assessments across all of their schools, teachers & students.
3) Redesign the landing page to attract more admin level prospects to make purchases for schools & school districts, while also increasing our rate of free teacher sign ups, to build on the vocal advocates within schools.
THE RESULT:
These product updates and releases were a hit! We made great strides in our annual revenue, increasing it by 667% over the previous year. There was just one issue. If you're familiar with EdTech, the sales process involving schools or districts can be painfully slow. Due to limited windows to take advantage of approved budgets, that are often planned & allocated a year in advance, on average, school sales can take 6-9 months and district sales take upwards of 12-18 months. And often times it can take just as long to get paid on the invoices.
So, while we were killing it with invoiced sales, the checks were not always coming in as quickly as we hoped. The founders had always been very transparent with us and we were aware that certain gambles were made on the runway (being pre-series A & highly dependent on revenue generation), by hiring additional higher cost teammates, getting a new office and other internal costs that ramped up. We suddenly we found ourselves in a precarious situation.
THE CHALLENGE
_________________
THE PREMISE:
This brings us to "the Challenge". We agreed as a team that we needed to divert some attention and resources to make individual premium teacher upgrades a priority. This could generate a faster (albeit smaller longer term) source of revenue, to help alleviate the runway concern.
We realized that as part of our effort to get our premium teacher upgrades up to a level that would make an impact on our runway, additionally we needed to improve activation rates for new teachers. We define activation as:
1) A teacher completes the signup process
2) They create an assessment or assignment
3) They assign it to their classroom
THE PROBLEM:
It was clear that in order to improve activation, we needed to improve our new user onboarding. While we did not completely forgo efforts to provide onboarding in the past to new teacher signups, it certainly was neglected. We had a powerful free tool, that we knew could change a teacher's classroom, improve student performance and more importantly, change lives! However, that only happens if they get activated. Once teachers are activated, our retention and engagement rates go through the roof!
The problem was our activation rate for new teachers was an abysmal 3.28%!
THE BUSINESS GOAL:
Our math told us that without any improvement to our upgrade flow (which was a different successful sprint that we ran later, with an upcoming case study discussing how we improved trial signups by 221.7% & premium upgrades by 88.1% with just a 1 week sprint) we would need to increase our new user activation rate up to 20%.
PAST SOLUTIONS
_________________
Before we delve into the "ah-ha" moment of discovery, let's take a step back and discuss what we had in place for onboarding, prior to the sprint.
VERSION 1:
When I first joined Formative, the onboarding was a nothing more than a welcome message, with an infographic describing how the site worked and a button prompting you to go to your dashboard. The metrics for this were unavailable, but we knew from talking to users that there was room for improvement, as there was no guidance in how to use the product, outside of a few help center articles.
VERSION 2:
My first involvement in any onboarding solutions was constrained to "what can we implement fast, with as minimal development involvement as possible, that could potentially make an impact"? We first explored a low development cost solution, in January of 2017, by simply opening an intercom welcome message from the CEO, with a quick how it works video, along with links to our community & help pages for more info on how to use the site.
Our community pages were actually filled with incredibly useful how-to guides and videos, made by our customer-facing team. However, while we did not have much previous benchmark data to compare to, as the data showed us, this approach was not establishing impressive activation numbers...we were at 3.09% activation.
Intercom welcome message with links to the help center.
VERSION 3:
Next, in Feb of 2017, we allocated 1-2 days (between other pressing priorities) to try exploring an iteration on the previous concept, by bumping the welcome message out into a centered popup modal and changing the "how it works" video to a GIF. Additionally, when the user closed the popup, our help button drop-down auto-expanded to show the steps of the new user setup, with step one completely as a sense of momentum. These steps could be clicked to automatically take you to the associated screen & play a video of how that area of the product worked.
This was a complete failure. The activation rate barely budged to a whopping 3.28%. The simple truth was that for this neglected part of our site, we did not allocate the proper time, research or design process to make it work. We chose other priorities to give that attention to, and it showed.
Welcome screen with how it works video & prompt to learn more.
Setup steps appear after the welcome screen.
If the user skipped the optional steps / how to videos, there were no tool tip walkthroughs in the Builder to guide them. The only indication of what to do was a bouncing blue "+" sign button, to add questions or content.
THE DESIGN PROCESS
_________________
PLANNING:
Every Thursday, during my time at Formative, I would do sprint planning with the founders to review the roadmap, discuss strategy, review metrics & analytics, past user feedback or needs, create a game plan for the upcoming week and set measurable goals or accountability for designs, along with the live release. Generally, we'd focus on 1-week sprints. However, during our 1-hour planning session, we determined this onboarding & activation effort would require extra attention, with a full 2-week sprint.
RESEARCH:
Before conceptualizing any potential solutions, it's imperative to quickly understand the user's needs, pain points, and frustrations.
A) Review Past Feedback & Notes -
I began the sprint by reviewing user feedback that we continually collected from sources such as Intercom, email, conversations & message boards. Which was then organized into folders within ProductBoard, based on specific types of user needs and user types. I also reviewed past user testing session notes that had been sorted and categorized within Airtable, looking for trends. Upon review, I compiled questions that need to be answered and considered assumptions that needed to be validated.
I reviewed & grouped feedback related to onboarding, activation & new user experiences, in order to get a better understanding of needs.
Staying organized with previous notes in ProductBoard & AirTable, allowed me to easily access user feedback relevant to the sprint at hand and look for trends.
B) User Surveys -
I sent out surveys to users who had activated, users who didn't activate, teachers who had never used our site before & users who had completely fallen off.
I inquired into everything from expectations & needs within a new user experience, to what they liked, disliked or needed to be in our current flow or competitor's onboarding experiences. As well as, what types of needs, benefits or features had helped convince them of using our site. I asked their thoughts on what was missing, along with what they weren't even aware was a part of our product and which of those may have motivated them to engage with the product.
Less experienced or new users indicated one of the top features they wanted to see was a "Library of Formatives & Question Bank".
Less experienced or new users also rated an "Automated Formative Builder" high on their list.
Interestingly, users who had fallen off also wanted a "Library" feature, giving them pre-made content.
C) User Observations & Discussions -
I conducted both group and individual discussions, along with observations of both current users and new users who had never used Formative while exploring our product. I also observed them using the onboarding flows of competitors. I tried to find trends in what they liked, disliked, stumbled through or succeeded with.
D) Formative Message Boards -
At Formative, we've built a great relationship with our active users and we have a thriving community, which has allowed us to maintain daily user testing, user research and user pair design sessions. One of the most useful tools for immediate feedback has been active message boards we've cultivated, which allows us to immediately engage our user base with questions.
An example snippet of the many message board interactions.
These convos helped us quickly ask questions & gain insights.
E) Competitor & Pattern Research -
I spent time researching existing user patterns that have proven successful for other products and reading case studies as to why. I reviewed our competitors for clear strengths and weaknesses, looking for any applicable insights into our own approach. I collected these into a mood board of inspiration and notes.
Creating a research & inspiration mood board with notes of ideas, along with what works & what doesn't work can be useful in the ideating process.
UNDERSTAND:
A) Trends -
The research yielded a number of trends, such as:
- Users wanted more thorough tool tips &/or explanations of how the builder & question types worked.
- Users seemed to have a higher "wow" factor over a select few question types.
- Users did not grasp the power of the Live Results & Feedback tool within the product, until using it in their classroom. However, if that "ah ha" moment could be captured during the onboarding, many users indicated higher likelihood of assigning it to their classroom & thereby activating.
- Teachers were accustomed to products with much heavier upfront information gathering with other EdTech products, than we asked for. So, asking for their school, grade, subject & standards info did not seem to turn too many off. The main edge case was international teachers needed a way to bypass or manually enter standards and subjects that are different than US based ones.
B) Insights -
One insight stood out from the rest. A sizable majority of new users have no idea whether or not they even want to use formative in their classroom, as they're signing up. They're just halfway committed to exploring a product that they often times heard about from another teacher and they just want to see if it's the right fit for them first, before taking the big leap of assigning it to their classroom. Generally speaking, their initial mindset is they just want to "toy around with it" for a bit first.
C) The BIG Pain Point -
Adding to the trends and insights that I uncovered through the research phase, there was one potentially significant pain point that could potentially have the greatest impact, if properly solved. I discovered that other than providing the "how it works" video and setup steps, once the user closed the popup they were essentially landing on an empty dashboard with no indication of what to do and no motivation to do it.
The thought of creating an assignment from scratch with a new tool that they were only testing out, was a daunting task.
THE HYPOTHESIS:
It became increasingly clear that based on user research, that the missing link was likely offering new users pre-made content or templates, immediately accessible upon sign up. And, so, a hypothesis was born.
THE CONSTRAINT:
One of the main constraints with tackling this hypothesis is that the manpower and hours needed to make quality content for every conceivable grade, subject and standard combination was beyond our capabilities. Even if we focused on just a few core subjects & grades, the likelihood of us creating assignments that users would consider high enough quality to assign it to their classrooms with little or no need to customize it would be unlikely.
THE CORE CONCEPT:
We landed on a core concept, that we needed to validate - what if we develop a "public library", where current active users would be encouraged to upload assessments or assignments that they've already made, tagging them with the relevant subject, grade, and standards, to share publicly with other Formative users.
Users could then give ratings on shared content, allowing us the ability to not only stock our product full of user-approved content for all users, but to also capitalize on this in our onboarding process by providing a selection of pre-made templates for new users, specifically filtered down to the grade and subject they chose in their sign up process.
However, we needed to test the concept out against other ideas & remain open to recognizing the best direction, at this point.
SKETCHING & ROUGH USER FLOWS:
A number of rough concepts & flows were sketched out on sticky notes & stitched together in Adobe XD as rough clickable prototypes. Many of these ideas came together in collaborative pair design sessions with individual teammates or users.
Over the course of many sketching & white boarding sessions some concepts were beginning to take shape around guided flows for either creating your own content or choosing pre-made from the library. A gamification concept, a badges concept, mobile inspired walkthrough & more.
DAILY USER TESTING & TEAM FEEDBACK:
I conducted daily user testing, pair design & feedback sessions throughout every phase of the design process. However, it became especially important in the "Concept" phase, as we ideated different approaches. This allowed the design to be agile & evolve quickly. Methods included A/B tests, observations, discussions & collaborative design.
WIREFRAMES:
As certain ideas took hold, I moved past sketching & concept notes to mid-fidelity wireframes.
HIGH FIDELITY PROTOTYPES:
The successful pieces of the prior concepts began to take hold. I iteratively began creating higher fidelity mockups, which were then stitched into clickable prototypes & A/B tested.
A hybrid concept began taking shape, as I A/B tested different concepts. Taking pieces from concepts that worked, and iterating on pieces that didn't.
USER FLOWS:
Earlier in the design phase, the user flow concepts were more rough, using sticky notes to map out the basic flows. However, as a winning concept began taking shape, I assembled a more comprehensive user flow to help understand all the variables. As well as, to provide extra documentation for the development team.
An example of one of the user flow charts that was being ideated during the concept phase.
THE SOLUTION:
By the end of the 2 week sprint, I felt pretty confident that the pair designs, continuous feedback and ongoing user testing had produced actionable results that led me an onboarding solution that solved the primary user pain points & business goal.
The screens below show some of the core elements in the onboarding experience. You can also visit a clickable prototype that was used in testing (note that there are some dead end routes, so use your back button when needed & due to the limited functionality of the prototype, various states such as hover or automatically triggered tool tips require an extra click in this format):
https://xd.adobe.com/view/f8e695a2-3fb1-4884-958d-f4284b6546e4/?fullscreen
Below are screens from the "Upload / Create" path, had the user went that route (up to the assign portion, where the paths converge):
THE OUTCOME
_________________
POST-RELEASE USER TESTING
Further validation is always needed, after releasing a high priority feature, concept or flow. We observed users trying the release & talked to them at great length afterwards. The consensus was that while there were still areas for improvement, overall this was a big step in the right direction.
METRICS RESULT:
After the 2 week design sprint, we felt confident in the concept that we had iterated along the way. We chose to move forward with development. Over the course of roughly 6 weeks, only about 1/3 of the flow had been implemented, as the dev team had other performance & optimization priorities they were also juggling (a number of tool tips had been incomplete, amongst a few other pieces). However, the core elements were in place & we had data to review.
The activation rate increased to 26.38%! An increase of 754%!
Review of the following weeks of data showed that the increase remained, fluctuating from 24.7% to 30%+.
NEXT STEPS
_________________
SMALL, MEASURED WINS:
While there hasn't been a sprint since then solely dedicated to onboarding. We have been exploring ways to tweak, iterate & improve it based on data, feedback & A/B testing of the release. I've allocated pockets of time in subsequent sprints to iterate the onboarding concept & test out new mockups. Below are a few examples of potential iterations that may be implemented, if they test well:
RECOMMENDATIONS & REFERENCES
_________________
"Mike has an uncanny ability to create stunning designs that are both breathtaking to look at and functional for the user. He prides himself on doing user research and is often able to discover what the user really wants and turn that into elegant and practical designs.
He cares about analytics and can really focus on dynamic results when given a goal. He designed a strategic premium upgrade system for our website that nearly DOUBLED THE CONVERSION of our free users to premium. He is also a joy to be around."
CRAIG JONES, CEO of FORMATIVE
"Mike Joyce is a very talented UI/UX designer and contributed greatly to the direction of the product at Formative. I've had the opportunity to work with many high level creative minds throughout my career and Mike falls within the top echelon of talent.
Through extensive user research, user testing, analytical thinking and problem solving, Mike gained invaluable insights and understanding into our user's needs. This translated into creative solutions, engaging features, design strategy and beautiful user interfaces that helped our sales team feel confident in the product that we presented to prospective buyers.
Mike is very easy to work with, a great collaborator and team player, who has the rare ability to turn vague or confusing concepts into a powerful and easy to use product. I would highly recommend his design services to any organization."
JONATHAN MANN, VP of SALES & SUCCESS, FORMATIVE
"I've known Mike outside of work for years, but, that doesn't take away from the unbiased recommendation I'd like to give him, based on our work experience together. Mike is an exceptionally talented designer, manager, leader & overall creative mind. The ideas & solutions he's come up with on projects we've both been a part of were truly amazing. I'd give him the highest recommendation for any project that needs creative direction, UI / UX conceptualization, innovative ideas, or, leadership & management."
CHERYL LAYUG, HEAD SUCCESS MANAGER, FORMATIVE
"Mike understands the importance of gathering different perspectives and will make informed decisions based internal team design sessions and regular user testing. He has partnered up with me to lead our super user program and regularly facilitates group design sessions as well as discussions. He is well liked by our super users because he listens well, is prompt in responses, and provides thoughtful explanations for anything he shares.
He also exhibits those same qualities in communicating with our internal team and can effectively collaborate remotely, as well. Mike actively engages in research to constantly improve within his role on the team. For example, has led the team in researched-backed, efficient design sprints. He has also found and helped invest our team in systems that make our collection and analysis of user feedback extremely more efficient.
In summary, I have really enjoyed working with Mike because of his effective communication skills, his thorough methods of gathering feedback, and how he constantly works to improve what he does."
DAVID KWAN, HEAD OF COMMUNITY, FORMATIVE
