select work
Acciyo Chrome Extension

How Usability testing helped increase weekly retention and remove barriers to onboarding

Client

Acciyo (now "Below The Fold") is a Chrome extension that provides a timeline of articles that were previously written on a topic so that readers could better understand the context of the news they're reading.

Challenge

Prior to launching on the Chromestore, the Acciyo Timeline faced several usability issues along with barriers to registration, which negatively affected the onboarding process for our users.

role

Usability testing | qualitative research | interaction design

  • usability testing
  • qualitative research
  • interaction design
impact

After implementing our research findings, we simplified the sign-up process and solved interaction painpoints as well as improved the onboarding experience. We saw the following results:

  • Increase in weekly retention to 29.9% from 11.1%

Now with a revamped product, we achieved the following wins: 

  • Raised $147k on Republic
  • #5 Product of the Day on Product Hunt
  • Growth in Acciyo's newsletter to over 5k readers
  • 01. Usability findings
  • 02. Ideating Solutions
  • 03. Measuring impact
usability Findings_

What did we want to learn?

Before launching the extension on the Chromestore, our team needed to observe how our beta audience experienced the early prototype. In particular, we wanted to observe how our users perceived its ease of use, the time it it took to accomplish certain tasks, and assess how successful our users were in accomplishing those tasks.

We decided to test for the following user tasks:

  • Get onboarded
  • Access the Acciyo Timeline
  • Navigate along the Timeline

These tasks were broad enough, which allowed us to follow up with specific questions as our testers experienced the prototype.

After 5 rounds of user-testing, we observed the following issues:

01/04. Lack of education

Upon getting onboarded, our users were greeted with a generic landing page that directed them to the app but was not deemed useful in teaching the user on how to experience the product; we learned that users preferred an onboarding tutorial.

"Even here on this screen...Diving me straight to these 3 things that you're recommending is less useful than me having an overview of how to use this tool"

-Session 1

"I'm assuming it's going to take me to a relevant tutorial about how it adds value (onboarding articles)"

-Session 2

02/04. Significant barriers to onboarding

Our testers were not keen on sharing their email with an unknown product upon signing up. We learned that they would have preferred to experience the product firsthand as anonymous guests.

These observations were confirmed after conducting a Funnel Analysis on Amplitude that tracked how many users moved along along the onboarding process from initial sign-up up to eventually interacting with the Timeline. We were surprised to see that no users continued with the sign up process and over 46% ended their session immediately.

"I'd expect to try the product for free first."

-Session 2

"If your product has something that I want and you're forcing me to sign up, I would use a temporary email as a test account.

-Session 3

03/04. Interaction painpoints

The Timeline posed a conflict to our users' mental model on scrolling vertically. Typically, a vertical scrolling gesture on a trackpad or mousewheel is associated with the intent of moving upwards or downwards. Instead, our users were surprised to see a "zoom and pinch" effect that stretched and expanded the Timeline instead of moving along a y-axis.

🙂
Our users expected to move vertically along the Timeline when scrolling.
😕
Instead, users saw a zoom and pinch effect which stretched the Timeline.

"I'm scrolling down but it's not allowing me to continue down on the timeline."

-Session 3

"It took me a while to know how to interact with this."

-Session 3

04/04. Hidden browser extension icon due to Chrome update

In order to save real-estate on the extensions menu, Chrome unpins newly installed extensions by default, thereby hiding stateful signifiers by which users would normally refer and interact. Our users were confused most of the time and were not able to access important features because they couldn't access the extension icon in the menubar.

"If the acciyo extension were here, I would be able to click on it."

-Session 2

"Acciyo gets left behind, it's difficult. It's easy for people to forget what they've downloaded. It's easy to forget."

-Session 7
Ideating solutions_

How we solved various UX and UI problems after our first round of user testing

After our first round of beta-testing, the product team hit the ground running with fresh insight on how to decrease friction during onboarding and on improving retention by teaching first-time users how to successfully navigate the Timeline.

We saw the following results:

01/04. Increased weekly retention after introducing onboarding demo

Nearly all of the users whom we interviewed prior to the launch expressed that Acciyo was unique in the way it presented articles dynamically along a timeline- however they expressed confusion at its novelty. We hypothesized that a brief onboarding tutorial would have a positive impact on the overall experience and thereby positively impact retention.

The data supported our hypothesis that an onboarding tutorial will positively impact retention. According to our Amplitude report,  the week-1 retention rate for our users who completed the tutorial was 29.1% compared to 11.7% for users who did not engage with the tutorial.

“Throughout 12 weeks, those users who completed the tutorial were more likely to remain on Acciyo.”

02/04. Removed barriers to onboarding

The first sign up flow had too many social authentication options to choose from. Our users expressed sentiments of "analysis paralysis" at being forced to choose to sign up with Twitter, Linkedin, Google, or email. After a round of user-testing, we learned that Google is the preferred social authentication choice for a quick sign-up process.

03/04. Solved interaction painpoints with a redesign of the Timeline

The current version of the Timeline displayed articles that were tightly packed together which negatively affected readability. It was also difficult for users to scroll vertically without experiencing a "zoom" effect which challenged exisiting mental models on how vertical scrolling works.

The redesigned version would create more categories to separate information, include affordances to encourage user-engagement, and make use of space to enhance readability.

Measuring Impact_

How solving for the user helped us achieve key wins

We did not limit the term "user" to be defined solely as our end-consumer. We treated our internal teams as users who also experienced painpoints that needed solutions. I helped spearhead design workshops to get us talking about our internal needs and goals:

01/02. Over 5k readers subscribed to our newsletter

Here's how we increased our newsletter subscriber growth: Along with Acciyo's product and leadership teams, we brought everyone together to discuss each department's painpoints and how to solve them by establishing team wide principles.

In the case of Acciyo's content team, the biggest painpoint was increasing the readership of its "Below the Fold" newsletter. I proceeded to identify the main roadblocks that held us back from reaching out to more readers.

After our remote workshop, we learned that our main painpoint was not having a content-publishing plan. We decided on the following principles for solving this problem:

1. Get the entire team involved to pitch stories

"'Open up a slack channel for below the fold pitches"

-Session 3

"Discuss headlines that aren't being reported during weekly meetings"

-Session 3
2. Content publishing should be timely

"Start to develop, have a list of what to use on a given day or schedule beforehand. Develop our calendar to get rid of that painpoint."

-Session 3

"We can plan ahead, but if the goal is to share things from the newsletter it has to be from that day. 60% should be from a preplanned calendar, but 40% is what we come based on the news cycle."

-Session 3
3. Strive for brevity

"Trying to consolidate the entire content into a title, all summaries should explain what happens and why it’s important."

-Session 3

02/02. We met our crowdfunding goals

With in-depth understanding of the painpoints our users experienced in the beta version, we are in a better position to deliver a revamped product that will help all of us understand the news better.

01. Usability findings
usability Findings_

What did we want to learn?

Before launching the extension on the Chromestore, our team needed to observe how our beta audience experienced the early prototype. In particular, we wanted to observe how our users perceived its ease of use, the time it it took to accomplish certain tasks, and assess how successful our users were in accomplishing those tasks.

We decided to test for the following user tasks:

  • Get onboarded
  • Access the Acciyo Timeline
  • Navigate along the Timeline

These tasks were broad enough, which allowed us to follow up with specific questions as our testers experienced the prototype.

After 5 rounds of user-testing, we observed the following issues:

01/04

Lack of education

Upon getting onboarded, our users were greeted with a generic landing page that directed them to the app but was not deemed useful in teaching the user on how to experience the product; we learned that users preferred an onboarding tutorial.

"Even here on this screen...Diving me straight to these 3 things that you're recommending is less useful than me having an overview of how to use this tool"

-Session 1

"I'm assuming it's going to take me to a relevant tutorial about how it adds value (onboarding articles)"

-Session 2
02/04

Significant barriers to onboarding

Our testers were not keen on sharing their email with an unknown product upon signing up. We learned that they would have preferred to experience the product firsthand as anonymous guests.

These observations were confirmed after conducting a Funnel Analysis on Amplitude that tracked how many users moved along along the onboarding process from initial sign-up up to eventually interacting with the Timeline. We were surprised to see that no users continued with the sign up process and over 46% ended their session immediately.

"I'd expect to try the product for free first."

-Session 2

"If your product has something that I want and you're forcing me to sign up, I would use a temporary email as a test account.

-Session 3
03/04

Interaction flaws

The Timeline posed a conflict to our users' mental model on scrolling vertically. Typically, a vertical scrolling gesture on a trackpad or mousewheel is associated with the intent of moving upwards or downwards. Instead, our users were surprised to see a "zoom and pinch" effect that stretched and expanded the Timeline instead of moving along a y-axis.

🙂
Our users expected to move vertically along the Timeline when scrolling.
😕
Instead, users saw a zoom and pinch effect which stretched the Timeline.

"I'm scrolling down but it's not allowing me to continue down on the timeline."

-Session 3

"It took me a while to know how to interact with this."

-Session 3
04/04

Missing app cues due to update on Chrome

In order to save real-estate on the extensions menu, Chrome unpins newly installed extensions by default, thereby hiding stateful signifiers by which users would normally refer and interact. Our users were confused most of the time and were not able to access important features because they couldn't access the extension icon in the menubar.

"If the acciyo extension were here, I would be able to click on it."

-Session 2

"Acciyo gets left behind, it's difficult. It's easy for people to forget what they've downloaded. It's easy to forget."

-Session 7
02. Ideating Solutions
Ideating solutions_

How we solved various UX and UI problems after our first round of user testing

After our first round of beta-testing, the product team hit the ground running with fresh insight on how to decrease friction during onboarding and on improving retention by teaching first-time users how to successfully navigate the Timeline.

We saw the following results:

01/04

Increased weekly retention after introducing onboarding demo

Nearly all of the users whom we interviewed prior to the launch expressed that Acciyo was unique in the way it presented articles dynamically along a timeline- however they expressed confusion at its novelty. We hypothesized that a brief onboarding tutorial would have a positive impact on the overall experience and thereby positively impact retention.

The data supported our hypothesis that an onboarding tutorial will positively impact retention. According to our Amplitude report,  the week-1 retention rate for our users who completed the tutorial was 29.1% compared to 11.7% for users who did not engage with the tutorial.

“Throughout 12 weeks, those users who completed the tutorial were more likely to remain on Acciyo.”
02/04

Removed barriers to onboarding

The first sign up flow had too many social authentication options to choose from. Our users expressed sentiments of "analysis paralysis" at being forced to choose to sign up with Twitter, Linkedin, Google, or email. After a round of user-testing, we learned that Google is the preferred social authentication choice for a quick sign-up process.

03/04

Solved interaction painpoints with a redesign of the Timeline

The current version of the Timeline displayed articles that were tightly packed together which negatively affected readability. It was also difficult for users to scroll vertically without experiencing a "zoom" effect which challenged exisiting mental models on how vertical scrolling works.

The redesigned version would create more categories to separate information, include affordances to encourage user-engagement, and make use of space to enhance readability.

03. Measuring Impact
Measuring Impact_

How solving for the user helped us achieve key wins

We did not limit the term "user" to be defined solely as our end-consumer. We treated our internal teams as users who also experienced painpoints that needed solutions. I helped spearhead design workshops to get us talking about our internal needs and goals:

01/02

Over 5k readers subscribed to our newsletter

Here's how we increased our newsletter subscriber growth: Along with Acciyo's product and leadership teams, we brought everyone together to discuss each department's painpoints and how to solve them by establishing team wide principles.

In the case of Acciyo's content team, the biggest painpoint was increasing the readership of its "Below the Fold" newsletter. I proceeded to identify the main roadblocks that held us back from reaching out to more readers.

After our remote workshop, we learned that our main painpoint was not having a content-publishing plan. We decided on the following principles for solving this problem:

1. Get the entire team involved to pitch stories

"'Open up a slack channel for below the fold pitches"

-Session 3

"Discuss headlines that aren't being reported during weekly meetings"

-Session 3
2. Content publishing should be timely

"Start to develop, have a list of what to use on a given day or schedule beforehand. Develop our calendar to get rid of that painpoint."

-Session 3

"We can plan ahead, but if the goal is to share things from the newsletter it has to be from that day. 60% should be from a preplanned calendar, but 40% is what we come based on the news cycle."

-Session 3
3. Strive for brevity

"Trying to consolidate the entire content into a title, all summaries should explain what happens and why it’s important."

-Session 3
02/02

Over $147k raised in crowdfunding

With in-depth understanding of the painpoints our users experienced in the beta version, we are in a better position to deliver a revamped product that will help all of us understand the news better.

Next case study
NYC Mayor's Office for Economic Opportunity