Quora

Leveraging notifications for higher user engagement

Roles

User research

Visual & Interaction Design

Copywriting

Front end development

A/B Testing

Timeline

Jan 2018 - present

Team

Thomas Aquinas (engineering manager)

Gaurav Pasari (engineer)

Michael Yi (engineer)

Max Lu (data scientist)

Alan Yiu (product manager)

The Project

Quora is a question and answer platform with a mission to grow and share the world's knowledge.

At Quora, we use notifications (in-product, push, and e-mail) to interact with users and make the platform feel alive. I was drawn to working on notifications because it is a messy problem area with high potential. When notifications became central to a new company strategy to drive engagement, I led a team of engineers and data scientists to improve the end-to-end experience of our notifications product.

Solution

The Product

My solution optimized scannability and user trust so that users could easily identify the notifications important to them and want to continue to receive notifications from Quora. Through product decisions, documentation, and thorough communication, I also built a foundation for notifications to be scalable and sustainable system.

Experimentation showed that this work increased push notification clicks and the subsequent time spent in the app by over 25%, with the effects continuing to compound, and reduced opt-outs by 43%. We also received positive feedback from users that notifications were easier to prase and from employees that the simplified systems were much easier to understand work with.

Before (left) and after (right) of various notification product areas

Problem

Tragedy of the Commons

I took stock of the current state of the system from three different angles:

  1. User mental models
  2. According to product surveys, users found notifications to be irrelevant, excessive, and sometimes confusing. I conducted some card sorting exercises to better understand and learned that users preferred certain types of content per channel, cared about different notifications based on their usage and perception of Quora, and found SMS and push to be highly intrusive.

  3. Visual design
  4. After years of neglect, notifications lacked rhythm. I found inconsistent styling, unpredictable formatting, and wide variation across platforms.

  5. State of the system
  6. Notifications are a major way to drive users to the product, and without checks and balances, different teams were indiscriminately adding new notifications. Like a game of Jenga though, you have to take out a block to add a block. Adding new notifications meant reducing a user's capacity to receive more, and in the worst case, unsubscribe or churn. Even without adding new notifications, we didn't understand the tradeoffs between sending one notification over another.

Scannability

The Magic of Tidying Up

Better scannability allows users to quickly parse through a large volume of notifications and decide with which to engage. To get to this ideal, I focused on consistency, distinct visual hierarchy, and clear messaging.

Design System

I standardized patterns across the 160+ types of notifications and made rules for where there were discrepancies. To highlight a few decisions, I

  • defined a content template
  • separated user generated content from system copy
  • standardized voice, tone, and product vocabulary
  • pared down in-notification actions and links

Categorization

With so many notifications, user research and an internal audit pointed towards leveraging chunking. I categorized notifications by intent and product area, using this system across push notification groups, in-product notification filters, and icon badges.

User Trust

Honor Matters

Trust keeps users coming back to the product, so I took a hard look at exactly what and how users were interacting with notifications.

Machine Learning

For such a complex and valuable system, we decided to build a machine learning model to decide whether or not to send a given notification to a user. Our V1 model traded off between notification clicks and unsubscribes.

To garner more signal for the model and let users know that we were curating their experience, we let users give feedback via their in-product notifications and also in a random survey on push notification clickthrough.

Left: push notification survey; right: in-product negative actions

User Settings

We also made it easier for users to control channel sensitive notifications like push and email. We made the settings page directly accessible from the notifications page and brought parity between the individual settings and notifications.

Left: settings page access; center + right: corresponding notification and setting

Messaging

Trust starts with not misleading people. An A/B test making user generated content, rather than the action taken, the title of the notification was metrics positive. Even so, we saw this as click-bait and didn't think the wins were worth infringing on product integrity. While engagement type notifications still highlighted content, transactional notifications highlighted the action taken in the titles.

Left: engagement notifications highlighting content; right: transactional notifications highlighting action

Badging

A second contributor of integrity, badging is a high risk, high reward way to drive people to your app. Years of growth hacking made our badging behavior so arbitrary that even employees didn't know how clear their app badge! To make badging feel more intuitive and deliberate, I

  • untangled the badge count making every notification count once
  • moved badge contributing elements to the notifications page
  • preserved the red badge count UI for app badge contributors
  • consolidated notifications across all language silos for one true count

Actions

While I had removed most buttons to reduce bias and cannibalization between notifications, relevant notification actions should still be accessible. These, I moved to an overflow menu where their location is predictable.

Clickthrough Experience

For more active users, aggregation makes notifications digestible. In addition to updating aggregation logic to be the same across all notifications, I also updated the clickthrough experience. Rather than taking users to a page with a static list, I redirected them to the aggregation source and injected notification content.

Before (left) and after (right) aggregated notificaiton clickthrough page

Scaling

Planting the Seeds

To prevent another re-haul in 5 years, I documented the new product design principles, as well as all contextual and historical information. Where documentation fails, I had also deliberately made product (e.g. a single categorization system rather than multiple) and implementation (e.g. warnings and abstractions) decisions to accommodate internal processes.

I also became an advocate for our notifications product. From strategy meetings to bug reports, I surfaced relevant information and considerations so that other teams would also come to feel responsible for up-keeping the notification system.

Next steps

The work done until now is only the tip of the iceberg. Future areas of work include more iteration and optimization, more uniquely addressing edge cases, and addressing email.