Visual & Interaction Design
Front end development
Jan 2018 - present
Thomas Aquinas (engineering manager)
Gaurav Pasari (engineer)
Michael Yi (engineer)
Max Lu (data scientist)
Alan Yiu (product manager)
Quora is a question and answer platform with a mission to grow and share the world's knowledge.
At Quora, we use notifications (in-product, push, and e-mail) to interact with users and make the platform feel alive. I was drawn to working on notifications because it is a messy problem area with high potential. When notifications became central to a new company strategy to drive engagement, I led a team of engineers and data scientists to improve the end-to-end experience of our notifications product.
My solution optimized scannability and user trust so that users could easily identify the notifications important to them and want to continue to receive notifications from Quora. Through product decisions, documentation, and thorough communication, I also built a foundation for notifications to be scalable and sustainable system.
Experimentation showed that this work increased push notification clicks and the subsequent time spent in the app by over 25%, with the effects continuing to compound, and reduced opt-outs by 43%. We also received positive feedback from users that notifications were easier to prase and from employees that the simplified systems were much easier to understand work with.
I took stock of the current state of the system from three different angles:
According to product surveys, users found notifications to be irrelevant, excessive, and sometimes confusing. I conducted some card sorting exercises to better understand and learned that users preferred certain types of content per channel, cared about different notifications based on their usage and perception of Quora, and found SMS and push to be highly intrusive.
After years of neglect, notifications lacked rhythm. I found inconsistent styling, unpredictable formatting, and wide variation across platforms.
Notifications are a major way to drive users to the product, and without checks and balances, different teams were indiscriminately adding new notifications. Like a game of Jenga though, you have to take out a block to add a block. Adding new notifications meant reducing a user's capacity to receive more, and in the worst case, unsubscribe or churn. Even without adding new notifications, we didn't understand the tradeoffs between sending one notification over another.
Better scannability allows users to quickly parse through a large volume of notifications and decide with which to engage. To get to this ideal, I focused on consistency, distinct visual hierarchy, and clear messaging.
I standardized patterns across the 160+ types of notifications and made rules for where there were discrepancies. To highlight a few decisions, I
With so many notifications, user research and an internal audit pointed towards leveraging chunking. I categorized notifications by intent and product area, using this system across push notification groups, in-product notification filters, and icon badges.
Trust keeps users coming back to the product, so I took a hard look at exactly what and how users were interacting with notifications.
For such a complex and valuable system, we decided to build a machine learning model to decide whether or not to send a given notification to a user. Our V1 model traded off between notification clicks and unsubscribes.
To garner more signal for the model and let users know that we were curating their experience, we let users give feedback via their in-product notifications and also in a random survey on push notification clickthrough.
We also made it easier for users to control channel sensitive notifications like push and email. We made the settings page directly accessible from the notifications page and brought parity between the individual settings and notifications.
Trust starts with not misleading people. An A/B test making user generated content, rather than the action taken, the title of the notification was metrics positive. Even so, we saw this as click-bait and didn't think the wins were worth infringing on product integrity. While engagement type notifications still highlighted content, transactional notifications highlighted the action taken in the titles.
A second contributor of integrity, badging is a high risk, high reward way to drive people to your app. Years of growth hacking made our badging behavior so arbitrary that even employees didn't know how clear their app badge! To make badging feel more intuitive and deliberate, I
While I had removed most buttons to reduce bias and cannibalization between notifications, relevant notification actions should still be accessible. These, I moved to an overflow menu where their location is predictable.
For more active users, aggregation makes notifications digestible. In addition to updating aggregation logic to be the same across all notifications, I also updated the clickthrough experience. Rather than taking users to a page with a static list, I redirected them to the aggregation source and injected notification content.
To prevent another re-haul in 5 years, I documented the new product design principles, as well as all contextual and historical information. Where documentation fails, I had also deliberately made product (e.g. a single categorization system rather than multiple) and implementation (e.g. warnings and abstractions) decisions to accommodate internal processes.
I also became an advocate for our notifications product. From strategy meetings to bug reports, I surfaced relevant information and considerations so that other teams would also come to feel responsible for up-keeping the notification system.
The work done until now is only the tip of the iceberg. Future areas of work include more iteration and optimization, more uniquely addressing edge cases, and addressing email.