Click to Pay Comprehension and Usability Testing
Project Overview
Click to Pay is a product that allows customers to get their online shopping done quickly and with fewer clicks. (You might have seen it while checking out at Costco!) There’s one big problem though: Online shoppers have no idea what it is when they see it. For this round of testing, we wanted to test the comprehension of messaging included in 3 enrollment flows.
Impact
The findings that came out of this research directly influenced the messaging guide made by our content design partners. This messaging guide outlines how Click to Pay can be effectively messaged in enrollment and education touch points to aid in shoppers’ comprehension and future usability.
My Role
I was the lead researcher for this project and had support from one other researcher. After understanding the research objectives from product and design teams, I put together a research plan, presented a research approach and timeline, wrote a test plan, recruited participants, conducted sessions, and presented the findings to the broader team.
Methods
Over the course of one week, 13 remote usability sessions were conducted with frequent online shoppers. Participants were sourced through usertesting.com’s participant panel.
Research Objectives
Research Questions
Do participants understand that C2P is an offering that is part of their card?
After reading the messaging, how do participants feel about signing up for C2P?
Do participants understand what setting up their card means?
After setting up, how do participants expect C2P to work? How does this compare to the experience they encounter on the merchant flow?
Project Goals
Understand users’ comprehension of the value of Click to Pay after going through enrollment
Understand if the messaging is enticing enough to participants for them to enroll or provision their cards to Click to Pay
Validate the messaging and how C2P is introduced to user through 3 different enrollment flows
Methodology
14 Participants took part in 1 hour long usability sessions where they were asked to walk through three different Click to Pay enrollment flows. The order of the flows shown were counterbalanced to maintaining consistent results. Researchers noted participant’s comprehension of Click to Pay at each stage of enrollment as well as their cumulative comprehension at the end.
Sudy Protocol
Introduction (5 minutes)
Walking through Flow 1 (20 minutes)
Walking through Flow 2 (10 minutes)
Walking through Flow 3 (10 minutes)
Merchant Flow (10 minutes)
Wrap Up (5 minutes)
Recruiting Participants
Participant Demographics
The target demographic for this study were frequent online shoppers with a mix of attitudes towards data security.
Age: Even mix from 18-50
Gender: Even mix
Technology: All own a smartphone
Education: High school graduate or above; prefer full- or part-time employment; students, homemakers, and retired are acceptable
Online purchasing:
o Must own and use a Visa credit/debit card
o Must share or have full responsibility for decisions regarding online purchases made with their credit/debit cards
Recruitment
The live interviews for this study were conducted with usertesting.com and participants were sourced through the tool’s participant panel. After writing the screener on usertesting.com, the tool automatically invited qualifying participants to join the study. This is a great option for a study with a more general participant demographic and a desire to hasten the timeline.
Sample Screener Questions:
What devices do you use to make online purchases?
Mobile Phone [Must Select]
Desktop/Laptop Computer [Must Select]
Tablet [May Select]
Other [May Select]
None of the above [Reject]
How often do you shop online, if ever?
Every week [Accept]
Several times per month [Accept]
Once per month [Reject]
Several times per year [Reject]
Once per year [Reject]
None of the above [Reject]
Testing Sessions
Comprehension Testing
Sample Questions:
Scenario: Let’s imagine that you opened your online banking app to check your accounts, and you are now logging in.
Text Message notification
Now that you are signed in to your banking app, what do you notice? [Notice the notification that has appeared]
If you were doing this at home, would you read that notification or dismiss it?
Please read over the notification that has appeared out loud.
How do you feel about your bank giving you this notification?
What is the notification telling you?
At this point would you be interested in learning more about C2P or not??
What do you think click to pay is? How do you think it might work?
Click to Pay Education Screen
What’s your first impression of this screen?
How do you feel about this screen? Is there anything that stands out?
Please read the header of this page out loud. What do you think this means? How do you think it will work?
What do you think of the Easy, Secure, Smart headings?
Please read the text under where it says “Easy” what do you think this means? How do you think it will work?
Please read the text under where it says “Secure” what do you think this means? How do you think it will work?
Please read the text under where it says “Smart” what do you think this means? How do you think it will work?
Based on what you’ve seen so far, assuming you were doing this with your own card, how interested are you in setting up your card with this service, on a scale of 1 to 5, 1 being not at all interested in and 5 being extremely interested? Why?
Usability Testing
Usability testing was included in this test to measure the effectiveness of the enrollment messaging participants were primed with earlier. The aim of this portion was to see how participants expect Click to Pay to work and how this compared to the experience they encounter on the retailer’s site.
There was only one task for this portion: Add a coffee machine to your cart and complete the purchase.
Synthesis
After wrapping up our interview sessions, we moved on to synthesizing our notes. Typically, I like using Mural during this phase of a research study, however, this project had a tight timeline that required us to deliver findings as soon as possible.
Instead of laying out virtual sticky notes and conducting a thematic analysis, I had our notes grouped my the different messaging copy that was shown to participants. Parsing through each group one at a time, I wrote my findings and pulled evidence directly into the final deliverable.
Sample Insights and Recommendations
Notification Banner Messaging Effective
“You set up your card here and somehow when you're at a retailer there's a button to access Click to Pay and you click it and it takes the information you previously entered and pre-fills it to complete the purchase, I guess.”
After reading this notification, participants generally understood the value proposition of Click to Pay. Some immediately estimated the interaction will happen on a retailer website and that their payment information will be auto filled.
“One click, like PayPal, you're always logged in. I presume it'd be like PayPal so you'd be logged into Click to Pay and then you could use that with certain websites.”
User’s experiences with other payment products like PayPal and browser autofill filled in the gaps of in the notification messaging.
Misleading Messaging About How it will Work
“I’m picturing that little icon next to where you're asking for your card information. You could click that and it auto-fills it for you.”
The frequent mention of the icon in the enrollment messaging indicated to participants that they will be clicking on the Icon in the merchant flow to complete payment. This understanding set participants up for usability errors in the upcoming merchant flow.
Usability Error: Customers Want to Click the Icon
Multiple factors seemed to influence participants’ misunderstanding of where to click:
Enrollment messaging emphasized the icon
‘Click’ verb present in the Click to Pay name
Design treatment of ‘Look up my cards’ appeared to separate it from Click to Pay
‘Look up my cards’ is plural, which is not intuitive for users who have only enrolled one card
“I'd be tempted to click ‘Click to Pay’... Because it talked about how anywhere you see the Click to Pay icon you can use it. So having seen the icon and the word ‘click’, it makes you want to click (the name/icon).”
“I can see the I (information symbol), but maybe I click on the (icon). I saw the “Look up my cards” but I didn't think it was related.”
Recommendation: Consider having the icon and name be clickable (potentially adding words like “Use” or “Pay with” before the name) instead of using ‘Look up my cards’.