A/B Experiments leading to increased conversion on checkout
Over the course of 1.5 years, I helped re-shape the checkout experience by designing A/B experiments, adding new payment methods, preparing for internationalization, & designing updates to comply with industry regulations. We went from a limited design that was causing user confusion to an optimized experience that resulted in a +12% increase in revenue.
COMPANY
Skillshare
timeline
Q1 2020 - Q3 2021
MY ROLE
IC Product Designer
problem
Low conversion rate, customer confusion, not optimized for mobile web & upcoming compliance requirements that were difficult to fit in the old design.
solution
Iterate through experimentation to improve the conversion rate, provide clarity about the subscription, & make the design more flexible for future changes.
impact
The new design resulted in a +12% increase in revenue & a +4.77% increase in conversion.
The Process
It all started with an entire site rebrand. The checkout design needed to be optimized after it was updated with new colors, typography, & visual elements. We also wanted to address the low conversion rate and feedback of user confusion. Here is a timeline overview of the work we did to get to the new design:
Phase 1:
Membership Plan Experiments
The First A/B Experiments
The first thing we did after the brand refresh was experiment with different membership plan options. We started with 2 plans (monthly & annual) and ended with a 3-tier plan (monthly, annual, & direct to paid). The previous design was not built for a 3-tier plan, so I got to work on some new design options.
Blue Sky Design Proposal
I analyzed user feedback and conducted a competitive analysis to inform a future-thinking design proposal for the 3-tier plan. I knew this would be a tough sell, so I also created a proposal to run iterative experiments that could get us to the Blue Sky designs in a controlled, step-by-step way. I presented both proposals to the C-suite. They were not comfortable with building the Blue Sky designs, but they approved the iterative experiments.
Validating with User Research
I also got approval to run a user research study to validate the first experiment. I showed a group of users the old design & a new UI design prototype for the 3-tier plan. They overall preferred the new design & gave us 2 key insights about the old design.
user research insight 1
Overall, the membership plan options were difficult to understand in the old design.
user research insight 2
The majority of users felt uncomfortable choosing a payment method with the old design.
First Design Experiment
The response from users helped us to feel confident that the new designs would be a success, so we ran a live A/B experiment using the same designs I shared in the user research.
Spoiler Alert: It didn’t win
-15%
Direct to Paid
Users selecting the new 3rd plan option.
-37%
Refund requests
Users asking for refunds, especially for the direct to paid plan option.
This A/B experiment was a valuable failure. The new designs resulted in less users selecting the 3rd plan option. However, they were also asking for fewer refunds (down -37%). This signaled we made the 3rd plan option more clear to our users, but they didn’t select it as much now that they understood it.
The decreases were too significant for us to accept at that time, so we reverted to the old design. But, the results of this experiment combined with another round of user interviews helped us to reconsider our entire membership plan model a few months later.
Phase 2:
More iterative experiments
Knowing our business needed a win, we put the larger design changes on hold in favor of making smaller changes that we had more confidence in. Over the next few months we ran 3 experiments that change the overall structure of the checkout page.
Experiment 1
• Added in 2 new payment method options
• Consolidated our CC fields to Stripe single card elements
• Reduced the information on the page
Experiment 2
Moved the total box with the CTA higher up on the page.
Experiment 3
Finally, the biggest change we made was to change the membership plan model to a single tier plan.
Phase 3:
The Next Big Swing
While the smaller experiments were running, I continued to work on new design concepts in the hopes that we could take another big swing one day. Finally, that day came and we got stakeholder buy-in to run another design update to checkout. I shared my reasonings for each of the changes with the C-suite and we got approval to run it as an A/B test.
Design Considerations
To ensure the final designs were addressing the original concerns, I kept these core ideas as a focus when creating the final designs:
Flexibility
The design had to be flexible enough to accommodate 24+ variations and have space for the regulations, payment method changes, & internationalization we had upcoming on our roadmap
Meet User Needs
I also wanted to address the concerns we heard from our users by making the membership plan information clear & making it more comfortable to select a payment method with confidence.
Future Thinking
This new design also had to unblocked future experimentation capabilities like an enhanced discount code experience, further opportunities to run A/B experiments, & be optimized for mobile web.
Design Highlights
Unified Background
The previous design used 2 colors to separate the action from the information on the page. Now that the information was streamlined, I unified the background color to create a sense of focus and unity to the content, signaling that all information on this page was important for the user to process in order to complete the transaction.
New Header & Components
The previous design didn’t include instruction for what action the user should take first. In this new design I added the header “Select a payment method” to give that instruction & to accommodate the reassuring message tooltip explaining why they need to enter payment information.
New Total Box
The biggest leap in the new design was an updated total box design. This new design was optimized for the 35+ variations that could occur. It also provided space for potential future transaction items and for clarity on what the user could expect to pay after their free trial if discounts were applied.
Outcome
+12%↑
Revenue
Increase in up-front cash, net of refunds.
+4.77%↑
Conversion
Increase in checkout completion rate.
The new design was a success. On one of our highest trafficked pages, these increases were significant wins for our users & for the company. The new design was more in line with our design system & had less dynamic text which made it easier for engineers to iterate on further. It was also flexible enough to accommodate the changes we had upcoming on the roadmap and fit our internationalization efforts better.
Learnings & Key Takeaways
The checkout page is deceptively complex. Every time an experiment was run, I designed for and helped QA 24-30 possible variations of this flow. I created & updated expansive copy docs of all the dynamic text options, heavy Figma files of the visual variations, & developed 4 guiding principles based on the learnings from our experiments. By the end, I was nicknamed “the checkout expert,” and gained valuable experience in documenting complex user flow variations.
More Case Studies
Designing a “Duplicate” feature →
I led the design process by validating the prioritization, creating the user flow, hi-fidelity designs, prototypes, & internal reviews. The internal tools feature had high adoption within just 4 days.
Running a Design Sprint to Deep Dive into Data →
I conducted a week-long design sprint to uncover opportunities to retain our subscribers, resulting in 4 projects validated & prioritized on our roadmap.
Landing Page re-design for SEO & conversion→
I combined user research with engineering & SEO requirements to solve user needs & business needs with a mobile optimized design that led to higher quality sign-ups.