MICROSOFT STORE - EXPERIMENATION
Objective: Increase revenue and sales throughout the Microsoft store e-commerce experience using quantitative testing.
Background
In 2016 I was asked to help form and grow the Experimentation Team at Microsoft. We were trained in best practices from top agencies and grew the program. We were then merged into Digital Stores and put in charge of optimizing the e-commerce experience and influence Microsoft culture toward more experimentation. We have not only been very successful at Microsoft; we are recognized across the industry as one of the top testing programs and our team is invited to speak at Experimentation conferences and webinars.
Objective
To optimize the Microsoft Store (e-commerce) site in key KPI such as orders, upsell, and cross-sell through experimentation.
Responsibilities
My role as a Senior UX Designer/Strategist on the Experimentation Team is to disrupt the Microsoft Store Marketing and Design Teams and pitch ideas to increase our key KPI's. I look at everything — improved design, flow, interaction, behavioral economics, and copy — and find opportunities to reduce friction and anxiety, and increase motivation. This leads to increased customer purchases via a great user experience. I use analytics and qualitative testing (elements I introduced to the team) to inform hypotheses. I also train design teams in qualitative and quantitative testing. All the ideas below are my own. Some of the designs are my own and others were created under my Art Direction by the Design teams that owned those areas.
Impact
Our team accounted for over +$206 million increase in Store revenue (extrapolation) for FY20. My hypotheses and tests are responsible for 62.6% of that revenue increase ($128.1 million).
Personalization & Targeting
Office standalone apps to trial
I was looking for ways to use Personalization and Targeting technology to increase Office sales. Data showed significant traffic coming to the Office Trail page from Office standalone apps product pages. I examined the flow and realized there were issues we could address and fix that would lead to increased attach rates and revenues.
HYPOTHESIS
If we personalize the banner based on the source of the referral, then the Trial rate will increase because users will understand that their app is included in Office.
Control
On the individual Office app product pages the “Try for free” link takes the user to the Microsoft 365 (the new name for Office 365). The user is strictly in the mindset that they want PowerPoint and they want to try it, with the expectation that their next action will get them that try. Instead they are dropped into a different product page with no context or clarity that they can only trial PowerPoint by trying Microsoft (Office) 365.
Variant
To reinforce to the user that the app they want is included with Microsoft 365 we included that app name in the headline, and made the App icon bigger. We did this for each standalone app (Word, PowerPoint, Excel, Outlook).
Items in cart
I was tasked with investigating ways of increasing orders, with a focus on cross-sell orders. My theory was that if we could remind customers of their cart content, and recommend complementary items, we would likely be able to positively impact order and cross-sell volume. Using cart content to encourage transaction completion, and to drive custom recommendations, would involve a significant development effort. We decided to start by first testing to confirm that reminding customers about cart content would measurably increase orders. The results exceeded all expectations.
HYPOTHESIS
If we remind customers what they have in the cart, then “Start Checkout” and order close rate will increase; reminding customers about what they wanted to buy helps “nudge” them along the process.
TEST 1- Module in page
LEARNINGS:
Data showed that most people that engaged with the new module were in the same session, whereas before they were reminded that they had items in their cart in a different session.
TEST 2- Introduce module across site, not just on the Home Page
The first test was highly successful but it was only introduced on the Home Pages. Most customers who were engaged with module were in the same session; we wanted to try having modules displayed across more of the site; this required designing different modules.
LEARNINGS:
We increased checkouts and closed orders significantly but also found that customers were checking out to soon and that the average order value went down. Also, the Design team didn’t want to use the “Top Bar” for any promotions other that those targeting Microsoft Edge browser.
TEST 3- Lessen footprint to diminish distraction and prevent decrease in average order value
After observing the results of Test 2, we designed a smaller modality to decrease customer distraction and increase items added to their cart. We tried one version that was generic (no programing would need to be implemented) and one version that showed a specific cart item.
LEARNINGS:
The version that was specific with copy and image generated less cognitive load on the user and that translated to increased movement through the sales funnel and actual closed orders.
Consistent Flow
Cart to Microsoft Sign-In Page
I observed that the flow from the Microsoft site to the Microsoft sign-in page and back to Checkout didn’t provide a consistent experience and was very jarring to the customer. The background and overall experience didn’t feel like the rest of the Microsoft site. We didn’t own and could not optimize the sign-in experience but I was able to identify key issues and possible solutions and we were able to test the flow with simple changes that still fit into the design template and required no engineering resources.
HYPOTHESIS
If we make the checkout flow look and feel consistent, then orders will increase because users will not have to wonder whether they are still on the Microsoft site and will feel more comfortable making a purchase.
Control
Variant
LEARNINGS:
Users needed to be reassured they are in the correct flow to purchase; simple background and copy changes resulted in significant revenue increases.
Increase Upsell
Surface Essential Bundles
Business goal is to sell Office with every device sold. To incentivize customers to buy Office with their Surface device purchase, the group created an “Essential” bundle that includes key items they think should come with every device purchase, including: keyboard ,Office 365, and 2-year protection plan.
HYPOTHESIS
If we describe and show what is included in the “Surface Essential Bundle”, then bundle orders will increase because the user will know upfront exactly what they are buying.
Original flow
Test series
TEST 1
Copy and location and test skipping Bundle PDP
Isolated copy changes to list what is included and switched the location of the offer from in the Buy Box to the promo blade. Variant 4 changed the CTA to “Build your bundle” vs. “Shop now” and skipped the Bundle PDP page (this was the major winner).
LEARNINGS:
The new copy (which listed what was included) in bottom location with the new CTA that skipped the PDP increased Add To Carts. However, Educational Discount dropped when moved out of the promo area. The big winner was skipping the Essentials Bundle PDP page with the new CTA of “Build your bundle”. The winning page had a lot of variables so we decided to isolate them in the next experiments.
TEST 2
Winning copy from Test 1 and isolate Call To Action
Took the winning copy from Test 1 and wanted to isolate the CTA change of “Build Your Bundle” vs. “Shop Now” in the Buy box only to see if we could get same results without dropping clicks to Educational discounts.
LEARNINGS:
“Build Your Bundle“ CTA was the winner
TEST 3
Winner of Test 2 against new module
With the winner from Test 2 implemented, we tested it against adding images in the Promo blade. I created this new pattern from what I saw on other e-commerce sites, understanding that the “X+Y+Z” pattern is a common practice and instantly recognizable as a “bundle and save” metaphor for customers.
LEARNINGS:
Showing what a customer gets in the bundle in the familiar “X+Y+Z” pattern increased Add To Carts. Less cognitive load for the user and easy to scan and decipher.
Final winning combination that was built for implementation
Leverage the new module for other scenarios
Leveraging the same module I created for the Surface Essential Bundle test, I wanted to test it against the Office stand-alone apps (PowerPoint, Word, Excel, and Outlook) to show that buying the full version of Office is a better deal for the customer. This supports the goal of the Business Unit which is to encourage customers to buy full Office instead of individual apps.
