The process of split testing is not as complicated as you may think. You contrive a specific goal, such as gaining more visitors onto another sub-page or generating more signups. Then, you create various alterations of said webpage and track analytics for which version performs better. These tests are often done privately with Internet marketing firms, but recently there’s been a lot more available data on blogs.
Today, I’ll share some case studies related to A/B testing for website performance. The results are quite insightful but remember that not every website is the same, because the audience and traffic sources generally differ in key areas. Each website should be examined as its own entity and user experience trends are greatly dependent on other elements in the page (navigation, content width, font size, etc).
So methods which work for another site may not work exactly for yours. However, the beauty of A/B testing is that you can always attempt new goals and determine what’s holding you back. Now, let’s take a look.
Recommended Reading: Guide To A/B Testing With Google Website Optimizer
Customer Testimonials and Sales
Here’s one of the cases on Visual Website Optimizer (VWO) and their A/B Split Testing blog. This work was done on Wikijob’s Aptitude Tests page. The goal was to improve conversions from visitors to the purchase page. The team worked on various implementations and finally settled with adding customer testimonials onto the page.
This increased product sales by 34%. There were no major design updates, color changes, or repositioned elements. Merely a new box underneath the heading with actual testimonials from past customers.
Goal: Improve the number of conversion from browsing around the site onto the final purchase page.
Conclusion
The value of peer review and social cognition is crucial. Just a few testimonials added into the page makes a world of difference. People visiting your website are interested in what other people think.
Lean & Slim Newsletters
The VWO blog has an excellent case study on lean newsletters and their benefits from higher clickthrough ratings. Based on Sheena Lyengar’s study on the demotivation of too many choices, users are busy and have too many things to choose from, hence they often end up choosing nothing at all.
The case study options were to limit each newsletter down to a single call-to-action button. This means readers would open the e-mail to see only a single link to one landing page or blog entry. The results showed a 17% increase of clickthrough rates from the e-mail onto the website.
People are more willing to click onto your webpage when your newsletter is offering only a single, straightforward option as opposed to a dynamically generated RSS newsletter.
Goal: Increase the clickthrough rate of e-mail newsletter subscribers.
Conclusion
Less is more. When your subscribers open an e-mail and there are 5 different sections of related articles, it can be overwhelming. By offering a single link back to the site it sends a clear, unwavering message and allows people to immerse themselves into the content.
Read Also: 9 Tricks To Design The Perfect HTML Newsletter
Masthead Image Backgrounds
The company OpenMile used VWO to test the masthead design on their landing page. Initially, their control had a large stock photograph of trucking, greatly related to their product. The new design features larger text and a deeper focus on the signup form with subsequent submit button. The goal was to measure conversions and see which variation performs better.
Generally, the updated version without a background image has pulled in more signups. This doesn’t mean a single masthead image will always perform poorly. But for this specific case study the designers believed a background image was too overwhelming and made the signup form appear off-putting.
You can read more about it here.
Goal: To increase the number of lead generations from the site’s landing page(s).
Conclusion
Masthead images work nicely when there is not much content other than text. Visitors are more likely to signup with a bright masthead, or by moving the signup form beneath the background image.
Product Listing Pages
This is a case study following the Vancouver 2010 Olympic store. The goal was to introduce a small bit of content into sidebars which would directly relate to visitors with the aim of improving sales and persuading visitors into making a purchase.
The control version had no sidebar content, variation A had a left sidebar with product categories, while Variation B had a right sidebar displaying “related products” from a recommendation engine. The last variation performed the best and produced the highest number of sales (7% more than Variation A). These results demonstrate how users are more interested in products which they can directly see right on the page.
Consider this for your own eCommerce webshop. The addition of a recommendation engine can help visitors buy related products with ease; as can extended thumbnail previews for each garment of clothing in a different color which users found helpful.
For more details, check out this post on Get Elastic.
Goal: To improve conversion rates from visitors into customers.
Conclusion
Offering related products will bring more attention to items which users may not have found otherwise. Just a small update to your eCommerce website layout can bring about a large increase in the number of product sales.
Videos in Landing Pages
A great blog post on the website Unbounce follows the use of videos in landing pages. This is from a startup named VidYard back when it was only a landing page. The tests had a signup form and a demo video which was embedded into the page, displayed in a lightbox, or not even displayed at all.
This method was a test into what video promotion can do for your product when users have no other alternative to learn about what you do.
The best results were with the video embedded into the page (up 69%) and even higher with the video in a lightbox (up 100%). The biggest point is to make sure that your video actually explains your product and how it can impact the user.
The best results found simplicity is the easiest way to get visitors interested. Make a simple video or animation that people can follow easily and present it in a clear manner on the page.
Goal: How does web video impact conversion rates on landing pages?
Conclusion
Both location and quality of video affects conversion rates. People are more willing to watch a video, if embedded into a lightbox. But they also want to be informed. This study found that bright animations and cartoons tend to keep people interested for longer periods of time.
Stimulating Webcopy
The testing for conversion lifts in user signups follows a website Bettingexpert, with tips and tricks for betting. The original signup form had very basic web copy and the team wanted to bring in more users onto the website.
The in-test actions and subsequent results are fascinating. There were no changes made to the website design itself, only updates to the form text. The question of “why should I fill out this form?” was not orginally answered. But updating the header text and the submit button value dramatically improved signup rates by 32% with a 99% confidence rating.
Just by looking at these two variations, it’s easy tell which one is better and more interesting, isn’t it? All webmasters should keep this in mind from the research done: “The copy you use in your signup form has direct and measurable effects on your conversion rate.”
Goal: Increasing the number of members/new signups.
Conclusion
With a clearer message in your webcopy, you can make a difference. By getting into your visitor’s head(s) and practicing different ideas for webpage copy, you’ll find a solution which is more direct and influential to the average person.
A Valentine’s Day Newsletter
Our next study investigates catching e-mail subscribers with an interest to purchase a certain product based on holidays. The Marine Mammal Center has an online gift shop, which handles a surge of purchases on various holidays like Mother’s Day and Valentine’s Day
During Valentine’s Day in 2013, the team sent out two variations of the same e-mail with different images for the product.
The test was to determine which photo would result in more purchases – the related baby seal, or a photo of the product itself. Naturally, the product photo got 6 more purchases and garnered 2 genuine donations from visitors. The product that they’re purchasing is more captivating and it’s no surprise as the results speak for themselves.
Read more about the study here.
Goal: Increase the number of purchases through various holiday e-mail newsletters.
Conclusion
People are more willing to spend cash on items if they can see exactly what it is. Imagery placed inside the e-mail is key. Holidays are also a period where sales tend to spike therefore it’s perhaps worth redesigning a custom layout for the more important holidays.
Primary and Secondary Form Inputs
Although this was recorded and published in August 2007, the results are still relevant today. The article discussing primary & secondary actions in web forms delves into a case study following user’s interactions with submit and cancel buttons.
The theory is that by keeping the buttons closer together users are less likely to get confused. Also that by having the cancel button appear smaller and less in the way, users would not accidentally click and erase all of their input data.
The team ran several tests over 20+ different people to get their opinion. There were seven form variations and options A, B, and C received the highest praises from users. Option B had the best results with nobody making any mistakes in the form.
Yet many users also confirmed that option A works well using a single text link for the cancel button. Additionally, options D & E only caused confusion. Unless your form is particularly aligned to the right, it’s best to keep these two buttons aligned on the left.
Goal: What is the best location for submit/cancel buttons in web forms?
Read Also: Login / Registration Form: Ideas And Beautiful Examples
Conclusion
Visitors will perform best using form interfaces which they are familiar with. Keep the submit button on the left side, and always the same size as Cancel (if you even need it). Don’t use alternate colors for each button. Overall common sense web design works the best when creating long forms with 10+ input fields.
Wireframe Usability Testing
This case study comes from, online testing company Loop11. It’s a wireframing case study that took place at a University in Sweden.
They made two distinct website wireframes for a tourist information webpage. Participants in the project were given either one of these layouts. Each person was asked to complete a number of 6 different tasks. These tasks included finding a city map, tourist attractions, language courses, and other typical website information.
They found that having two columns full of navigation links helped more people find what they needed. But this led to users generally taking longer to find what they needed. Potentially an issue with clutter or link text. But using these results and some other ideas provided a great starting point for the new website launch. Read more about it here.
Goal: How to structure a webpage layout for easy access to a lot of information?
Read Also: 30 Quick Ideas To Make Your Website Look Nicer
Conclusion
It’s best to provide 2 or 3 separate navigation menus if needed. Try to keep pages no further than 2-sub levels deep as this will lead them to have to dig around more. Other wireframe structures may perform better for other websites.
Further reading
Final Thoughts
Web designers who’ve never gone through the process of A/B split testing may find all of this information to be foreign and strange. But as you research more, it starts making sense. I hope that these case studies open doors for even further marketing research. My examples are only some of the best that I could find. If you know any interesting A/B testing ideas or similar links to case studies, why not share them in the comments below?
View the
Original article
Reading and browsing the web on your phone doesn't have to be a painful experience. / © AndroidPIT
Link Bubble pops up when you hit a link and loads in the background. You can then read or share the page. / © AndroidPIT/Chris Lacy
Not only is Javelin a nice clean browser, but Reading Mode (right) is a fantastic addition to web browsing. / © AndroidPIT/Steven Goh
Pocket is a nicely designed offline reading tool that also saves images with your content. / © AndroidPIT/Read It Later
Want to test drive the Galaxy Note 3 without commiting to it? Check out this new app. / © AndroidPIT
The Galaxy Note 3 experience on a Nexus 7. © AndroidPIT
