Product Page Usability Study Methodology

This study examines the user’s product page experience. It specifically looks at how sites can improve the product page design, structure, and features to ensure users are better equipped in evaluating and purchasing products. This study includes areas such the product page layout and structure, product images, the image gallery UI, ‘Added to Cart’, the user reviews UI, cross-sell logic, the structure of the product description and spec sheet, etc.

This study contains the research results from 2 years’ worth of large-scale e-commerce testing of only product pages by the Baymard Institute. More specifically, it’s based on:

  • In-lab qualitative usability testing following the “Think Aloud” protocol (1:1 moderated testing)
  • In-lab eye-tracking testing
  • A combined 236 test subjects/site sessions for the qualitative test sessions
  • Extensive UX benchmarking of the world’s 60 top-grossing e-commerce sites

Below the methodology for each of the research methods is described in detail.

To purchase access to the Product Page Usability Report & Benchmark go to: baymard.com/product-page

1:1 Moderated Qualitative ‘Think Aloud’ Testing

The 1:1 usability testing was conducted following the qualitative “Think Aloud” protocol. Each test subject was given 2–4 tasks, depending on the type of task and how fast the subject was. The duration of each subject’s test session was ~ 1 hour long, and the subjects were allowed breaks between each site tested.

The subjects mainly tested the product details page, and the 1:1 qualitative test sessions used a task design that facilitated comparison shopping across sites. Each task started by asking the subjects to choose between two product types, and pick the one they could relate to best. For example, asking subjects to choose between shopping for a grill or a fridge, or choosing between shopping for a backpack or a baby gift. After selecting a product type, they were sent to two different product pages, each from a different site (e.g., a gas grill at Lowe’s and another at Home Depot). The two products at each site weren’t identical, but very closely related substitutes, so subjects had to perform an actual comparison to determine which suited their needs the best.

The subjects were told to behave as they would on their own, including abandoning a site for a competitor and going off-site to search for information. When experiencing problems, the test subjects were asked open-ended questions such as What are you thinking right now?, Why did you click there?, What did you expect would happen?, etc.

If the test subjects got completely stuck, they were not helped at first. If they were still unable to proceed, they were given help to get past a specific problem, were given another task, or were directed to another test site for the same task. On such occasions, it was noted in the research log and counted as a “failed task.” A task was also recorded as failed if test subjects completely misinterpreted a product due to the product page layout or features; for example, if a subject ended up concluding they wanted product A because it was smaller than product B, when in fact the opposite was the case.

Any personal information submitted by accident has been edited out of the screenshots used in this report or replaced with dummy data.

The 12 sites tested in 1:1 qualitative “Think Aloud” were Lowe’s, Sears, Home Depot, Sephora, L’Occitane, Nordstrom, Patagonia, Crutchfield, Bose, KitchenAid, Cole Haan, and eBags.

The test sites were rotated between the test subjects to distribute subjects evenly on the tested sites. The compensation given was up to $100 in cash.

Eye-Tracking Testing

The eye-tracking test study included 32 participants using a Tobii eye-tracker, with a moderator present in the lab during the test sessions (for task and technical questions only), which took approx. 20–30 minutes. All eye-tracking test subjects tested 4 sites: Cabelas, REI, L.L.Bean, and AllPosters. The eye-tracking test sessions began by starting the test subjects at a product listing page and asking them to, for example, “find a pair of shoes you like in this list and buy it.”

The eye-tracking subjects were given the option to use either their personal information or a made-up ID handed on a slip of paper. Most opted for the made-up ID. Any personal information has been edited out of the screenshots used in this report or replaced with dummy data. The compensation given was up to $50 in cash.

PDP Benchmarking Methodology

The other part of this research study is a comprehensive usability benchmark. Using the 98 product page usability guidelines from the large-scale usability tests as the review heuristics and scoring parameters, we’ve subsequently benchmarked the product details pages at 60 top-grossing US and European e-commerce sites.

The UX performance score for each of the 98 guidelines is weighted based on its Frequency and Severity rating, and each site was graded on a 7 point scale, across all 98 guidelines. This has led to a comprehensive benchmark database with 5,700+ manually assigned and weighted product page UX performance scores, 4,100+ additional examples for the 98 guidelines, and 285 product page examples from top retailers, each annotated with review notes.

The total UX performance score assigned to each benchmarked site is essentially an expression of how good (or bad) a product page user experience a first-time user will have at the e-commerce site – based on the 98 guidelines documented in the Product Page Usability report.

The specific score is calculated using a weighted multi-parameter algorithm:

Below is a brief description of the main elements in the algorithm:

  • An individual guideline weight: A combination of the Severity of violating a specific guideline (either Harmful, which is the worst, Disruptive, or Interruption, as defined in the usability report), and the Frequency of occurrence of the specific guideline (i.e., how often the test subjects experienced it during the usability study).
  • A Rating describing to which degree a specific site adheres to each guideline (Adhered High, Adhered Low, Neutral, Issue resolved, Violated Low, Violated High, N/A).
  • The scores are summarized for all guidelines, and then divided by the total number of applicable guidelines (to ensure “N/A” does not influence the score).
  • The Highlights that include site screenshots are specific examples that the reviewer judged to be of interest to the reader. It’s the site’s overall adherence or violation of a guideline that is used to calculate the site’s usability score. Thus, you may find a specific Highlight that shows an example of how a site adheres to a guideline, even though that same site is scored to violate the guideline (typically because the site violates the guideline at another page), and vice versa.
  • Lastly, the score is normalized with a fixed multiplier for the benchmark study. The normalization is based on a “state of the art” implementation, to equal a total UX score of 1000. This normalization enables year-over-year and cross-study comparison.

All site reviews were conducted by Baymard employees in Q2 2017. All reviews were conducted as a new customer would experience them — hence no existing accounts or browsing history were used. Two product pages were chosen for each website review to act as the screenshot and UX-scoring foundation. The two products chosen were among the most featured and popular products on the website.

Other Research Studies

Besides these 3 main sources, select test observations and sessions that relate to the product details page are included from our other usability studies, primarily from Baymard’s studies on Mobile E-Commerce Usability, Product Lists & Filtering, and Homepage & Category.

Baymard Institute provides this information “as is”. It is based on the reviewer’s subjective judgement of each site at the time of testing and in relation to the documented guidelines. Baymard Institute can’t be held responsible for any kind of usage or correctness of the provided information.

The screenshots used may contain images and artwork that are both copyright and trademark protected by their respective owners. Baymard Institute does not claim to have ownership of the artwork that might be featured within these screenshots, and solely captures and stores the website screenshots in order to provide constructive review and feedback within the topic of web design and web usability.

Citations, images, and paraphrasing may only be published elsewhere in limited extent, and only if crediting “Product Page Usability study by Baymard Institute, baymard.com/product-page

Close overlay