Benchmark Methodology

An explanation of how the scores are calculated and the reviews conducted

Usability Score

The benchmark score is essentially an expression of how good (or bad) an experience a first time customer will have during checkout - based on the 63 e-commerce checkout usability guidelines documented in the E-Commerce Checkout Usability report.

The specific score is calculated using a weighted multi-parameter algorithm:

Methodology scoring equation

Below is a brief description of the main elements in the algorithm:

  • An individual guideline weight: A combination of the Severity of violating a specific guideline (either Harmful (worst), Disruptive or Interruption, as defined by the Checkout Usability Report), and the Frequency of occurrence of the specific guidelines (as the test subjects experienced it during the usability study).
  • A Rating describing to which degree a specific site’s overall checkout process adheres to each guideline (Adhered High, Adhered Low, Neutral, Issue resolved, Violated Low, Violated High, N/A).
  • The product is summarized for each guideline 1-63, and then divided by the total number of applicable guidelines (to ensure “N/A” does not influence the score).

The Highlights marked at the screenshots of a site’s checkout process are specific examples that the reviewer judged to be of interest to the reader. It is the site’s overall adherence or violation of a guideline that is used to calculate the site’s score - thus there can be a specific Highlight that shows an example of how a site adheres to a specific guideline, even though the site’s overall checkout process is registered and scored to violate the very same guideline (typically because the site violates the guideline in another checkout step), and vice versa.

Review Methodology

  • All reviews were performed by Christian Holst and Jamie Appleseed, from March 5. to April 8. 2012.
  • All the checkout reviews were performed as a new customer would experience them - hence no existing accounts were used. In general all required steps to complete a purchase for a first time visitor (when he has selected a product) was documented and benchmarked as the checkout process, including the “cart” step.
  • The shortest path through the checkout was always chosen. E.g. if there was a “guest checkout” option at the site this is the one documented and benchmarked.
  • All the checkout steps were reviewed based on the 63 e-commerce checkout usability guidelines documented in the E-commerce Checkout Usability Report.
  • A valid US address, phone number and credit card was used at all sites. The name, John Newman, is made up.
  • The checkout process is defined to start at the “Shopping Cart” step and proceed all the way to where the order is placed - typically an “Review Order”-step. This is the checkout process documented. (A few sites required account registration before the customer could even enter the homepage and browse products - on those sites these “pre-site account registration”-steps are counted and documented as a part of the checkout process as they are required for a first time visitor to complete a purchase, and because the account data is reused during the actual checkout process).
  • Occasionally the reviewer chose to create an account (in a separate checkout session) to further investigate if specific guidelines were broken or adhered to in the case it wasn’t clear during the first checkout review, but these steps aren’t counted as being a part of the documented checkout process.
  • Guideline #13 that relates to geo-targeting features weren’t tested at any of the sites as the target customer for each site might differ from the reviews' testing context. This guideline was therefore excluded from both the test, benchmark and scoring.

Legal

Baymard Institute provide this information “as is”. It is based on the reviewers subjective judgement of each site at the time of testing and in relation to the documented guidelines. Baymard Institute can’t be held responsible for any kind of usage or correctness of the provided information.

The screenshots used may contain images and artwork that are both copyright and trademark protected by their respective owners. Baymard Institute does not claim to have ownership of the artwork that might be featured within these screenshots, and solely capture and store the website screenshots in order to provide constructive review and feedback within the topic of web design and web usability.

Close overlay