“That Works” – SLG Research Methodologies

The Basics

The Top 20 methodology scores vendors on two dimensions: Reviews and Web Search Interest.

  1. The Reviews component considers the “overall” user rating (1-5 stars) of products based on reviews left in the last 24 months.
  2. The Web Search Interest component considers average monthly search volume for a standardized set of keywords for each product, as derived by a proprietary search methodology, as well as the position of the vendor’s domain on the search engine results page (SERP) for each keyword.

Each score is weighted and scaled to a value between 1-50. The two scores are combined into a total score (out of a total possible 100 points), which dictates the rank order of products in the report.

Data sources include approved user reviews, public data sources and data from technology vendors. Please refer to the “That Works” – SLG Community Guidelines for more information.

Inclusion Criteria

To be eligible for inclusion in a category, products must meet the following criteria:

  • Product has at least 20 unique product reviews published on “That Works” – SLG within 24 months of the start of the research process for a given report. Two years of reviews provides a sample that we have determined is large enough and recent enough to be valuable to buyers. The criteria is also set low enough to ensure emerging vendors can be represented.
  • Product achieves a minimum normalized overall user-review rating, after application of the Top 20 method for normalizing and weighting review recency (see description under “Scoring” section).
  • Product achieves a minimum web search interest score.
  • Product shows evidence of offering required functionality as demonstrated by publicly-available sources, such as the vendor’s website.
  • Product serves North American users, as demonstrated by product reviews submitted from that region.
  • Product is relevant to software buyers across industries or sectors—in other words, no “niche” solutions that cater exclusively to one specific type of user, as determined by our analysis of user reviews and/or market research.

Information for Vendors About Top 20

External Usage Guidelines

Providers must abide by “That Works” – SLG’s Top 20 External Usage Guidelines when referencing Top 20 content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all Top 20 reference(s) and graphic use:

The “That Works” – SLG Top 20 report constitutes the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, “That Works” – SLG or its affiliates.

Escalation Guidelines

We take the integrity of our research seriously. If you have questions or concerns about Top 20 content, the methodology, or other issues, you may contact the Top 20 team at digitalmarketsresearch@gartner.com. Please provide as much detail as you can, so we can understand the issues, review your concerns and take appropriate action.

Tentative Publication Schedule

Please visit our full publication schedule here. Note: This schedule is intended to be used as a guide only and is subject to change at any time at our discretion.

Leave a Comment

Your email address will not be published. Required fields are marked *