Website Review: Evaluating "Good Guide" Ratings

The Good Guide is a good idea.  No, it's a great idea.  In case you aren't familiar with it, Good Guide is a website (and iPhone app) that lets you search for food, cosmetics, and other personal use items and get ratings on how healthy and green they are.  Their goals are to "provide the world's largest and most reliable source of information on the health, environmental, and social impacts of products and companies." If you've ever been in the cosmetics aisle, the food isle or in any store, and have found yourself wondering what the environmental impact and general 'green-ness' or nutrition value is of any product you have bought or are thinking of buying may be, Good Guide is the site (or app) for you.

But, I have to bring attention to a major usability issue I have with their site, and then subsequently an issue I discovered with their food ratings.  I follow Good Guide on Twitter, and today they posted the following Tweet:

Worst-rated energy and snack bars. Check your bar, get a better boost http://bit.ly/p1x5Y

I clicked on the link and I got this:



I've been on Good Guide before, but I was really confused when I looked at this list.  Was a higher rating good or bad?  Was the overall scale - 1 - 10 or 1 - 20?  Was a 4.2 rating that much better or worse than a 5.2 rating?   What did the colors of the rating circles mean?  This page gives me absolutely no clue even in simple terms how their rating system works or what the numbers mean. (I did crop these images, click here to see the actual page.)  This was partly because I'd gone to a page with the LOWEST rated items, so it was sorted from low to high, but it took me a few minutes to figure out what was going on. I definitely don't need lots of details on this page, but I do need a little bit of guidance about what the numbers mean so I can compare and contrast as I go down the list.

If you click into any of the items, you do get a lot more details and ways to discover what the ratings mean.  But most importantly, they have this little indicator which is super simple and explains a lot.

If they put this colored scale on the list page, it would  help explain what the numbers of the ratings mean.  A simple change would make a huge difference.  Another simple change would be to add numbers to the list to the left of each item that shows the number RANK of the item in the list. 

I found this other scale indicator at the bottom of the detailed info that would be even more helpful if they put it on the browse pages.   It is a very simple diagram that explains a lot.

Since I was on the site, I started taking a more detailed look around.  I had been on the site before, looking for information on how "green" a face lotion was that I had just bought.  But I hadn't looked at their food ratings.  After spending some time, I have to say that I have some real issues with the way their rating system seems to be working.  

Having designed online rating systems before, I know how difficult they can be to get right.   With any rating system, you have to find or develop your rating criteria and data, create an algorithm that assigns weight and relative value to each criteria, and then play around with it A LOT to see how it actually performs.

The Good Guide has created a very complex rating system based on many different sources of data.  They have 3 groups of criteria that they rate products by:  Health/Nutrition Performance, Environmental Performance, Social Performance.  They are fairly self explanatory - Health & Nutrition measures and rates how healthy and/or nutritious a product is, environmental performance measures and rates how environmentally responsible the company is that creates the product and social performance measures and rates how socially responsible a company is.  I think they've got the criteria and the groupings right (though I'm not sure of the QUALITY of their data which I don't have any insight into and is another issue entirely), and I think they are very valuable metrics for evaluating products that you buy.  My issue is with how their algorithm is working and how it is assigning nutrition values to products - especially to food products.

Let's stick with the energy and nutrition bars as the example.  I admit that I eat 'energy' bars occasionally.  They are tasty, convenient and can sometimes mean the difference between getting something to eat and nothing to eat.  But so many energy bars just seem like high-end candy bars to me so I was especially interested in this category.  I resorted the list to view the "Top Products" which looks like this:



Going down the list, it's looking like it's making sense until I get to item number 5: Snickers, Marathon Low Carb Energy Bar, Peanut Butter  which has a rating of 7.5 (at this point I've figured out that it's a scale of 1-10.)  This seems strange to me that, Snickers mostly know for candy bars, has an energy bar that scores so high.  I continue going down the list, looking for energy bars I've actually eaten, and to the next page.  The top item on the next page (number 12) is Odwalla, Super Protein Bar  with a 6.3 rating.  I'm familiar with Odwalla bars, having eaten several, so I'm very intrigued by why it scored so much lower than the Snickers bar.  I start doing some investigation by clicking into each detail page (a comparison feature would be really handy here - something I hope they are working on).

The detail page of each product is very detailed.  I think they do a good job of showing summary information with links to see more so you aren't totally overwhelmed with information.  The types of information they have make sense as well: the ratings across each category, some information that explains why the product got the rating it did with visual check marks or x's that clearly label something as a positive or a negative, a nutrition summary, the standard nutrition facts label and the helpful color grading scale with an indicator of where the product is on the scale.  So from a user experience/usability perspective, users can find the information they might be looking for easily.

But what I do have an issue is with how the ratings don't seem to accurately reflect the nutrition data that is displayed on this page.  According to the data, the Snickers bar gets a better score (it has a score of 10) than the Odwalla bar (which gets a 6.3)  Looking at the data, I assume this is because it is lower in sugar.  Both bars are fairly close when it comes to environmental and social performance (6.8 and 6.8 for Mars, the parent company that makes Snickers, and 7.0 and 5.5 for Coca-Cola (!!) the company that owns Odwalla) respectively.   It's true that I don't really want a high sugar energy/nutrition bar - because then it's just a candy bar, right?  Calorie-wise, they aren't far apart - Snickers has 160 calories while the Odwalla has 210. 

But where they really start to make a difference, and where I feel like the ratings are very inaccurate in summarizing which product is better than the other, is the ingredients list: 

Here's the ingredient list for the Snickers bar:
Quadratein (Soy Protein Isolate, Whey Protein Isolate, Peanut Flour, Calcium Caseinate), Polydextrose, Peanuts, Glycerin, Maltitol, Palm Kernel Oil, Tapioca Starch, Sugar, Salt, Skim Milk, Sorbitol, Sucralose, Natural Flavors, Vitamin and Minerals, Tricalcium Phosphate, Magnesium Oxide, Sodium Ascorbate, Niacinamide, Vitamin E Acetate, Biotin, Ferrous Fumarate, D-Calcium Pantothenate, Pyridoxine Hydrochloride, Vitamin A Palmitate, Zinc Oxide, Thiamin Mononitrate, Riboflavin, Cyanocobalamin, Folic Acid.

There are a lot of things on this ingredient list that I needed to look up: Quadratein  is a "patented new protein blend".  Polydextrose  is  a "multi-purpose food ingredient synthesized from dextrose, plus about 10 percent sorbitol and 1 percent citric acid."  There are also three different sugar substitutes: Sorbitol, Sucralose,  and Maltitol (a sugar substitute that can cause gastric issues) as well as some actual sugar.  A lot of the other science-y sounding ingredients are vitamins and minerals.

The Odwalla bar ingredients list looks like this:
Grape Juice Concentrate, Isolated Soy Protein, Raisins, Organic Soy Nuts, Date Puree, Brown Rice Syrup, Organic Soy Butter (Organic Soybeans, Organic Soy Oil, Sea Salt), Soy Protein Crisp Rice (Isolated Soy Protein, Rice Flour, Malt Extract), Vegetable Glycerin, Less Than 2% Of Dried Unsulfured Unsweetened Coconut, Organic Rolled Oats, Barley Flakes, Barley Malt Extract, Calcium Carbonate, Natural Flavor, Cinnamon, Soy Lecithin, Oat Fiber, Vitamin C (Ascorbic Acid), Baking Soda, Betacarotene, Vitamin E (D-Alpha Tocopheryl Acetate), Lemon Bioflavonoids, Green Tea Extract, Folic Acid, Niacinamide, D-Calcium Pantothenate, Vitamin B6 (Pyridoxine Hydrochloride), Riboflavin, Vitamin B1 (Thiamin Mononitrate), Biotin, Chromium Picolinate and Vitamin B12.

There isn't anything in this list that I need to look up, and no ac tual sugar.  I feel like I could take this ingredient list and actually make something similar in my own kitchen (without the added vitamins and minerals).  It also contains organic and the natural versions of items in the Snickers bar that are processed and created in a food lab.

I thought maybe the energy bars were an aberration.  But I looked at other categories of products and found the same thing: 

Cold Cereal:  4th on the list is Wheaties which has an overall score of 8.0 (10 H&N, 8.2 Enviro, 5.8 Social) while a similar yet organic and natural product Health Valley Amaranth Organic Flakes has an overall score of 7.2 (10 H&N, 5.5 Enviro, 6.0 social).  In this case, it seems like there simply is more environmental data available for General Mills (the makers of Wheaties) than for Hain (the parent company of Health Valley).   Wheaties ingredients include BHT, a synthetic antioxidant preservative and Trisodium Phosphate,  "a cleaning agent, food additive, stain remover and degreaser, once commonly used to prepare surfaces for painting."  The Amaranth flakes are totally natural and mostly organic.

Admittedly, the Good Guide is in beta still and they have a disclaimer at the top of their site: "GoodGuide is currently in early beta. Our site, ratings and data will change frequently. We welcome any feedback or ideas."  If the aim of Good Guide was just to provide comparative health and nutrition data, I wouldn't have as big of a problem with these ratings.  I do think that it is good that they have foods listed on the site that people usually see walking down their supermarket aisles.  But because they have a specific and stated goal of helping people make 'green' and environmentally responsible choices in the things they buy and the food they eat, they need to revisit the weights they've given to organic and natural ingredients and make them score higher than highly processed foods.  Until they make this change, I would highly suggest taking the number ratings given for food on the Good Guide with a big pinch of salt, and focus instead on the useful data they have compiled about ingredients.