T.D. Smith: My Review Methodology

Update: I've moved all music review content to my new blog: music.sdtrv.com

Assigning a numeric value to the 'goodness' of something is a deceptively difficult task, yet many of us base decisions on values arbitrarily assigned by strangers. Whether we're deciding what movie to watch or what to buy on Amazon, we're strongly influenced by how many stars, thumbs up, or points a product was given.

Since I've decided to review things, I think it's important to lay out my methodology for assigning 'goodness' values.

Some assumptions:

1. Most things are close to average.
No matter what we're discussing, whether it's cargo jets or bowler hats, the majority of the offerings available are close to average as long as there's a big enough market and diverse supply.

2. Ratings are almost always subjective.
Unless you're doing a quantifiable test for battery life, durability, or other measurable factors, a rating for a product will depend on your personality, preferences, and experiences. My ratings will be primarily subjective. When I have objective numeric data, I will provide it as well.

How I'll do reviews:

a. I'll consider the product in the context of its rolling genre.
The concept of a 'genre' suffers from instability - genres are constantly evolving and changing. In order to ensure a large sample set, I'll evaluate products in the context of the genre to which they would have belonged 10-20 years ago. For example: Heavy metal, speed metal, nu-metal, power-metal, etc will all be evaluated in the context of 'metal'.

b. I will aim for a normal distribution of ratings within the context of each rolling genre.
This means I'll have many more average reviews than outstanding reviews, but items from different genres won't be competing with each other.

From wikipedia

That means if I were to review everything available in a given genre, most of my reviews would be between 4/10 and 6/10, with 1/10 and 10/10 being extremely rare.

c. I won't actually have time to review everything in a genre in long form.

I am going to be selective in what I take time to review long form. While I'm aiming for normal distribution of notional ratings, I won't actually write fifty times more average reviews than excellent reviews (because that would be boring).

I am aiming to provide fifty times more average reviews than excellent reviews, but I will only write long form reviews of products I find particularly interesting. For the others, I'll just provide a single rating value.

Why I'm doing it this way:

I want a number I assign to a product to have meaning. When I look at a review, I typically look at the numeric value before looking at the text. I want anyone reading my blog to get as much information as possible from this value. If I gave every album a 9 or a 10, that number loses much of its meaning.