What Makes for a Good RPG Review?

I enjoy and appreciate many of types of writing that are being pumped out by our nook of this hobby. Reviews of new products tend not to be one of them.

Yes, of course, there are good reviews, but for everyone of them I read four or five that leave me feeling a bit underwhelmed (and to be fair I include my own here). But I come here not to bitch and moan, dear readers, but to think out loud about some ways to improve. 

What are the features of a good review?

My own list is something like this:

Punchline. A good review answers, at the minimum, one simple, yet vital question: should I buy this? Our gaming world suffers from the same embarrassment of riches as we do about real-world information, a bewildering range of choices in our mediums. What makes a product worthy enough to be picked out of the crowd or mediocre enough to stay there?

A Sense of Audience. Related to the above question, a good review tailors that question to a sense of audience. An audience of DIY RPG kitbashers can be different in what it is looking from than simply a group of people who play older edition D&D. Those differences in expectations must be addressed.

Take James at Grognardia as a positive example (at least for how he delivers his ending punchline). He ends each review with a “Buy this is if you...” or a “Don't buy this is if you...” with a specific recommendation of a need/desire of a particular subset of his readers in mind.

Acknowledge Bias. An unfortunate by-product of our chummy inter-connected network of classic play/DIY RPG enthusiasts is that we tend to downplay the fact that we want to help promote our friends. By being upfront and honest we not only give fair warning to our readers but we start to develop the mindset that lead to the next point.

Be Critical. I have to admit I enjoy I love the blood sport of watching the occasional, mean-spirited thrashing of a crap product, but that is a different beast from the kind of constructive criticism that helps us all push forward. If you only hear praise of your virtues, you will never progress as a writer, game designer, artist, or whatever. Similarly, if you only hear what's good about others work you miss an opportunity.

Playtesting. Playing it before reviewing it is not always an option, but something that happens way too infrequently. A common pitfall of RPG products is that they may read beautifully on paper—this is especially true for the new school bias toward heavy plotting—but play terribly at the table. 

Think of the vast difference of quality of the reviews of say James Raggi's Death Frost Doom between those who only read it and those who played it in their game (and had both positive and negative experiences).

Broader theme. Not as essential as many of the points above, but something I always enjoy is the working in of a broader theme. A great review doesn't just discuss the particular product, but contextualizes it in something bigger. 

If it's a review of a campaign setting it compares the presentation to the historical arc of such books in the last few decades. If plays around with a non-linear plotline it contrasts it with the current love of adventure paths. Etc. etc.

Now back to you fine people. What do you look for in a review? What questions do you need answered? What features and themes do you like to see examined?