Tuesday, March 22, 2011

What Makes for a Good RPG Review?

I enjoy and appreciate many of types of writing that are being pumped out by our nook of this hobby. Reviews of new products tend not to be one of them.

Yes, of course, there are good reviews, but for everyone of them I read four or five that leave me feeling a bit underwhelmed (and to be fair I include my own here). But I come here not to bitch and moan, dear readers, but to think out loud about some ways to improve. 

What are the features of a good review?

My own list is something like this:
Punchline. A good review answers, at the minimum, one simple, yet vital question: should I buy this? Our gaming world suffers from the same embarrassment of riches as we do about real-world information, a bewildering range of choices in our mediums. What makes a product worthy enough to be picked out of the crowd or mediocre enough to stay there?

A Sense of Audience. Related to the above question, a good review tailors that question to a sense of audience. An audience of DIY RPG kitbashers can be different in what it is looking from than simply a group of people who play older edition D&D. Those differences in expectations must be addressed.

Take James at Grognardia as a positive example (at least for how he delivers his ending punchline). He ends each review with a “Buy this is if you...” or a “Don't buy this is if you...” with a specific recommendation of a need/desire of a particular subset of his readers in mind.

Acknowledge Bias. An unfortunate by-product of our chummy inter-connected network of classic play/DIY RPG enthusiasts is that we tend to downplay the fact that we want to help promote our friends. By being upfront and honest we not only give fair warning to our readers but we start to develop the mindset that lead to the next point.

Be Critical. I have to admit I enjoy I love the blood sport of watching the occasional, mean-spirited thrashing of a crap product, but that is a different beast from the kind of constructive criticism that helps us all push forward. If you only hear praise of your virtues, you will never progress as a writer, game designer, artist, or whatever. Similarly, if you only hear what's good about others work you miss an opportunity.

Playtesting. Playing it before reviewing it is not always an option, but something that happens way too infrequently. A common pitfall of RPG products is that they may read beautifully on paper—this is especially true for the new school bias toward heavy plotting—but play terribly at the table. 

Think of the vast difference of quality of the reviews of say James Raggi's Death Frost Doom between those who only read it and those who played it in their game (and had both positive and negative experiences).

Broader theme. Not as essential as many of the points above, but something I always enjoy is the working in of a broader theme. A great review doesn't just discuss the particular product, but contextualizes it in something bigger. 

If it's a review of a campaign setting it compares the presentation to the historical arc of such books in the last few decades. If plays around with a non-linear plotline it contrasts it with the current love of adventure paths. Etc. etc.

Now back to you fine people. What do you look for in a review? What questions do you need answered? What features and themes do you like to see examined?


  1. I do like James' "Buy this if" approach. Unfortunately due to a variety of reasons I am not in a gaming group (but it looks like I might get one started soon) so ALL of my current reviews are based on how it reads or how it inspires me. Having said that, I think play test reviews are incredibly invaluable for the reason you touched upon with James Raggi's Death Frost Doom (I thought it incredibly evocative but I don't know how it plays). I'll probably be doing more reviews as I get my gaming group together. If that falls through, I'm growing my own crop of gamers that should be fully sprouted in the next year or two ;).

  2. Hah, I just wrote a crappy review today about the Castle Keeper's Guide...Chris, you and I are on the same page.

  3. Hi Chris, I've been trying to do a lot of reviews over at Dreams in the Lich House.

    I know you've mentioned the newspaper background, I'm wondering if you've seen any and would have a quick comment. Latest Review

    I'm thinking I could tighten up "the punchline" piece - I tend to rate commercial products compared to what I feel like I could do on my own as a home brew.

    I've avoided critical reviews, but I've bought some stinkers.

  4. Okay, giving feedback is pretty much my day job but my comment was getting long, so I think I'll make a post for my blog.

    Short version: I think "should you buy it?" is the wrong focus, give me enough information about what the product is and I'll make that decision myself, the reviewer doesn't need to.

  5. @Johnathan
    Yes obviously, given all of our gaming constraints as busy adults this is going to be one of the tougher things to do, but I think we should be striving to be this rigorous if we want to up our game in this area.

    We do indeed, fools rarely differ. I was just enjoying your crappy review. In fact, inspiration from this post came from reading Ken Rolston's review sections from the late 80s Dragon magazines you gave me.

    I tend myself to subscribe too much to the "If you don't have anything nice to say, say nothing at all" rule of my mother, the Southern Lady, when it comes to reviews inside our community.

    But I don't think this does anyone any favors: it doesn't alert readers to products they might not appreciate and it doesn't produce a vital piece of feedback for the designer.

    An old mentor of mine went on to become the lead food critic for an alternative weekly. She created a reputation for herself as a tough but fair critic. She was catty as all get out about restaurants that she disliked, but you knew if she recommended something that you could trust her review. (Years later the editorial policy of the weekly shifted and she was instructed to not write about a restaurant that she had a negative experience with.)

    I will boogie over to your site and take a look.

    I appreciate a punchline not so much that I need someone to make a decision for me, I will all on my own, but as a marker of what the bottom line is for the reviewer. It's kind of question that helps shape the clarity and coherence of the review.

  6. My needs begin and end with play-testing. Yes, an overview and all the necessary up-front information any good review should provide is important, but these I assume necessary to even bad reviews.

    When it comes down to it the most useful RPG product reviews for me aren't those where the author suggests to buy or not buy based on hypotheticals. Rather, its the rare review informed by the author's direct experience that is most valuable. Most of these aren't even presented as reviews in our world so much as play reports, rules discussions, etc...

  7. @James
    A good point about play reports and rules discussions being the best "reviews".

    Come to think of it I tend myself to rest my decisions more on them then many reviews. I am certainly always, always suspicious of new adventures, settings,rules sets, and the like that garner a wave of chummy reviews and then never (or at best rarely) generate a report of actual play.

  8. I've done pretty well by picking out things that I know I'll like. I don't purchase verything, but there is actually a lot of good stuff out there. So far from what I've seen the proportion of good gaming material being put out by the OSR/Old School Hobbyist DIY types is much higher than the stuff that was being put out during the d20 boom.

    I've read a few posts that have accused the OSR blogosphere of being a bunch of ass kissing fanboys. Well, from what I've seen so far, most reviews come from fellow creative types within the OSR. Sure, there may be some fanboyishness going on and self congratulation, but on the whole, I see folks really feeding off eachother to make more interesting and inspired material. Can we do better, sure; but really, who can't.

    I do think we need to be more critical where it counts in reviews. Constructive rather than bear baiting. I think there is enough negativity at the periphery of the hobby that we don't need to encourage just outright thrashing. I've don a few reviews and they have been on the rather high side rating wise. I've got a few that I'm going to do that will be a little less so, but I fully intend on pointing out why they didn't earn my highest rating. Well, that's my two cents at any rate ;).

  9. When I went with a really detailed & sophisticated line of critique in my review of LOTFP, people generally seemed to go; "meh"

    My conclusion is that as much as I would like to see such work, it is not generally desired by the community.


  10. Timely post--I'll be starting my own infrequent review column on my blog tomorrow. I've done a lot of thinking about what goes into making a good review, and I came to largely the same conclusions you did in your post. I imagine the experience of actually writing reviews will go a long way towards sharpening those ideas.

    I also agree with Telecanter--the review should speak for itself, rather than giving a thumbs up or down. But it's also important to acknowledge your biases and how you approach reviews in order to help the reader make up their mind about whether they'd like the product or not.

  11. @Johnathan
    I think I know the exact post you are talking about and found it pretty hyperbolic. In general I think we should aim for being supportive to our fellow hobbyists and respectful in our criticism.

    I missed your original post, a shame since I like your approach in it--though I don't think the need for thorough-going objective, detailed analysis was as much my point as it was we need to be a bit more thoughtful and honest in our reviews. I am fine with the subjective, visceral, off-the-cuff, and impressionistic being in reviews too as long as they are tempered with some insight.

    I don't want to get too much into a "review the reviewers" meta-narrative, but your review is much longer than what most people expect to read at one sitting in this medium. I find it difficult to read on a computer screen something over 1,000 words without my eyes--let alone my poor brain--getting very tired.

    I almost always notice a small down-tick in readership and comments when my own posts stray too long. (On the flip side breaking those posts up into more manageable chunks tends to create the opposite--as long as the series doesn't tarry too long itself.)

  12. What I'm looking for: What is the product about? It has to be short so that I can decide as quickly as possible whether I should continue reading. I want a verdict, and some supporting evidence. What's good about it and why do you think it is good? A note on the value/money aspect. I don't need a table of contents, a summary, chapter by chapter notes. I really prefer short reviews.

  13. This post 100% Shen Approved. Shen now copy and paste noble words of honorable beer-monkey into guideline for how shen to do reviews of Ahrpigees and suchlike things. Much-many thanks to you for expert guidance on how to do such things rightest. Save much smackings with sword-axes...