Steam News 2

User Reviews Revisited
16 MAR @ 5:47AM	- TOMB
Some time ago we made some changes to how we presented the User Reviews for games, and their resulting Review Score. We talked about those changes in this blog post. As we describe in that post, we want to ensure that players who've played a game can voice their opinions about why other people should or shouldn't buy the game, and that our summary of those opinions into a single Review Score should represent the likelihood that a future purchaser will be happy with their purchase.

Since that post, we've continued to listen to feedback from both players and developers. It's clear to us that players value reviews highly, and want us to ensure they're accurate and trustworthy. Developers understand that they're valuable to players, but want to feel like they're being treated fairly. We've also spent a bunch of time building analysis tools to help us better understand what's happening in the reviews across all titles on Steam. With that feedback and data in hand, we think we're ready to make another change.

That change can be described easily: we're going to identify off-topic review bombs, and remove them from the Review Score.

But while easy to say, it raises a bunch of questions, so let's dig into the details. First, what do we mean by an off-topic review bomb? As we defined back in our original post, a review bomb is where players post a large number of reviews in a short period of time, aimed at lowering the Review Score of a game. We define an off-topic review bomb as one where the focus of those reviews is on a topic that we consider unrelated to the likelihood that future purchasers will be happy if they buy the game, and hence not something that should be added to the Review Score.

Obviously, there's a grey area here, because there's a wide range of things that players care about. So how will we identify these off-topic review bombs? The first step is a tool we've built that identifies any anomalous review activity on all games on Steam in as close to real-time as possible. It doesn't know why a given game is receiving anomalous review activity, and it doesn't even try to figure that out. Instead, it notifies a team of people at Valve, who'll then go and investigate. We've already run our tool across the entire history of reviews on Steam, identifying many reasons why games have seen periods of anomalous review activity, and off-topic review bombs appear to only be a small number of them.

Once our team has identified that the anomalous activity is an off-topic review bomb, we'll mark the time period it encompasses and notify the developer. The reviews within that time period will then be removed from the Review Score calculation. As before, the reviews themselves are left untouched - if you want to dig into them to see if they're relevant to you, you'll still be able to do so. To help you do that, we've made it clear when you're looking at a store page where we've removed some reviews by default, and we've further improved the UI around anomalous review periods.

	Finally, we've also enabled you to opt out of this entirely, if that's your preference - there's now a checkbox in your Steam Store options where you can choose to have off-topic review bombs still included in all the Review Scores you see.

While we're working on some other features around User Reviews, we thought this one was worth shipping by itself. As always, if you have thoughts or concerns, feel free to voice them in the comments below.



Q&A

Q: I care about some things that I worry other players don't, like DRM or EULA changes. Review bombs have been about them in the past. Do you consider them unrelated or off-topic?

A: We had long debates about these two, and others like them. They're technically not a part of the game, but they are an issue for some players. In the end, we've decided to define them as off-topic review bombs. Our reasoning is that the "general" Steam player doesn't care as much about them, so the Review Score is more accurate if it doesn't contain them. In addition, we believe that players who do care about topics like DRM are often willing to dig a little deeper into games before purchasing - which is why we still keep all the reviews within the review bombs. It only takes a minute to dig into those reviews to see if the issue is something you care about.

Q: So if I post a review inside in the period of an off-topic review bomb, my review won't be included in the Review Score?

A: Unfortunately, this is correct. We've tested our process of identifying off-topic review bombs on the entire history of reviews on Steam, and in doing so, we've found that while we can look through reviews and community discussions to determine what's behind the review bomb, it isn't feasible for us to read every single review. But as we mentioned back in our first User Review post, our data shows us that review bombs tend to be temporary distortions, so we believe the Review Score will still be accurate, and other players will still be able to find and read your review within the period.

Q: Are you deleting reviews?

A: No, we are only changing the way we calculate a game's Review Score. All reviews are left untouched, and if you still want to see the raw Review Score, you're welcome to make that change in your Steam Store Preferences.

Q: Does this mean you'll be going back to identify and remove old off-topic review bombs?

A: Based on internal conversations and consultation with some of the partners that have experienced off-topic review bombs, we decided not to unilaterally grandfather in what's happened in the past.

Q: What about when I don't agree with you about what's an "off-topic review bomb?"

A: We'd suggest setting your Steam Reviews default to "Include reviews from all Steam purchases in Review Scores" in under the "Review Score Settings" in your Steam Store Preferences

Q: Reviews are extremely important to me: they're one of the few tools customers have to react to deception or discovering something they've been sold is sub-par. I guess that really wasn't a question, but I just wanted to say that.

A: We agree. We remain in active conversation with you, the community, about what you want from reviews along with the various partners who sell their games on Steam. Reviews are an important part of Steam, which is why we continue to do the work to make sure that they are not being manipulated by anyone. It's the same reason that we decisively ban partners who engage in review manipulation -- customers need to be able to trust the system for it be valuable.