The Offbeat: A Quick Note on Individual Defensive Ratings

This is somewhat of a non-sequitur post, but the explanation of this is far too long to explain on Twitter, and I see the mistake often enough that I think it needs addressing, at the very least so I can link to it later.

The issue is Defensive Rating. Very often, in a discussion about a player's relative strengths and weaknesses, one wants to make a point about a defender's impact on the game, either in a positive or negative light. Naturally, stats are useful for this, as they tend to show that you're serious about what you're talking about. Furthermore, you might even have evidence for your assertions and are not just making stuff up as you go along. Clearly, the next logical step, is to use a defensive statistic to prove your point, and Defensive Rating is just sitting there on BasketballReference.com, calling you to it like Odysseus' Sirens.

Unfortunately, like those sirens, DRTG wants you to shipwreck. Here's is BasketballReference's glossary definition of the statistic:

Defensive Rating (available since the 1977-78 season in the NBA); for players and teams it is points allowed per 100 posessions. This rating was developed by Dean Oliver, author of Basketball on Paper. I will point you to Dean's book for complete details.

Okay, let's go to Oliver's book! Luckily, he has an entire appendix on how he calculates his DRTG, perfect for this explanation. His idea is sound:

The basic theory behind individual defensive ratings is that a player's defense impacts each possession to a limited degree, providing a base defensive rating, but that a player is more prominent in those possessions where he directly allows a score or contributes a stop, providing a modification to the base defensive rating. The formula reflecting this theory is:

DRtg = TMDRtg + %TMDPoss * [100 x DPtsPerScPoss * (1- Stop%) - TMDRTG]

Ack, a formula! Let's understand each of the individual terms. DRtg is the individual defensive rating that we're looking at. TMDRtg is the number of points the team allows per 100 possessions. %TMDPoss is a statistic of Oliver's creation: it reflects the percentage of defensive possessions on which the player is the primary defender. Stop% is the percentage of times the primary defensive player successfully forces a missed shot or a turnover, while DPtsPerScPoss is how many points, on average, the player allows when the shot is made (or a foul is committed).

In short, the formula takes the team's defensive points allowed, and makes an adjustment to it based on how often a player makes an impact, and how positive that impact is. In many ways, it's like a mix of team defensive ratings and Synergy's individual Points Per Possession allowed. This is looking good.

The problem, unfortunately, is that BasketballReference's DRTG uses only box score stats. That means we don't know %TMDPoss, DPtsPerScPoss, or Stop%, because NBA scorers don't track those things. We know, essentially, team DRTG, and that's it. So, significant estimations need to be made.

First, Oliver chooses to estimate that each player uses an equal amount of his team's possessions, i.e., 1/5 or 20%. Without any other data to go on, this is the best he can do.

Second, DPtsPerScPoss is simply set equal to be the team's defensive points divided by the scoring possessions, rather than being set on an individual level. This means that, for example, guards that give up a lot of three-pointers are not-adequately punished for their failed stops.

Third, Stop% must must be estimated. The main part of this estimation is given by the formula

Stops = STL + BLK + FMwt * (1 - 1.07 * DOR%) + DREB * (1 - FMwt)

Which basically totals the player's steals, blocks, and defensive rebounds, and discounts the defensive rebounds by the percentage of times a missed shot happens (about 55% of the time) divided by the percentage of defensive rebounds gathered (about 70% of the time). Yes, DRTG gives players defensive credit for defensive rebounds. In fact, the system gives them about 80% of the credit as it gives normal blocks or steals. That means that for most players, DRTG is heavily, even primarily, impacted by the number of defensive rebounds gathered.

Let's look at an example: the 2006-07 Jazz. Here are the Basketball Reference DRTG's of every player on that team:

The Jazz TMDRtg was 107 this season. Look who's at the top: Paul Millsap, Andrei Kirilenko, and then Carlos Boozer. Millsap finished 2nd on the team in defensive rebounds per 36 minutes, and added a pretty good number of steals and blocks to get the #1 title. Andrei Kirilenko contributed his usual assortment of pretty good rebounding, plus team-highs in blocks and steals. Carlos Boozer? He just defensive rebounded. That's it, but that contribution alone takes him to 3rd on the list. DRTG sees him as an above average defender, but most observers would probably agree that he was not.

On the other hand, this calculus hurts players like Ronnie Brewer, who played very good defense, but had only an above-average number of steals to show for it. While steals, blocks, and defensive rebounds are important indicative stats for a defender, having them (especially defensive rebounds) doesn't necessarily mean a positive defensive impact. BR's DRTG largely asks two questions: how good was your team overall on defense, and did how many steals, blocks, or rebounds did you get?

This isn't to say that all Defensive Rating statistics are bad: for example NBA.com's DefRtg simply keeps track of how many points per 100 possessions the player's team allows while they're on the floor. While that too is obviously skewed by lineups and other factors, it offers a very different picture than Basketball Reference's stats.

It's just something to keep in mind: while statistical analysis does provide a fantastic and objective way forward in analyzing the beautiful game, there are some potential pitfalls that can lead one to making some questionable conclusions.