An RAPM Model to Assess NBA Players

If you are like me, you watch NBA basketball games with a healthy mixture of enthusiasm for the game and skepticism of the media-driven narratives. In the last several years, I noticed a wide variety of advanced stats being thrown around by fans and media members alike, seemingly to justify their biased conclusions about players with highly subjective analysis. What’s more, these stats are frequently used as definitive arguments for end-of-season awards, such as MVP.

You hear things like: “Player X is clearly superior to Player Y because he ranks higher in categories A, B, and C.” Inevitably, you hear the response: “We all know stats A, B, and C don’t necessarily translate to winning games!”

Even with such advanced stats, there is sometimes widespread disagreement about a player’s impact or value. There are multiple issues at play with these stats:

  1. They are heavily influenced by simple box score stats, some of which have arguable relevance to a player’s overall impact.
  2. Many of them are generated using historical data, biasing more recent evaluations.
  3. They are often taken out of applicable context (perhaps due to lack of transparency).

Below, I will discuss my own attempt to “objectively” assess player impact by creating a RAPM model and compare those results with existing stats. However, some words are in order about the existing landscape of advanced stats.

Player Impact Metrics

To get a broad survey of existing methodologies, let’s take a look at just some of the most popular advanced player stats that exist (follow the links to get a more detailed description):

Such player impact stats can be roughly categorized as follows: purely box score–based stats, Statistical Plus-Minus (SPM) models, and actual Plus-Minus models (usually based on RAPM).

PER belongs to the first category. It is an all-in-one stat which uses a fixed formula to give a measure of player production from simple box score stats (e.g. points, rebounds, assists, turnovers, steals, blocks, personal fouls). While PER succeeds in combining box score stats into a single number, there are plenty of criticisms surrounding its methodology and biases.

BPM falls into the category of SPM models. While it is also based on box score stats, it is a regression against historical RAPM datasets to determine the coefficients which weight each feature (RAPM, a variant of plus-minus, will be discussed below). In other words, SPM models are linear estimators of RAPM.

It is easy to see why box score–based stats are limited; this can’t possibly encompass the entirety of a player’s impact! Consider some of the things not taken into account by simple box score stats:

We would like to capture a player’s intangibles, or anything that is not shown by the box score. Plus-Minus based stats may be used for this purpose, since they are based purely on overall team success when the player is on the floor. The advanced stats we have not yet discussed, RPM and PIPM, are both based on RAPM, which is a successor of Plus-Minus. Let’s take a deeper look at what this means, and why this technique is so prominent in basketball analytics.

What is Regularized Adjusted Plus-Minus?

The original Plus-Minus stat is simply the point differential a team experiences when a player is on the floor (i.e. points scored by the team minus points scored against the team). However, it is well-established that this metric is deceptive, since a player surrounded by highly skilled teammates will have an inflated Plus-Minus.

To answer this, an Adjusted Plus-Minus (APM) was created to account for these effects. This essentially means creating a matrix of n possessions by m players involved in all those possessions. We can set entries to if the player is on offense in the possession, if on defense, and if not on the floor. We can then formulate a matrix equation to solve for the coefficients that give the resulting point differentials :

We then approximate the coefficients by solving for the least-squares solution to the matrix equation. This yields values which actually represent the player contributions, removing the effect of teammates and opposing players.

These coefficients are the player APM values. However, because of multicollinearity, the results have enormous variance. Luckily, there is a statistical technique for dealing with this very issue: L2 regularization, or ridge regression. This is a well-known method for penalizing outlier values back to the central value, to reduce variance at the expense of bias.

Ridge regression is performed by modifying the least squares formulation with a diagonal perturbation matrix :

The value is a model hyperparameter that must be tuned to stabilize the coefficients. Note that in the limit that , we recover least squares; in the limit , shrinks to .

In this case, the resulting coefficients are the RAPM values, or the Regularized Adjusted Plus-Minus for each player. To be clear, these values are not absolute points added. This regularization technique results in coefficients that are scaled by a factor dependent on . The RAPM values should therefore be taken as a relative stat, or a differential contribution.

Circling back, how do our advanced stats incorporate RAPM? SPM models like BPM use RAPM strictly as an input to find the model coefficients. RPM and PIPM, on the other hand, are essentially “enhanced” RAPM models which use Bayesian priors to constrain the coefficients based on a fit to historical RAPM datasets. As we will discuss, this puts a potentially biased emphasis on box score stats.

Harvesting Matchups from Play-by-Play Data

In order to create my own RAPM model, I constructed a matrix as explained above, as well as the point differential resulting from each row (each matchup stint). I performed the following procedure:

To scrape the data, I used a Selenium web driver in Python to parse play-by-play data from NBA.com. These tables track the score and provide raw text data for analyzing the content of each play. In addition, the starting lineup for each quarter was fetched separately, allowing for player rotations to be tracked throughout a game. This is crucial for monitoring which lineups are out on the floor at any time.

The full scraping and wrangling codes can be found at my Github.

A figure with a caption
An example table of lineup matchup stints as a Pandas dataframe. Note that the raw point differential, number of possessions for each stint, and plus-minus per 100 possessions are shown.

Once we’ve isolated each matchup, we can parse the lineups into individual players for the home and away teams. Since we are taking the results of a stint, rather than single possessions, each team has been on both offense and defense. Therefore, we give home team players a matrix entry of , away team players an entry of , and players not on the floor an entry of . We then take to be the home team plus-minus per 100 possessions.

This can be done with a Python function as shown below:

def GenerateMatchupMatrix(df, player_list):
    matchup_plus_minus = []
    rapm_matrix = np.zeros(shape=(len(df),len(player_list)), dtype=float)
    idx=0
    
    for index, stint in df.iterrows():
        if idx%100 == 0:
            sys.stdout.write("\rOn Stint %i..." % (idx+1))
            sys.stdout.flush()
        
        matchup_plus_minus.append(stint.ht_pm_ph)

        f_ht_p, f_allp_ind, f_htl_ind = np.intersect1d(player_list, stint.ht_lineup.split(','), return_indices=True)
        rapm_matrix[idx,f_allp_ind] = 1
        f_vt_p, f_allp_ind, f_vtl_ind = np.intersect1d(player_list, stint.vt_lineup.split(','), return_indices=True)
        rapm_matrix[idx,f_allp_ind] = -1

        idx += 1

    print('\n Finished!')
    return rapm_matrix, matchup_plus_minus

Once we have and , we can build a ridge regression model to estimate the coefficients .

Developing a Ridge Regression Model

In order to optimize our ridge regression model, we will use the matchup matrix and matchup point differential as our training data and perform k-fold cross-validation to use the training data itself for validation. This is important, since we have only 3 seasons of NBA data (2017-2019).

In Python, we use the Ridge model from scikit-learn along with the GridSearchCV method to perform cross-validation on a comprehensive set of parameters (Note: in the Ridge model).

x_train = rapm_matrix
y_train = matchup_plus_minus

ridge_reg = Ridge()
alphas = [1, 10, 50, 100, 500, 1000, 1500, 2000, 3000, 5000]
param_grid = {'alpha': alphas}
grid_search = GridSearchCV(ridge_reg, param_grid, cv=3)
grid_search.fit(x_train, y_train)

print('Optimized hyperparameters:', grid_search.best_params_)

Output:
>> Optimized hyperparameters: {'alpha': 1500}

While k-fold cross-validation is typically used to set , we can also make sense of our choice by plotting the calculated RAPM as a function of .

It is clear that the values for small are highly unstable, as certain players experience a lot of variance. We can see that for our optimized parameter, the best– and worst–RAPM players are stable and begin to converge to the mean.

Plot showing the tuning of the ridge regression lambda parameter.
A plot showing the regularization of player RAPM ratings as a function of lambda. On the right, we see the tightening of the correlation between RAPM and raw player +/- as lambda is increased.

We therefore tune our hyperparameter with some degree of confidence. It is important to point out that our typical notion of out-of-sample model testing is compromised here; season-to-season variation in player performance is an unavoidable reality (for numerous reasons), and this model is meant to produce a descriptive stat rather than a predictive one. But more on that later.

Evaluating Player RAPM Ratings

Let’s take a look at how various players fare in RAPM.

Along with play-by-play data, individual stats were scraped for NBA players, including net rating, BPM, and RPM (again, see my Github). Note that net rating is actually just the player’s raw plus-minus per 100 possessions, which doesn’t take into account the caliber of a player’s supporting cast.

Below, we see the top 10 and bottom 10 players in RAPM (of players averaging minutes per game), along with these other metrics.

A figure with a caption

Stephen Curry has league-leading RAPM over the last 3 years, which is not outlandish given that he is a transformative player of this era. It is more interesting that the top 5 players are all guards – smaller players, 3 of whom are considered excellent 2-way players and all of whom can shoot 3-pointers and make plays for others. Notice also that some players, especially Damian Lillard and Jrue Holiday, are highly underrated by ESPN’s RPM.

If you are a basketball fan, I know you may scoff at the fact that Jrue Holiday is ranked “better” than LeBron James over a 3 year span. My response to this: the reasons you have for assuming LeBron James would have a better regular season RAPM may be very biased. If you take away LeBron James’s:

it is not so far-fetched. Think about it: Jrue Holiday is young, plays both ends of the floor, is an efficient scorer and play-maker, and has largely played a complementary role without an outstanding supporting cast. I’m not here to argue that Jrue Holiday is better than LeBron James in general, however the purpose of this exercise is to remove biased assumptions.

On the other hand, the players with the worst RAPM tend to be either aging, inexperienced, or considered lazy on at least one end of the floor. For instance, Trae Young is a rookie considered to be one of the worst defenders in the league.

An important note: I am by no means claiming these RAPM ratings are perfect or the end-all-be-all. These will of course have error, but still may provide some interesting unbiased perspective when evaluating players.

Comparing with Other Advanced Stats

To understand what we’ve calculated better, and how it relates to existing metrics, we can examine correlations with a variety of box score and advanced stats.

A useful tool for doing this is to make a Pearson correlation heatmap. This allows us to examine relationships between a large number of features and potentially informs us on what dependencies to examine further.

Plot showing Pearson correlation of RAPM with other advanced stats.

Shown in the matrix above are correlations between various player statistics averaged over the 2017-2019 seasons, including the previously discussed PER, BPM, RPM, and RAPM (within the green lines). The other abbreviated stats are: true shooting %, field goal attempts per 100 possessions, free throw rate, opponent field goal %, assist-to-turnover ratio, rebound rate, blocks per 100 poss., and steals per 100 poss.

We can summarize some interesting points from this table:

Taking all these observations together, it is reasonable to conclude that

  1. Other advanced stats appear to favor big men, including stats that are typical for big men (e.g. rebounds and blocks)
  2. ESPN’s RPM, one of the most popular and quoted advanced stats, appears to be highly dependent on box score stats

The explanation for this is that the RPM methodology uses box score priors calculated from a historical 15-year RAPM dataset (a cursory overview of such a technique is given in this article about PIPM). These Bayesian priors are then used in the ridge regression to constrain the range of for various players, thus pre-estimating their value using their box score stats. While these priors are proprietary and not publicly available, it appears that the box score component is significant (and even such factors as height and age are directly included).

Conclusions

At the beginning of this article, I provided 3 criticisms of current player impact metrics. Let’s discuss how my RAPM ratings address each:

  1. Box score dependence – there is none, not even assumed box score priors.
  2. Historical bias – RAPM in this context is calculated from only a 3-year dataset, providing sufficient statistics without biasing regression models to a completely different era of basketball.
  3. Context – as I will make clear, my RAPM model is not intended to be predictive, but is a fair assessment of player impact over the games included.

After comparing with these other metrics, let me define RAPM as follows: the relative ability of a player to positively influence possessions, regardless of their box score numbers, usage, or position. This is meant to be a descriptive stat over selected seasons, and thus no attempt is made to develop box score priors to improve the season-to-season accuracy.

This exercise to create a minimally-biased player ranking is informative when comparing to the publicly available advanced stats, because it shows just how subjective these models are. In Bayesian statistics, it is common to employ Bayesian priors to constrain results. However, the choice of these priors and selection of features is subjective and somewhat defeats the strength of the RAPM method: to measure a player’s overall impact and intangibles. If man-on-man defense is considered less valuable than a block, if a rebound is given more weight than a well-executed pass, and a taller player is given the edge over a shorter one, then our model loses its context-independence.

The true power of metrics like ESPN’s RPM is its predictive power. By regressing against a very large RAPM dataset to calculate priors, it can then predict a player’s future RAPM with better accuracy. However, RPM is often used as a descriptive stat to explain why a particular player has the edge in the current season, though RPM is tuned on box score stats from other seasons which don’t necessarily reflect their current impact. This inevitably leads to underestimating over-achievers and vice versa.

As a descriptive stat, pure RAPM models may still be the most objective. It may not accurately tell us what impact a player will have next season, but yet again, can any stat do that reliably given unavoidable year-to-year variations (coaching change, team change, injuries, etc.)? No all-in-one stat gives a perfect player assessment, however we must be careful about how we apply any of these advanced stats and clearly state our assumptions.