August is the wonderful time of the year when everybody knows that preseason deceives us, and nobody would fall for drawing strong conclusions from it. Well, that is until their favorite team takes the field or a top rookie has a strong showing on his first snaps with the pros…or…literally just when any preseason game is on. Oh yes, we love to discuss and conclude from preseason games.
We have seen anecdotes in both directions. As a fourth-round pick in 2016, Dak Prescott had a stellar preseason with a passing grade of 83.1 while passing for 8.5 yards per attempt, foreshadowing that he would go on to have the fourth-highest-graded rookie season of the decade, leading his Cowboys to a 13-3 record and the first seed in the NFC. On the flip side, Matt Ryan’s incredible run to the Super Bowl in the same year is well remembered, but only a few people recall that he suffered through an atrocious preseason, posting a 50.9 passing grade along with zero big-time throws and two turnover-worthy plays, leading to premature panic within the Falcons' fanbase.
Preseason performance generally isn’t predictive
How much does preseason really matter? To find out, we have to dig into all of the data. We first look at the offensive side of the ball and consider all players since 2013 who have played at least 20 preseason snaps and 100 regular-season snaps in one facet of the game (passing, rushing, route running, pass-blocking and run-blocking). We can compute the correlation coefficient between the preseason grade and the regular-season grade to obtain a measure of how good preseason performance predicts regular-season performance. The different facets of the game are differently stable over time, which is why we have to put those numbers into context. For that matter, we can consider all players with at least 120 regular-season snaps, split the snaps into two samples of the same size and compute the correlation coefficient between the grades in the different samples. Since there is no canonical choice for the split, we choose to split randomly and resample 200 times. The resulting correlation coefficients compare to the prediction from the preseason as follows.
For each facet, we can see that the predictive power of a part of the regular season is higher. But wait a minute, 50% of all regular-season snaps is usually a much higher sample than preseason snaps, right? That’s correct. Comparing the predictive power of roughly 40 preseason snaps to the predictive power of 200 regular-season snaps first and foremost tells us that 40 snaps aren’t enough to learn much about a player. While this is already a result worthwhile to keep in mind, it cannot tell us whether there is another more inherent reason for the preseason to be less predictive.
To account for this, we run another resampling in which we not only choose the splits but also the size of the splits randomly, corresponding to the preseason data. For any regular-season player in our data, we randomly pick one preseason player in our data, take the number of preseason snaps and then split the regular-season snaps into a sample of the same (small) size and all other snaps. This ensures that the predictions are based on the same sample sizes. The result is as follows.
We observe that the predictive power is decreased when lowering the sample size, but it is still significantly higher than the predictive power of the preseason. For blocking and route-running, every single instance of our resampling produced better predictions than the preseason — for passing this was the case 96% of the time. Rushing is an exception as for rushing grades, only 99 of our 200 resamples produced a better prediction than the preseason snaps of the same sample size, indicating that a running back rushing in the preseason has the same predictive power as a running back rushing in the regular season. This could stem from the fact that, as apparent from the chart above, the predictive power of rushing performance (especially compared to run-blocking) is low to begin with, an observation ultimately leading to the idea that running backs are replaceable.
We performed the same analysis for the defensive side of the ball and, as a surprise to no one, we found very similar results.
© 2024 PFF - all rights reserved.