Evaluating Models (General Thoughts)
When we build models, we apply standard evaluation metrics against a hold-out set of data to evaluate if it’s performing well. Also, we have developed non-standard methods that apply directly to models created for fantasy sports (those methods are internal intellectual property so I can’t share).
In the context of the rookie model, we randomly select a percentage of all rookies. The model is never trained on this data. Below is a table of the actual fantasy points scored in the players rookie season.
My Opinion on Transparency
Throughout the season, I generate a weekly postmortem (when I have time) so I can address both the successes and shortcomings of our model for the previous week.
My goal of those posts isn’t to cover up the poor performance or overhype successes. I try to highlight both. We at Gridiron feel our process is reliable and the results, throughout a whole season, will be better than other sites or just using one single projection. There is going to be a week to week variance in the model performance. Football is a sport that has lots of randomness in it.
I am always on our slack channel to answer questions about the current projections (link is at the bottom of this post).
Details for the Tables Below
pts season, pts min, pts max, pts act, game played count are all fields predicted from our model
game played count actual, pts act are actual values from the season
PPR (Including QBs because their scoring doesn’t change)
- All values are within the range of projections we output.
That sentence sounds good, but in reality, the scoring ranges (min and max) are so high, it’s surprising if they weren’t!
D.J Moore and Lamar Jackson and Nyheim Hines are the examples I am going to focus on but overall, the model does well predicting the games played and the points the player would score
In general, I feel playing time is the most significant factor leading to how the model will perform. I use the projections to figure out who has the most upside and draft on the assumption the best players will have the best return during the season.
D.J Moore - We ended up being ~30 points off D.J Moore’s final score last season. The model only expected him to play in 14 games and he played 16 games. If we scaled the projections to a whole 16 game season, we would have had him at 137, which is pretty darn close!
Lamar Jackson - Another case like D.J. Moore where we were really close with the playing time being the most significant factor that would have driven that variance.
Nyheim Hines - We were way off on the playing time. IF we scale the projections to the whole season, we ended up being at 150 points, which is ~27 pts off his actual total.
Standard
I am not adding analysis here. See the above section for my commentary.
<table> <thead> <tr> <th> name </th> <th> pts season </th> <th> pts min </th> <th> pts max </th> <th> pts act </th> <th> game played count </th> <th> game played count actual </th> </tr> </thead> <tbody> <tr> <td> Lamar Jackson </td> <td> 218 </td> <td> 52 </td> <td> 272 </td> <td> 174 </td> <td> 12 </td> <td> 15 </td> </tr> <tr> <td> D.J. Moore </td> <td> 84 </td> <td> 15 </td> <td> 182 </td> <td> 101 </td> <td> 14 </td> <td> 16 </td> </tr> <tr> <td> DJ Chark </td> <td> 78 </td> <td> 0 </td> <td> 168 </td> <td> 15 </td> <td> 12 </td> <td> 11 </td> </tr> <tr> <td> Nyheim Hines </td> <td> 58 </td> <td> 0 </td> <td> 139 </td> <td> 98 </td> <td> 8 </td> <td> 16 </td> </tr> <tr> <td> Keke Coutee </td> <td> 54 </td> <td> 0 </td> <td> 140 </td> <td> 52 </td> <td> 9 </td> <td> 6 </td> </tr> <tr> <td> Jordan Akins </td> <td> 53 </td> <td> 0 </td> <td> 86 </td> <td> 23 </td> <td> 9 </td> <td> 14 </td> </tr> <tr> <td> Marcell Ateman </td> <td> 38 </td> <td> 0 </td> <td> 126 </td> <td> 21 </td> <td> 7 </td> <td> 7 </td> </tr> <tr> <td> Auden Tate </td> <td> 33 </td> <td> 0 </td> <td> 123 </td> <td> 4 </td> <td> 7 </td> <td> 5 </td> </tr> <tr> <td> Jordan Thomas </td> <td> 20 </td> <td> 0 </td> <td> 63 </td> <td> 46 </td> <td> 7 </td> <td> 14 </td> </tr> </tbody> </table>Questions/Feedback
If anyone has any questions or wants to talk to us, you can find us on in our Slack Channel. Click here to join: PLEASE CLICK ME, WE LOVE TO TALK.