Before you use anyone’s predictions, it’s good to understand a little about the model creation process and performance on a held-out set of data. In this case, we did a full backtest. Let’s dive into it!
Current Limitations
Below is a list of the current known limitations in our model:
- The model does not know about significant roster changes that occur before the recent game.
Backtested Performance Week 6 to 21 From 2019
Let’s deep dive into our backtesting results to better understand how our predictions fared. When we created the model for predicting who will win games, we left out all data from week six and later from 2019 to evaluate its performance. The sample is small, but it’s important not to fly blind.
Overall Backtest Results
- Overall, the backtested model was correct 64 percent of the time.
- This is the current baseline for our V1 2020 model. Throughout the season, we will be making improvements by introducing new data and more advanced methods
Backtest by Binned Scored
Diving in further, we can evaluate how the model does when it strongly feels who will win. Let’s review some basic machine learning concepts!
- Classification models output a probability of an event occurring, in the case of this model, the likelihood is the home team winning. When that value is over 50%, we pick the home team; when it’s below 50%, we pick the away team.
- The closer to 1, the more the model likes the home team.
- The closer to 0, the more the model likes the away team.
The Pred_Bin column in the table below is the binned prediction score for the individual game. For example, the .6 bin means the model was between 50 and 60% confident the home team would win or lose. The prediction column will tell you which prediction was made. In general, we want the bins’ accuracy to match the bin value. .5 should win at a 50% rate, .6 should win at a 60% rate, etc.
In this view, the samples are small, but we see the higher value bins’ performance is better than the .4 and .5 bins.
- One thing to note - .4 is an outlier, and those are basically between .49 and .5.
If we remove the prediction (win/lose) from the column group, we see the above comments still hold and are easier to read. We also see better adherence to the accuracy within the bins.
<table> <thead> <tr> <th style="text-align:left;"> Pred\_Bin </th> <th style="text-align:right;"> Pct\_Correct </th> <th style="text-align:right;"> Total\_Games </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> .4 </td> <td style="text-align:right;"> 0.50 </td> <td style="text-align:right;"> 4 </td> </tr> <tr> <td style="text-align:left;"> .5 </td> <td style="text-align:right;"> 0.51 </td> <td style="text-align:right;"> 70 </td> </tr> <tr> <td style="text-align:left;"> .6 </td> <td style="text-align:right;"> 0.68 </td> <td style="text-align:right;"> 66 </td> </tr> <tr> <td style="text-align:left;"> .7 </td> <td style="text-align:right;"> 0.82 </td> <td style="text-align:right;"> 28 </td> </tr> <tr> <td style="text-align:left;"> .8 </td> <td style="text-align:right;"> 0.86 </td> <td style="text-align:right;"> 7 </td> </tr> </tbody> </table>Conclusion
When using our models to help you win a Pick’Em league, the stronger the model feels about an outcome, the better you should feel about the pick. We’ll improve the model throughout the season to hopefully make every pick a good one.
Make sure you remember the stochastic nature of football and how quickly a game can change for things that are not predictable (like an injury to a QB, RB, OL name) and its impact on the outcome.
Questions/Feedback
If anyone has any questions or wants to talk to us, you can find us on our Slack Channel. Click here to join: PLEASE CLICK ME, WE LOVE TO TALK.