Proabilites of NFL Home Team Covering The Spread- Results from The Backtest

Before you use anyone’s predictions, it’s good to understand a little about the model creation process and performance on a held-out set of data. In this case, we did a full backtest. Let’s dive into it!

Current Limitations

Below is a list of the current known limitations in our model:

• The model does not know about significant roster changes that occur before the recent game.

Backtested Performance Week 6 to 21 From 2019

Let’s deep dive into our backtesting results to better understand how our predictions fared. When we created the model for predicting who will cover the spread, we left out all data from week six and later from 2019 to evaluate its performance. The sample is small, but it’s important not to fly blind.

Overall Backtest Results

• Overall, the backtested model was correct 57% of the time.
• This is the current baseline for our V1 2020 model. Throughout the season, we will be making improvements by introducing new data and more advanced methods
• 57% is not excellent; with the juice in a spread bet, you need to win >55%, which does not give us much wiggle room.
• Let’s see if we can get better results based on the type of bet and the scoring bins

Backtest by Binned Scored

Diving in further, we can evaluate how the model does when it strongly feels who will win. Let’s review some basic machine learning concepts!

• Classification models output a probability of an event occurring; in this model, the likelihood is the home team winning. When that value is over 50%, we pick the home team; we like the away team when it’s below 50%.
• The closer to 1, the more the model likes the home team.
• The closer to 0, the more the model likes the away team.

The Pred_Bin column in the table below is the binned prediction score for the individual game. For example, the .6 bin means the model was between 50 and 60% confident the home team would win or lose. The prediction column will tell you which prediction was made. In general, we want the bins’ accuracy to match the bin value. .5 should win at a 50% rate, .6 should win at a 60% rate.

• This looks all over the place and reflects the overall score’s low performance.
• This is not surprising and what I’d expect from an initial model. The spread is designed to split the money down the middle and are created by smarter people than us. Ideally, we need to get the win rate to 60%, which is what I’ll be doing throughout the season.

One thing to note - .4 is an outlier, and those are basically between .49 and .5.

If we remove the prediction (win/lose) from the column group, we still see performance is all over. This table is not great to look at if you’re expecting to get rich off these predictions :).

Removing the .4 and .5 Bin

If we look at the overall performance if we remove the lowest two bins, the performance pushes us closer to that 60% mark.

Conclusion

When using our models to help inform any bets, be very careful. The performance is all over the place, and we will be working to improve these over the season. It’s good to keep in mind that this is my V1 model for the year and I will be investing lots of time and energy into making these better. Sports betting is tough, and from my experience with experiments last year, the week to week performance can be all over the place. For now, I’ll be using this mode to supplement other analyses and other expert opinions.

Finally, make sure you remember football’s stochastic nature and how quickly a game can change for things that are not predictable (like an injury to a QB, RB, OL name) and its impact on the outcome.

Questions/Feedback

If anyone has any questions or wants to talk to us, you can find us on our Slack Channel. Click here to join: PLEASE CLICK ME, WE LOVE TO TALK.