Performance Of NFL Fantasy Football Projections in 2019

An essential part of machine learning and artificial intelligence is evaluating the results of your models. My goal here is to provide some standard metrics from our 2019 season that are used to assess our site. Below are the metric definitions for the evaluation. My commentary is at the bottom of the article. The current model should have ~68% of all predictions fall within the intervals

There will be three different sets of analysis:

1. Overall Performance
2. Performance By Position
3. Performance By Top 20 At Each Position

In addition to this analysis, you can view other model analysis here:

2019 Rookie Projection AnalysisEvaluation of our Weekly Projections Vs Other Sites

Metric Definition

• Total Predictions - Total predictions in the dataset
• Avg Interval - AVG difference between the Max and Min Predictions
• Total In Range - Total predictions where the actual points were between the Min and Max prediction
• Total in 5% - Total predictions where the mean projections were within 5% of the actual points
• In Range Rate = Percentage of projections where the real points was between the Min and Max prediction
• In 5 Pct Range - Percentage of predictions where the mean predictions were within 5% of the actual points
• Mean Absolute Error - The average absolute value of the error in our predictions vs. the actual points scored.

Overall Performance

Commentary

• 69% of all predictions fall within the min/max predictions each week which is where they should fall.
• It’s hard to gauge the total in 5% metric; this is a metric we created on a whim and decided to keep in this analysis. On the scale of fantasy football points, 5% is not a lot of points, if someone scored 10 points, we’d have to be within .5 pts of that prediction to have it right!
• A mean absolute error of 4.42 seems reasonable to me because if I am starting someone, knowing the average prediction will only be off +-5 would make me feel confident I am building/using the optimal lineup.

Position Performance

Commentary

• QB and RB have a significantly higher number of predictions within the the interval on a per rate basis.
• TE and WR are not quite to the 68% margin
• Overall, the mean absolute error is what I would expect
  • Intuitively, I would say that QBs would be the highest because they can spike points in a given game in either direction.
  • There are also significant QB errors when one gets injured in a game for both the starter and backup
  • TE’s generally score the lowest points so their error should be the lowest
• WR predictions perform the worst. The poor performance of the model might be a function of the amount of WR we are trying to predict every week
  • The Top 20 Performance section lends some support to this theory

Top 20 Performance

Commentary

• Unlike the positions overall, the top 20 have high in-range rates, which are likely driven by the large average intervals for those players
• The average errors are generally higher, and this is being driven by significant week to week performances the model has a hard time predicting
  • For example, the top 10 most substantial weekly errors we’re huge games from this group of players that were outside our confidence bands.
  • E.g., Will Fuller’s ~ 50 pt game
• The average interval for this subset of players are wider than their position as a whole, which reflects the sheer upside this group of players has on any given week.

Take Aways

After examining the data and writing the post, my key takeaways:

• In general, our average errors seem good overall with the top 20 players at each position having slightly larger errors than the rest.
• The top players have more substantial errors, likely due to the nature of their upside for massive points in any given week
• The prediction intervals we provide are performing as expected.

Questions/Feedback?

If anyone has any questions or wants to talk to us, you can find us in our Slack Channel. Click here to join: PLEASE CLICK ME, WE LOVE TO TALK.