SP+ Prediction/Post-season Comparison

For all my analytics nerds out there, our season is over, so I thought it would be interesting to compare our post season standing against our preseason projections.

A quick definition for those unfamiliar:

In a single sentence, it's a tempo- and opponent-adjusted measure of college football efficiency. [Bill Connelly] created the system at Football Outsiders in 2008, and as [his] experience with both college football and its stats has grown, [he has] made quite a few tweaks to the system.

SP+ is indeed intended to be predictive and forward-facing. It is not a résumé ranking that gives credit for big wins or particularly brave scheduling -- no good predictive system is. It is simply a measure of the most sustainable and predictable aspects of football. If you're lucky or unimpressive in a win, your rating will probably fall. If you're strong and unlucky in a loss, it will probably rise.

Preseason Projection Current Standing
Overall 61 100
Offense 90 123
Defense 31 38
Special Teams N/A 75
Total Wins 6.6 3
Conference Wins 3.9 2

Here's a link to the 'final' preseason projections. And here is the link to the week 12 rankings. Obviously, other teams' seasons are not over, so our ranking may shift up or down, but nothing drastic.

Update to include some more miscellaneous tidbits:

  • If you're wondering, SP+ has picked the correct winner against the spread in 50.1% of FBS games this year.
  • VT is the third lowest ranked ACC team, in front of GT (106th overall, 105th offense, 73rd on defense) and BC (114th overall, 98th on offense, 95th on defense)
  • UVA is currently 98th overall, 115th on defense, and 45th on offense
  • Miami and Duke are 72nd and 57th, respectively. All other ACC teams are inside the top 50.
DISCLAIMER: Forum topics may not have been written or edited by The Key Play staff.

Comments

UVA is currently 98th overall, 115th on defense, and 45th on offense

so, in other words, the commonwealth cup would have been stoppable force vs moveable object

"Why gobble gobble chumps asks such good questions, I will never know." - TheFifthFuller

Years of being a VT fan has made me quite confident that having a terrible offense is worse because you can make even the most porous defenses' look competent.

A bad defense can get stops via turnovers and offensive miscues, including penalties and execution failures.

It probably would've been the sickos game of season

Twitter me

that offense...damn.
this is my go to meme for this season:

But anyway, that 6.6 wins jumps out at me. Given how we managed to snatch defeat from the jaws of victory against ODU, NCSU, and GT, that leaves a potential LOLUVa win to push us to 7 wins. So I guess he wasn't terribly far off

Warning: this post occasionally contains strong language (which may be unsuitable for children), unusual humor (which may be unsuitable for adults), and advanced mathematics (which may be unsuitable for liberal-arts majors)..

Given how we managed to snatch defeat from the jaws of victory against ODU, NCSU, and GT, that leaves a potential LOLUVa win to push us to 7 wins. So I guess he wasn't terribly far off

Exactly. If you go to the 'post game win expectancy' tab in the spreadsheet I linked, you'll see that VT had a 61% likelihood of beating NC St and a 75% chance of beating ODU. PGWE against GT was lower at ~35%. Had the UVA game happened, VT was projected to win 21-19.

Twitter me

Our pre-season ranking on offense was always suspiciously high to me. Supposedly it puts the most emphasis on returning passing game production and we almost literally returned none of it. Zero passing yards by current quarterbacks and Kaleb Smith the only receiver of note.

Supposedly it puts the most emphasis on returning passing game production

Going off memory, there is more emphasis on passing game production, but it's a difference of a few percent.

Twitter me

Either way we fell wayyyy short. Btw thank you for linking this spreadsheet again, I was scouring the internet for it.

I'm guessing it probably took into account Grant Wells' 2021 season where he was passing for ~300 yards a game. Plummeted all the way down to 197 this year.

Please forgive me if I am somehow not reading this right, but when it says the following, doesn't that mean that SP+ is completely worthless at picking winners of actual games? I mean a coin toss should get you the same statistical value.

"If you're wondering, SP+ has picked the correct winner against the spread in 50.1% of FBS games this year."

Recovering scientist working in business consulting

SP+ prediction is often around the spread. For example, if the spread is +3, and SP+ plus yields a predicted win of 3.1, it "picks" the favorite to cover the spread, but barely and without much confidence.

What would be more informative would be when SP+ predicts a final score that is a deviation from the spread by more than X points, how often is it correct? I think we would see it become more accurate the more deviated the prediction is from the spread, but idk. Definitely would be interesting to see how good of a predictor it can be when it suggests the spread is way off.

🦃 🦃 🦃

doesn't that mean that SP+ is completely worthless at picking winners of actual games?

Sigghhh... fair question. It depends how you define 'worthless'... Here's a stream of consciousness response that probably won't satisfy you:

  • SP+ should not be used (in a vacuum) to make bets
  • It doesn't account for underperforming players (eg; nagging injury), missing players (injuries, suspensions, etc), coaching changes, etc. It doesn't account for matchups (eg; why Bud Foster's defense caused issues for Urban Meyer's offense). It isolates the factors that frequently result in a win, and uses that information to predict how many points better than average a given team is. Going 50% ATS without any human intervention is actually pretty impressive.
  • Professional sports gamblers are aiming for ~53% ATS (more on that here).
  • What's interesting about SP+ is how it ranks teams: For example, Virginia Tech is the 100th best team, with a rating of -9.6. That means we would likely lose to the average (eg; 65th best team) by about 10 points on a neutral field. Iowa state is the 50th best team, with a rating of 6.1, which they would beat the 65th best team by about 6 points on a neutral field. Thus, you can conclude that Iowa State would beat us by about 16 points (15.9 points) on a neutral field.
  • This provides a really useful foundation for picking games - Again, I wouldn't use SP+ in a vacuum, but if you factor it into your own power rankings, and/or use it as a starting point, you might be able to more easily hit that 53% number over the long haul.
  • You can also use SP+ to observe trends and pick specific games. Bill C put out a stat in one of the previous offseason where he showed that SP+ is like 87% correct when it projects the underdog winning by a spread greater than Vegas thinks the favorite will win (I don't remember the exact stat, I'll have to look it up).

TL;DR - SP+ gives you a lot of good information, but I would not use it exclusively for picking against the spread.

Edit to address the final bulletpoint:

^This is where predictive analytics can be really helpful for gamblers - If you can find a small selection of games where the algorithm can outperform Vegas, you can hammer that specific segment and be successful. A good analogy is an online shopping website that addresses a niche need - will they ever make more money than Amazon? No. Can they go head-to-head in a very specific area and win? Yes. Predictive analytics can help you identify that niche market.

Twitter me

Thank you. I almost never gamble, so was not thinking of using it for that. Was just surprised to see no benefit. But then, I think I probably was misinterpreting it. It is helping to pick the winner of the game overall and not taking into account the point spread. So a team expected to win big in general will have that in this rating as well. Guess the way I would want to look at it should not be how well it predicts outcome against the spread (which takes into account 1 team being better than the other) but how well it picks the winner overall. With that, it should be a lot better. Just like saying if you always pick the team that is favored to win, they should usually win. Just that they may not win against the spread.

Sorry, think that was my misinterpretation.

Recovering scientist working in business consulting

It was a pretty reasonable question

Twitter me

Bill C's first point was I what I was trying to reach with my comment. SP+ will be more accurate against the spread when the spread is "more wrong" according to SP+. And it's nice to see the stat I was looking for, 2/3 would definitely yield a lot of winning in Vegas.

His betting formula is essentially:
Find the lines that deviate by more than 3 points from SP+. If that line is traditionally bettor friendly or less than three points, bet on it.

🦃 🦃 🦃

I laughed when I read that, too.

Most systems for picking Winners and Losers should regularly be batting well over .500, as there are certainly games on the schedule (like P5 vs FCS teams, and good P5 teams vs bad G5 teams) that are basically gimmes most of the time.

The spread is the handicap that gives all of the games coin flip odds- there is a 50% chance that Georgia will beat GT by 35.5 points or more on Saturday.

The gold standard of betting lines is the Vegas spread, aka "the line". The SP+ is about even with the line. That's pretty good (and probably won't make or lose you any money.)

Here's a thread on r/cfbanalysis if you're interested and a tracker for how the public prediction models fare:

Who has the best public prediction model in your opinion?

The Prediction Tracker