Is the Shootout Really a Crapshoot?

The Flyers have been terrible in shootouts lately. And pretty much forever, really. So every time they lose one, people start venting in the game day thread, and it's almost inevitable that someone will point to an excellent Behind the Net article that argued that the shootout is a crapshoot. But is this really true?

Well, of course it is. As I've outlined before, pretty much everything has both a talent component (how often do you succeed) and a luck component (did this happen to be one of those times). Think about a guy shooting a free throw -- if he misses it, it doesn't necessarily mean he did something wrong or that he's bad at them; they just don't all go in and there isn't always an explanation for it. If he's a bad shooter, he's more likely to miss, but random chance is always a factor.

That's where sample size comes in. The more shots he takes, the more random chance evens out and the more meaningful our observation gets. After ten shots, you might make a tentative guess of whether he's any good. After 100, you'd feel better about it. After 1000, the luck has probably evened out and you know his talent pretty well.

Back to hockey. A shootout is decided after somewhere around three or four shots per team. That's not a very large sample size, and any given team can win any given shootout as a result. The real question isn't whether the shootout is a crapshoot; it's whether we're playing with loaded dice, whether some players or teams are legitimately better at shootouts than others.

The original Behind the Net article concluded "there is not enough evidence to suggest that individual skill levels make a difference in player shootout performance". The wording here is precise and important; it is easy to lose sight of the difference between "all players are the same" and "no player has yet had enough shots to really prove that he's better (or worse) than average."

That article looked at 5711 shootout attempts over five years and saw that Marc Denis was the only goalie who could be said with 99% confidence to be better than average and no goalie could be said with 99% confidence to be worse than average. After accounting for head-to-head matchups, no shooter could be said with 99% confidence to be better than average, and only a few appeared to be worse than average -- and even those players seem to have regressed to the mean since then:

The ten shooters who looked most clearly below average over the first five years after the lockout are collectively better than the league average (32.6%) since then. This makes a pretty compelling argument that even five years' data wasn't enough to tell us who the good shooters are.

But as always, a larger sample is better, so let's update to include data for all shots through February 9, 2012. Much like what schuckers showed in the Behind the Net article, here's a plot of save percentage versus shots faced -- the red line is the league average and we can say with 99% confidence that anyone on the blue line is better/worse than average.

With 91 goalies facing at least 14 shots (the minimum number required to prove you are better than average), we would expect to have one goalie on the blue line by random chance. As it happens, two goalies exceed that threshold and a few others are near it -- we still can't feel very confident that this is anything more than random chance, but the fact that more goalies are moving towards the lines as the dataset grows larger does suggest that there may be a talent here that we just haven't been able to measure yet.

Here's the corresponding plot for shooters:

With 181 shooters taking at least 14 shots, we would expect to have about two at the 99% threshold by random chance. Like with the goalies, we see a little movement towards the significance line with the new, larger data set, which I'd like to believe implies that there is some underlying talent that we just can't measure yet. But like with the goalies, the takeaway is that for the vast majority of players, we still can't even say whether they're better or worse than average, and we certainly can't show that any team has enough above (or below) average players to be significantly better (or worse) than 50/50.

In other words, if the dice are loaded, we can't tell.