In the past, I argued that the best way to measure special teams was to look at the shot rates -- how many shots does a power play generate and how many shots against does a penalty kill face?
Some new evidence has come out and it's clear that I jumped to a bad conclusion. Click through to the jump to see me admit I was wrong.
This started as a reply to chrislanci's comment:
The Flyers PK was ranked 15th this year. That isn’t elite PK so why bring back that same PK crew.
You may recall that earlier in the year, I was saying that the Flyers had the best PK unit in the league because they had the lowest shots against of any PK unit, and I believed shot rate was the dominant factor in predicting future special teams success.
I got sold on ranking special teams units on shot rate instead of on conversion percentage by this article, in which Hawerchuk showed two things: 1) that shot rate was much more reproducible than shooting percentage or conversion percentage, and 2) that if you want to predict a team's future conversion percentage, you will do better by looking at their current shot rate than looking at their current conversion percentage.
However, all of that was for the power play. New research has shown that the same thing isn't true for the penalty kill, and in this article I'll walk through the latest information.
Penalty kill save percentage turns out to be a lot more reproducible than power play shooting percentage -- teams have a lot more control over how many shots they stop than over how many of their shots go in. As a result, JLikens was able to show recently that this changes how we should look at penalty kill units. He found that if you take a random 41 games from a team's season and try to predict their PK% in the other 41 games, you do much better if you use PK% in the first set to make your guesses than if you use SOG/60.
In other words, teams don't have all that much control over their shooting percentage, so a good power play is one that generates a lot of shots. But teams do have control over save percentage, and just using shot rate to evaluate penalty kills leaves out the skill of the goalie, so you actually do better to use PK%.
But let's go back to the original question -- how good were the Flyers' PK skaters this year? Do they share any of the blame for the low save percentage?
There's been a lot of work on the impact of defense on save percentage at even strength and it looks like if there is an effect, it's a minor one, but it has been hypothesized that maybe they play more of a role on special teams. However, this week JLikens posted an article looking at that, too, and he finds that the team impact on save percentage seems to be even smaller on the penalty kill.
He looked at it two different ways, by comparing a goalie's save percentage with a new team to his performance with the old team and by comparing a starter to his backup. Following the goalie to a new team suggested there isn't any team effect at all, but comparing starter to backup suggested that there was but that it was even smaller than what's observed at even strength. So it's still not completely clear whether there is a team effect, but for these purposes it seems to be negligible -- it's a fair approximation to say for now that on the PK, the skaters need to stop shots from going on net and the goalie needs to save the ones that get through to him.
So here's what we're left with:
- A good power play is one that generates a lot of shots on net. They have very little control over which ones go in. So I stand by my previous claims that the Flyers power play was well and truly terrible this year.
- A good penalty kill unit is one that has a high penalty kill percentage. The variation across teams in save percentage is large enough -- and reproducible enough -- that just looking at shot rate alone doesn't give a good enough picture. So I wasn't quite right before in saying the Flyers have the best PK in the league.
- The penalty kill skaters control the shot rate but have a negligible effect on the save percentage.
- The Flyers this year gave up the fewest PK shots on goal, so the PK skaters were the best in the league. But they had a middling PK% because the goalies were subpar in stopping those shots, ranking 19th at 4-on-5 and 28th at 3-on-5.
- Since the Flyers goalies were solid at even strength -- where you see more shots and know the true talent better -- my hunch is that the PK save percentage will bounce back next year, but there's some cause to be nervous about it.