A (vain?) run at goalie consistency

So, I'm an engineer, not a statistician - there are some pretty obvious statistical whoopsies in my analysis (averaging averages, for instance, is a no-no). But in the search for SOME basis of comparison between Goalie A (let's call him Andre) and Goalie B (let's call this one Bryzgalov), I attempted to look at variability in save percentage, both across the league averages and for a given goaltender game-to-game. Basically, Mr. Detweiler and I came to an impasse over the existence of a more consistent goalie, and whether such a thing would be better or worse than a goalie who has a so-called 'hot streak'.

Hopefully, what I have found will make some sense to someone ... because I'm not honestly sure what the numbers are telling me now. (I'm pretty certain the request for a PCOM hoagie from Koch's came from my stomach and not the numbers...) I started the project with the theory that Bryzgalov was a highly variable goaltender - on any given night, you didn't know if he was gonna be unbreakable or completely porous. My contention was that a goalie that gave you a chance to win every night would then be a better choice, and that such a goalie would have a save percentage close to the league average (defining close as one standard deviation from the league average).

I feel I should first apologize - I have no clue how to insert graphs or pictures into this format, so I'm gonna have to retype everything that seems important. If that bores you, I suggest you skip to the last paragraph and begin your unsubstantiated narrative from there with the inherent understanding that the rabid breed of statheads that populate this site will likely hunt you down and eat your liver. If you're OK with my status as a techno-tard, please continue into stultifying numerical miasma.

The first step I undertook was to look at the league averages in save percentage. To easily limit the field to starting goalies, I only took the top 30 goalies in games played. Then I averaged their save percentages, calculated the standard devation from that average, and that defined the range of goalies one would consider at least 'average' in the league. Turns out that the league average save percentage is pretty much the same this year (0.915) as it was the previous two seasons (both 0.914), with a slight dip in 2008-09 to 0.911 - so it seems that in the recent past an average goalie should be expected to post a season-average save percentage of 0.914 or so. In all four recent seasons, the standard deviation among the top 30 starting goalies in save percentage was around 0.010.

This led to a definition - an above-average goalie would post a season save percentage one standard deviation above the league average. So for the past four years, any goalie with a season average over 0.924 would be considered pretty damn good. This season, there are 6 of the top 30 goalies (by starts; remember, no judgment here) with a save percentage over 0.924 (Lundqvist, Rinne, Quick, Smith, Lehtonen, and Halak) - and Lundqvist is having a stellar year with his 0.935 Sv% exceeding TWO standard deviations over the average. This sort of distribution holds pretty true over the previous two seasons: in 2010-11, 3 goalies exceeded the 'above-average' mark (Rinne, Luongo, and of course Thomas); in 2009-10, 5 goalies were above one standard deviation (Rask, Halak, Vokoun, Howard, and Miller). Just considering the names mentioned, the only repeated names on the above-average list in the past three seasons would be Rinne and Halak ... does this denote they're the best around now? Going back another year doesn't help much - Halak and Rinne both exceeded the league average in 2008-09, but by less than one standard deviation (Thomas, Vokoun, and Backstrom were over). So maybe Rinne, Thomas, Vokoun (sound familiar, Flyers fans?) or Halak are the 'best' goalies around. Well, we shall see.

Obviously, the inverse definition would hold as well; any goalie with a season-average save percentage less than one standard deviation below the league average would be a below-average goalie. Thus far this year, there are only 4 such goalies below a 0.907 (Mason, Gustavsson, Garon, and Crawford), and Steve Mason holds the unenviable position of being the sole starting goalie below two standard deviations less than league average. Similar incidences occur over the previous seasons - each season has 5 goalies at less than one standard deviation below average, and one poor sap who was past two standard deviations under. (Just to touch another narrative - in 2008-09 that poor sap was Chris Osgood.)

SO - we've pretty well established now that the league average save percentage is about 0.914, and that there are consistent distributions of goaltenders above and below that mark each year - but not necessarily the same goalies. That leads to the next evaluative process - comparing a particular goalie to the league average. Since the league average and standard deviation is relatively consistent over the past 4 seasons, looking at a goalie's save percentage in any year recently compared to the average here of 0.914 with a standard deviation of 0.010 should be a pretty valid comparison. So let's look at some not-so-randomly selected goaltenders and see which are above, below, or just good ole average.

How about a household name ... as long as your house is in Calgary. Miikka Kiprusoff has been one of the league leaders in games played since the 2005 season, so he's pretty durable. Calgary saw fit to toss \$5,833,333 worth of cap salary at him, so it would seem they value his contribution pretty highly. But should they? Let's see - in 2008-09, Kipper's save percentage was 0.903. That's just barely in the 'average' range for that season, and mildly under our one standard deviation range from the four year look. But the next season in 2009-10, he bounced back and posted a 0.920, which is at the very top of the 'average' range but not quite above one standard deviation. Then in 2010-11 he 'regressed' again, with a 0.906 that is once again a low average goalie. And then thus far this season Kiprusoff has maintained a 0.921 - once again at the high side of 'average'. So each season over the past four, Kipper has been an 'average' goaltender - even while some years in the lower end and some in the higher end. (To be fair to Calgary's front office, Miikka ran up save percentages of 0.923, 0.933, 0.917, and 0.906 in the previous years to his contract signing in 2008, but we're not looking that far back.)

So who has been an above-average goaltender consistently in the past four years? As we saw already, the list of repeat performers includes only Rinne, Halak, Thomas, and Vokoun. So let's check them out. Rinne tacked up save percentages of 0.917, 0.911, 0.930, and 0.925 - two average years and two above-average years. That's pretty damn good. Halak, in slightly lighter duty overall, has put up save percentages of 0.915, 0.924, 0.910, and 0.928; again, two average seasons and two above the average. Timmy 'Tubby' Thomas placed 0.933, 0.915, 0.938, and 0.918 - two seasons of average goaltending, and two positively stellar seasons scoring two standard deviations above the average. That's damn impressive, considering that each season there is only one goalie in the league who exceeds the two standard deviations mark - and half of the recent years, it's been Thomas. Lastly, there's Tomas Vokoun. Look at the man's career - he was posting the same damn save percentages in 1998 as he is today, and he is VERY consistent season-to-season. In fact, over the past seven (7) seasons, he has been in the top end of the average range or better; in his last four he has 0.926, 0.925, 0.922, and 0.917. I think these numbers definitely show that all of these goalies are above-average.

Let's choose a goalie one may not think of readily as 'elite' ... or even average - Mike Smith of the oft-ridiculed 'Yotes. He's currently posting an above-average save percentage this season of 0.924. (I won't digress here about how the Phoenix system may have inflated Bryzgalov's numbers.) Smith's previous seasons were, shall we say, less stellar. In intermittent play with the Bolts he notched the following: 0.916, 0.900, 0.899. For those following along, you can readily see that two of those seasons are 'below-average' by at least one standard deviation. (And looking further, there's some serious mediocrity.) Does this mean the Bolts defense sucked? Perhaps - but it definitely shows that Smith is no Vokoun (who posted insane numbers on some rotten teams over the years).

Let's choose another goalie - one that you may not readily think of at all (but you should): Craig Anderson, currently with Ottawa. Craig's current 0.913 places him squarely at the middle of the 'average' goalie pack this season ... and his aggregate between the Sens and 'Lanche in 2010-11 is average as well at 0.910. In 2009-10 he made about 0.917 of his saves, and in 2008-09 he was up at 0.924 in a relief role. Anderson is the definition of an average goalie in a given season - and his status as the 20th-highest paid goalie accurately reflects that.

Another look at season averages - Nicklas Backstrom. Most people would list him as a top-level goalie ... but it seems he's not. See, in the past four seasons, Nick has only made one season's worth of 'above average' saves. The other three years have been relatively pedestrian - 0.920 (OK that's still upper-average), 0.916, and 0.903. That last one is actually a below-average save percentage in 2009-10! In all fairness, Backstrom also posted a 0.920 and a 0.929 in the preceeding seasons to our four-year sample, so maybe he's allowed one stinker, eh?

Obviously, this isn't complete without including your favorite and not mine, Ilya Bryzgalov. Bryz is putting up a 0.909 thus far this season - which is average, but in the lower half thereof. In his previous three seasons, Bryz saved 0.921, 0.920, and 0.906 - so two upper-end average and one lower-end average, but no above-average seasons. IN fact, Bryzgalov has yet to have any season in his career (since 2005) with a save percentage that we would find to be 'above average' by our one standard deviation criteria.

In summary for this section, the league average really doesn't change much from season to season (recently - not accounting for the rules changes or anything else). So one would expect goalies to be pretty consistent as well ... except they aren't. Some are pretty consistently good, like Vokoun or Rinne. Some are wildly inconsistent, with good years and bad, like Backstrom and Smith. And some are just average - like Bryzgalov and Anderson.

So of course, now that you're all asleep and drooling on the keyboard, I intend to make one last series of comparisons. Perhaps, one says, in a given season an 'average' goalie can go on a 'hot streak' and play like mad - and that's what you want for the playoffs. Nonsense, I say - you want a goalie your team is comfortable with and gives them a due chance to win every night, night in and night out. ("What about day games," says that smartass Geoff, but we all ignore him, don't we?) So what's the deal? How far does a goalie deviate over the course of a season, and is there any correlation between that and a winning percentage?

Let's start with a goalie we all agree is pretty good, with above-average save percentages and pretty consistent about it: Pekka Rinne. Rinne has a 0.925 thus far this season - just above one standard deviation higher than average, or 'above average'. But how does that look per game? Well, because his average is so high, it does skew things a bit. His personal season standard deviation is 0.075 ... which means he needs to get a shutout (1.000) to be one standard deviation over his average. It also means his one standard deviation under (below average) is a still-respectable 0.850. Rinne thus far this season has had 4 shutouts (so 4 above-average games) and 5 appearances with less than a 0.850 (below-average games). His 63 total appearances include 39 wins, for an overall 62% winning percentage - of which only one was a below-average game for Rinne. Of his losses (including OT), 6 were still above-average for the league save percentage (his personal above-average is a shutout remember, so a bit unfair). It is pretty safe to conclude that Rinne carries his team, every day.

In contrast, we have Tubby Thomas. His 0.918 this season is OK, but not above-average. His standard deviation over this season has been the same 0.075 as Rinne, so roughly the same variance from the mean, buut since his average is lower it bumps his bad games down below 0.843 and lowers the bar for excellence to 0.994 ... which is really still a shutout in any game with less than 167 shots on net. Thomas has also gotten 4 shutouts (above-average) and then 4 below-average games. His total appearances of 51 has gotten the Bruins 29 wins, for a winning percentage of 57% ... hmm. Maybe this isn't in contrast after all. In fact, none of Thomas' wins came in a sub-par game for him, and of his below-average games the team won exactly none. Looks like Thomas' up-and-down really changed the team's fortunes (and just scanning the other losses, 6 were above the league save percentage above-average mark - just like Rinne).

So let's look for a real contrast - Roberto Luongo. Everyone says he's been struggling this season, but lately he's back on track. He has a season average of a mere 0.916, so pretty average there in comparison. His standard deviation from that typical number has been ... oh. 0.074, almost identical to Rinne and Thomas. Interesting (and may point to the changes in defence being a for-real correlation). But hey, let's look at the winning percentage, where's he's only a 55% (27 wins in 49 appearances) ... so kinda like Thomas really. And his 'Nucks never bailed him out on a below-average game; they were all losses. Of Vancouver's 22 losses with him in net, only 5 times did Luongo have a save perecentage measuring above-average by league standards and see them lose. This is becoming a theme here.

How about Varlamov? Everyone poo-poohed the trade sending him to Colorado - how has he changed their fates? His season save percentage is a marginal 0.910, with a standard deviation of 0.066 (oops, even LESS than the others so far - more consistent). That means anything over a 0.976 is above-average, and anything under a 0.844 is below-average (duh, right?). His winning percentage overall is 51% (23 of 45), and he had nine (9!) below-average games thus season, with the team bailing him out on only one such evening. The team, however, let him down in 7 games where his save percentage was above-average by league standards. Looks like the 'Lanche need to score more.

Maybe we need a BAD goalie. How about Mr. Steve Mason, of the lowly Columbus Blue Jackets? His save percentage on the season is a woeful 0.894 - a full two standard deviations below the league average! His personal standard deviation is ... basically the same as Rinne, Thomas, and Luongo at 0.077. So I think we can see that a goalie generally varies around 0.075 in save percentage from game to game, regardless of team or player (with reduction in that variance being good ... but I'm getting ahead of myself). That means an above-average performance for Mason would be 0.971, and a bad day is REALLY bad at 0.817. He had (by that measure) only five bad days - all losses, of course. Mason has one shutout (ergo one above-average game) amongst his 12 wins in 38 appearances (32% winning percentage). And in games where he exceeded the standard for above-average save percentage around the league, the team won 12 of those games ... so all of his wins were above-average compared to the league save percentage. And only twice did Mason give his team all he had (exceeding the 0.924 above-average for the league) and see them fall anyway. Seems that Mason needs to win by himself, or not at all.

Oh, right - that guy. Bryzgalov. Well, shit - if you read this far you're one of those people who can quote his up-to-the-minute stats without my help. (His save percentage is a 0.909, so barely within the 'average' range.) I mean, the team wouldn't have spent so much effort and money getting him here if he wasn't critically important. (His personal deviation from one game to the next is a huge 0.089 - almost 20% more than other goalies. He has 8 above-average games, and 10 below-average ones) And considering how important save percentage is looking at the other examples, why even check Bryz and the Flyers? (Because, silly, his winning percentage of 62% hides a lot of sins. The team saved Bryz' bacon 4 times thus far, and crapped out on his better performances 6 times.) --- Alright, in honesty this is where I really get confused. In all the other cases, it looked like a strong correlation between save percentage and winning; somehow the Flyers get that all whacked out. I just don't get it - unless there's an intangible there. And in Bryzgalov's defense, before coming to Philly he had seasonal standard deviations of 0.066 and 0.052 - so REALLY consistent numbers over seasons where he logged save percentages in the upper end of the almost-above-average range for the league.

So what's my conclusion? I have none, other than circumstantial evidence that points to Bryz being a bad fit in Philly's system somehow. He's a definitely high-range average goalie (so around 0.920), which is really all a team can ask for - and he was pretty consistent about it from game-to-game before this season. His GAA isn't significantly higher than previous years; he's always floated right around 2.5 goals per game allowed. So why the extreme variability in his performances? Did the contract go to his head? Is he still 'adjusting' to Philly somehow? Does having fans at a hockey game scare him? Who the hell knows??? But yeah Geoff - until this year, Bryzgalov was a VERY consistent goalie, and his variance kept his save percentages well above league averages even on bad nights, because he was a high-end goalie to begin with (not above-average, perhaps, but among the best of the average ones).

So if someone out there knows what the heck is going on with this team, please let me know. I'm REALLY GODDAMN CURIOUS, now that I spent hours doing this useless analysis!!!

This item was written by a member of this community and is not necessarily endorsed by <em>Broad Street Hockey</em>.

Trending Discussions

forgot?

As part of the new SB Nation launch, prior users will need to choose a permanent username, along with a new password.

I already have a Vox Media account!

Verify Vox Media account

As part of the new SB Nation launch, prior MT authors will need to choose a new username and password.

We'll email you a reset link.

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.

You must be a member of Broad Street Hockey to participate.

We have our own Community Guidelines at Broad Street Hockey. You should read them.