Evaluating goalies in context: adjusted Save Percentage, or SV%+

One of my favorite stats in baseball is called Adjusted ERA, or ERA+. Here's how it works. If soon-to-be-extended Phillie (please?) Cole Hamels has a 3.07 ERA, you might look at that number and know what it means, but you might not actually have any idea how good/bad that is relative to everyone else, and that's especially so on a year-to-year basis when run production, pitching skill, etc. fluctuates as much as it does. ERA+ more or less fixes that. Basically what it does is takes someone's ERA and puts it on a scale that tells you how good it is compared to the rest of the league.

It's calculated with a fairly simple formula:

(League average ERA / Pitcher ERA) * 100

Such that 100 is average, higher than 100 is above average, and lower than 100 is below average. (It also includes some park effects, but I'm not going to get into that here. Here's further reading, if you're interested.)

So what does this have to do with hockey, exactly? Hit the jump and I'll show you.

That initial problem I mentioned, in which you don't really know how good a number like ERA is by itself--I think that, for a lot of people, the same problem applies to goalies. A lot of people don't really think or know how good a .930 save percentage is or how bad a .890 is or anything like that. And that's especially true when looking at percentages over careers or multiple seasons, given that save percentage has been going up fairly consistently over time. Whether you want to attribute that to changes in playing styles across the NHL or general increases in goalie talent, it's been happening, and it's often something that gets overlooked.

So what I've tried to do is take save percentage and put it on a similar scale, and for demonstrational purposes here I'll (unoriginally) call it Adjusted Save Percentage, or SV%+. It does the same thing as ERA+, in the sense that someone with a SV%+ of 100 is league average and anyone above that is better while anyone below that is worse. To think of it another way: if a goalie has a SV%+ of 105 based on their raw save percentage, then he'd give up 100 goals in the same number of shots against that it would take a league-average goalie to allow 105. (Programming note: Cam Charron over at Canucks Army has done something very similar, but as you'll see I scaled this differently than he did. Hopefully there's no problem here.)

How exactly is it calculated? As such:

SV%+ = [(1 - League-average save %) / (1 - Goalie SV%)] * 100

Why's it calculated that way? Well, truthfully, I've kind of always thought that the opposite of save percentage tells you more than save percentage itself--if one goalie has a .930 SV% and the other has a .910 SV%, I think that you can tell the difference between them more easily if you instead say that one guy allows goals on 7% of his shots against and the other guy allows them on 9% of his shots against. That's kind of a separate discussion, though. But there's no question it's easier to standardize and compare goalies (especially in the scope of SV%+) when you look at it this way instead of the typical other way.

As an example to the above: this year Henrik Lundqvist had a save percentage of .9298, and the combined save percentage for all goalies in the NHL this year was .9135. If you try to compare by using raw save percentage, you get (.9298/.9135)*100=101.8, and I don't think you'll catch anyone believing you when you say Henrik Lundqvist is only 1.8% better at preventing goals than a league-average goalie. However, if you flip it the other way and use goals-against %, you get a SV%+ of (.0865/.0702)*100=123.2, which looks a lot more accurate given his status as a truly elite goalie and the Vezina winner this season.

So how does the spread between goalies look when you put all of them like this? Here's what the SV%+ chart would look like for the 2011-12 season, featuring all goalies who faced a minimum of four shots per game (meaning a total of 328 shots, which is about 12 games' worth or so).

Player SV%+
Brian Elliott 144.899
Cory Schneider 136.178
Mike Smith 124.050
Henrik Lundqvist 123.226
Tuukka Rask 122.030
Jonathan Quick 121.112
Jaroslav Halak 116.340
Pekka Rinne 112.141
Kari Lehtonen 110.557
Miikka Kiprusoff 108.879
Jimmy Howard 108.696
Tim Thomas 108.668
Roberto Luongo 107.363
Niklas Backstrom 106.966
Jean-Sebastien Giguere 106.512
Johan Hedberg 105.220
Josh Harding 104.716
Tomas Vokoun 103.995
Jose Theodore 103.893
Jhonas Enroth 103.755
Carey Price 103.431
Ryan Miller 103.063
Thomas Greiss 102.026
Cam Ward 101.807
Antti Niemi 101.417
Devan Dubnyk 101.117
Craig Anderson 100.454
Evgeni Nabokov 100.331
Marc-Andre Fleury 99.912
Peter Budaj 99.825
Scott Clemmensen 99.754
Semyon Varlamov 99.432
Joey MacDonald 98.388
Anders Lindback 98.351
Jason LaBarbera 97.723
Curtis Sanford 97.622
Nikolai Khabibulin 96.319
Jonas Hiller 96.011
Richard Bachman 95.913
Ilya Bryzgalov 95.293
Jonathan Bernier 94.615
Martin Brodeur 93.583
Ondrej Pavelec 92.166
Martin Biron 89.748
Corey Crawford 89.246
Michal Neuvirth 88.829
Ben Scrivens 88.686
Jonas Gustavsson 88.547
Mathieu Garon 87.268
Ray Emery 86.879
James Reimer 86.819
Sergei Bobrovsky 85.243
Chris Mason 84.854
Steve Mason 81.928
Al Montoya 80.828
Dwayne Roloson 76.060
Ty Conklin 74.279
Brent Johnson 73.905

Obviously, the order of all of those guys exactly reflects the order they'd be in on a regular save-percentage listing. But like I said, I think it's easy to look at it a bit differently when you can clearly see how well that guy did relative to his peers/to the rest of the league.

Those big numbers at the top for the guys like Elliott, Smith, Lundqvist, and Quick are pretty staggering, as are the ones at the bottom for Johnson, Conklin, and Roloson. But as we'll see here in a second, those kinds of years are typically outliers from the career norms of a guy.

To calculate any given player's career SV%+ I took each respective season's SV%+ by multiplying it by the fraction of a goalie's career shots against that he happened to face in that season. Basically, it compares a goalie's performance to the one you'd see from a league-average goalie who faced the exact same amount of shots in every year that the goalie faced. So if some made-up goalie had a five-year career that looked like this, here's how you would find his career SV%+.

SA SV%+ SA % Career wt
2007-08 1000 100 0.111111 11.11
2008-09 1500 120 0.166667 20
2009-10 2000 90 0.222222 20
2010-11 2500 125 0.277778 34.72
2011-12 2000 105 0.222222 23.33
9000 109.17

He faced .11111 of his career shots-against in his first season, .166667 in his second, and so on and so on, and you multiply each of those percentages by the respective SV%+ of that year, and you sum each of those products up and you've got your career average. Simple enough, no?

So I went through the 50 goalies who have faced the most regular-season shots since the lockout and found their career numbers using the process described above. I've sorted them here by SV%+, but I've also included their regular career save percentages and their ranks in that, and provided the differences in their rankings in the two stats for comparison's sake. (Negative numbers in that last column mean that the SV%+ rank is higher; positive ones mean that the regular SV% rank is higher.)

Goalie SV%+ SV%+ Rank SV% SV% Rank Difference
Tim Thomas 117.74 1 0.9214 1 0
Roberto Luongo 114.31 2 0.9193 4 -2
Henrik Lundqvist 113.96 3 0.9200 3 0
Pekka Rinne 112.16 4 0.9209 2 2
Tomas Vokoun 110.88 5 0.9168 9 -4
Niklas Backstrom 110.23 6 0.9177 7 -1
Jaroslav Halak 109.13 7 0.9179 5 2
Martin Brodeur 108.49 8 0.9130 20 -12
Cristobal Huet 108.45 9 0.9134 18 -9
Jonas Hiller 108.17 10 0.9178 6 4
Ryan Miller 107.71 11 0.9147 13 -2
Miikka Kiprusoff 106.98 12 0.9136 17 -5
Jonathan Quick 106.75 13 0.9164 10 3
Jean-Sebastien Giguere 106.71 14 0.9130 21 -7
Jimmy Howard 106.55 15 0.9169 8 7
Kari Lehtonen 106.49 16 0.9141 15 1
Manny Legace 106.16 17 0.9116 23 -6
Martin Gerber 106.08 18 0.9109 24 -6
Carey Price 106.02 19 0.9160 11 8
Ilya Bryzgalov 105.98 20 0.9147 14 6
Evgeni Nabokov 105.63 21 0.9124 22 -1
Mike Smith 105.62 22 0.9138 16 6
Craig Anderson 104.82 23 0.9133 19 4
Antti Niemi 103.57 24 0.9155 12 12
Chris Mason 103.48 25 0.9104 27 -2
Marty Turco 103.45 26 0.9096 28 -2
Martin Biron 103.45 27 0.9106 25 2
Jose Theodore 103.29 28 0.9094 29 -1
Brian Elliott 102.17 29 0.9093 30 -1
Dwayne Roloson 101.86 30 0.9084 32 -2
Cam Ward 101.77 31 0.9105 26 5
Olaf Kolzig 101.57 32 0.9063 37 -5
Nikolai Khabibulin 101.48 33 0.9073 34 -1
Ray Emery 100.71 34 0.9070 36 -2
Marc-Andre Fleury 100.60 35 0.9092 31 4
Chris Osgood 100.10 36 0.9051 39 -3
Ty Conklin 99.10 37 0.9059 38 -1
Dan Ellis 99.06 38 0.9084 33 5
Pascal Leclaire 98.21 39 0.9036 42 -3
Alex Auld 98.15 40 0.9037 40 0
Rick Dipietro 97.07 41 0.9027 45 -4
Vesa Toskala 96.54 42 0.9024 46 -4
Brent Johnson 96.50 43 0.9036 41 2
Matheiu Garon 96.45 44 0.9035 43 1
Antero Niittymaki 96.01 45 0.9023 49 -4
Andrew Raycroft 94.87 46 0.9003 50 -4
Ondrej Pavelec 94.73 47 0.9071 35 12
Peter Budaj 94.28 48 0.9023 48 0
Johan Hedberg 93.94 49 0.9024 47 2
Steve Mason 92.58 50 0.9035 44 6

One of the things that immediately stuck out to me was that guys who are older and have been around for longer (Brodeur, Giguere, Huet, Kiprusoff) ranked much higher in SV%+ than they did in regular SV%, while younger guys who haven't been around as long (Niemi, Price, Howard, Pavelec) take a tumble when you put them in terms of SV%+. This isn't at all surprising--one of the things that made me want to look into this was the fact that the old guys mostly played in years where save percentages were a fair bit lower than they are now, so their career numbers need to be looked at in the proper context. Which is to say that these numbers are not necessarily meant as predictors of future performance; rather, they're meant as evaluators of what guys have done up to this point in their careers.

Having said that, I think there are a few observations to be made, and then just a few thoughts on points that I thought were noteworthy.

--First and foremost, an observation that anyone who's been paying attention could tell you and has probably been telling you for years now: the spread in overall goalie talent is just not that big anymore. Unless you hit the jackpot with one of the guys at the very top of the list, chances are that the guy you're getting isn't going to be THAT much better than an average goalie. 36 of those 50 guys are at least ranked above a 100, and 30 of those 36 are between 100 and 110. It's not a huge spread if, theoretically, there's at least one guy out there for every team who's been "average or better" during his career. (Obviously it doesn't quite work like that, since if everyone is above-average then eventually average has to go up, but hopefully you see what I mean.)

--In my opinion/judgment, based on the names above, I'd say that the line for "serviceable starting goaltender" is probably somewhere around 103, and "surefire starting goaltender" is probably at about 106. Curious as to whether people see those lines as a bit higher/lower.

--For a point of comparison, after going through those guys I looked at Dominik Hasek. His career mark was a 127. Good lord. Again, this kind of contributes to the idea that the gap between "elite goalie" and "average goalie" isn't anywhere near what it used to be. That kind of domination would be equivalent to posting a .932 this season and basically sustaining it for the rest of your career, which more or less isn't something that goalies do anymore.

--We'll talk more about the Flyers in a bit, but before we do that, let's look at our guy in net right now. First, the bad news. Ilya Bryzgalov rates 20th on the career list above, and 17th among players who are active and in the NHL (18th if Huet ever finds his way back over here). While I'd still say these save percentage numbers reflect positively on him as a whole, that's not exactly the kind of guy you want to have under contract for eight more years, at a cap hit that's an overpay for probably any goalie.

--But here's the good news, particularly if you're concerned about Bryz's performance this year. In calculating the numbers for the guys listed above, I went through every year of the careers of each one of them and found out this encouraging fact. Out of the 50 guys listed above, only four of them have started for at least four seasons and not had at least one year since the lockout in which their save percentage was below-average. Those four would be Henrik Lundqvist, Roberto Luongo, Tomas Vokoun, and Kari Lehtonen. All but one of the other 46 goalies has had at least one year between '05-'06 and '11-'12 in which they started a decent number of games and finished with a SV%+ below 100 (the only one who hasn't was Antti Niemi, and he's only been a starter for 2.5 seasons). Which is to say: don't spend too much time fretting over Bryz's tough year. It happens to almost everyone at some point. If it happens again next year? Sure, maybe then we've got a legitimate problem. But everyone gets a down year or two in there.

--Tim Thomas, while insane, is real damn good. Not quite as good as he was throughout 2010-11, but good enough that he may be the best goalie in the league. Similarly, while I can't blame the Canucks for giving Schneider a chance to start, whoever gets Luongo is getting a hell of a goalie. Don't forget that.

--Chris Osgood is totally going to get into the Hall of Fame. That's cool, but he's as average as they get.

--On that note, if this past April hadn't happened I'd think Marc-Andre Fleury would be on the road to becoming the next Chris Osgood. We'll have to see how much that perception has changed after his monstrosity of a first-round series. But yeah, he's pretty average too.

--Does Ondrej Pavelec have the worst contract for a goalie out there? OK no he doesn't our guy does fine shut up. But his numbers aren't exactly encouraging.

There are lost more other noteworthy things in there. Feel free to point them out if you please.


Finally, let's actually relate this whole thing to the Flyers. As we all know, the Flyers have had terrible goaltending and always have forever ever since Hextall. Or at least that's how the story goes. It's one that's been discussed on here a lot, and one that Geoff has partially debunked in articles in the past. I decided to take the last 10 goalies who played at least a portion of a season with the Flyers and look at both their career SV%+ and their SV%+ with the Flyers themselves. Here's what I came up with.

Goalie GP SV%+ (w/Flyers) Career SV%+
Brian Boucher 170 97.5159 94.0330
Roman Cechmanek 163 121.1550 114.9483
Antero Niittymaki 161 97.3724 95.9573
Martin Biron 133 108.4423 103.4524
Robert Esche 128 95.7632 94.6752
Sergei Bobrovsky 83 96.5945 96.5945
Ilya Bryzgalov 59 95.2926 105.9815
Michael Leighton 32 104.1566 91.7796
Ray Emery 29 93.8868 100.7101
Jeff Hackett 27 93.7155 104.1449

So what it seems like we've got out of this group is one legitimately fantastic goalie who, unfortunately, was apparently too crazy for some (Cechmanek), one solid goalie who played well in his time here (Biron), two guys who clearly had some talent but weren't/haven't been able to quite put it together while here (Emery, Bryzgalov), one guy who clearly doesn't have much talent but caught lightning in a bottle while he was here (Leighton), three guys who performed better than their career norms while here but were never really that good (Esche, Niittymaki, and Boucher), one guy who might be in that category but is probably too young and too much of a wild card to really tell (Bobrovsky), and one real old dude who played the last year of his respectable career here and did not do particularly well (Hackett).

Certainly a bit of a mixed bag, and not one I'd exactly call enviable. However, it gets more interesting when you aggregate them all together. If you take every season that each of these guys played with the Flyers and combine them all into one supergoalie and calculate THAT goalie's SV%+, it would be 102.3798, which is...above-average.

Wait, what? Let's add some qualifications to that statement:

--This isn't quite an exact science, because it doesn't include anyone else who played in that time-frame who may have only played a few games and done poorly in the process.

--I'd be curious to see what this looked like for most other teams. It's like just looking at the Top 50 goalies and finding that 36 of them are above 100--it's not going to be a perfect split above and below the line, or anything like that.

--Like I said above, even if 100 is "above-average", I still wouldn't quite say you've got a guy you're really OK with in net until about 103, so it's not like they've been great.

--Take Cechmanek away and the number drops to 99.0196, so it's pretty clear he was a bit of a driving force behind that number.

--For a point of reference, if you were to take every season that these guys played in goal with any team and combine them and calculate THAT supergoalie's SV%+, you would get 101.6216, which means that these guys were, on average, better here than they were elsewhere. That makes sense, given that six of the nine guys on the list who have played for at least two teams did better here than they did for their career as a whole, but I thought it was interesting. Maybe the Flyers aren't cursed at goalie--maybe the guys they've been getting are just pretty pedestrian, though not horrible.

So yeah. I wouldn't go around saying that the Flyers have had above-average goaltending over the last decade or so, but I'd say that they certainly haven't been terrible--definitely not as bad as you'd think from what you hear everywhere else.

This item was written by a member of this community and is not necessarily endorsed by <em>Broad Street Hockey</em>.