NHL scouts may have a harder job than any other professional sports scouts when it comes to preparing for the draft. Uniquely among North American professional drafts, the NHL draft forces teams into quite difficult comparisons and forces them to be incredibly accurate. Unlike the NBA and NFL drafts, where the vast majority—and, in some years, in the case of the NFL, all—of draftees have played together in the NCAA for at least a year, the NHL draft draws upon players from twenty different leagues spread over two continents, and there is very little opportunity for players from these different leagues to play together (and, of course, to be evaluated together). Of course, the MLB draft forces scouts into a similar position, but while the NHL draft has only seven rounds, forcing teams to pick wisely, the MLB draft has forty rounds, which allows for a much lower success rate. Finally, with the advent of the salary cap era, building a team through free agency has become increasingly difficult, which makes drafting well a priority for any team that wishes to win.
Because of the unique importance and challenge of the NHL draft, I thought it may be useful to look at past drafts and examine whether there’s anything in a player’s pre-draft performance that could indicate whether the player will end up becoming a successful NHL player or not. For this, I looked at all forwards drafted in the first or second rounds between 2005 and 2009 who had played in the same league for two years prior to their being drafted (one of the metrics that I wanted to look at was improvement between the pre-pre-draft year and pre-draft year, so I only wanted to take players that were playing in the same league for both of those years). While in the future I’d like to look more closely at different players’ NHL performance, for this first analysis, I chose to sort players into two categories—those who had "made it" five years after being drafted and those who had not. I defined a player as having made it if, five years after being drafted, he had played the majority of his games in the NHL.
The first conclusion I was able to draw is that front offices are actually pretty decent at their jobs. Of course, they make mistakes sometimes (Bobby Sanguinetti over Claude Giroux, for example), but on the whole, players drafted earlier are much more likely to make it to the NHL. In fact, of the 57 first round picks that I looked at, 40 made it, while of the 58 second round picks, only 16 made it. Of course, then, the next thing that I wanted to look at was what set apart the 17 first rounds picks that hadn’t made it from the 40 who had as well as what set apart the 16 second round picks who had made it from the 42 who hadn’t.
So, the first thing I did was compare the pre-draft statistics—goals per game, assists per game, and points per game in each of the two seasons prior to the draft, the improvement in each of these metrics between those two seasons, and the player’s listed height and weight—of players in the "made it" group to those in the "didn’t make it" group in each round. After looking at all of those statistics, the conclusion I found was that there isn’t much in those statistical areas to separate the groups. First rounds who didn’t make it had almost exactly the same GPG, APG, PPG, height, and weight as those who did make it. The one possible different is that, on average, those who did not make it from the first round only improved by .35 PPG between those two seasons while those who did make it improved by .50 PPG. This result is interesting, because it does seem to be intuitively plausible—players who showed more improvement in the year before their draft might also be expected to show more improvement in the years after their draft and, as such, have a higher chance of making it. The difference, however, was not particularly significant (P-value = .122).
Similarly, the difference between second round picks who made it and those who didn’t is almost non-existent. In fact, those who didn’t make it had better GPG, APG, and PPG the year before the draft and even had, on average, better GPG and PPG improvement between two years before and the year before.
So, the question now is, if there isn’t much to differentiate players who made it from those who didn’t by looking at statistics, the question becomes whether there’s anything else that might provide some insight here. The answer, in fact, is that there is: the player’s pre-draft league turned out to be significantly more predictive of his future success. While there aren’t really enough players from non-CHL/NTDP leagues in the analysis to give good information about those, the difference in success rates among WHL, OHL, QMJHL, and NTDP players is staggering. In particular, OHL players tend to dramatically overperform expectations based on draft round, while QMJHL and NTDP players tend to underperform expectations (and the results are mixed for WHL players).
While 70.2% of all first round picks made it, only 55.6% of QMJHL and only 1 out of 2 NTDP first round picks made it. On the other hand, 73.3% of WHL and a full 91.3% of OHL first round picks made it. The results are similar for second round picks. Overall, only 27.6% of second round picks made it, but even fewer WHL (1/9), QMJHL (1/9), and NTDP (0/9) second round picks made it, while a full 6 out of 12 OHL second round picks made it.
So, the final conclusion here is that there is very little in the way of pre-draft statistics to tip scouts off as to which first round picks might underperform expectations or which second round picks might overperform them. Indeed, even the league break-out results aren’t all that telling—sure, second-round OHL picks are more likely to make it than second-round NTDP, WHL, or QMJHL picks, but they’re less likely to make it than WHL or QMJHL first-round picks, which suggests that scouts are doing a pretty decent job of roughly ranking players by round. At the end of the day, however, if a scout has two players from different leagues ranked relatively closely and one of those players is from the OHL, it might be instructive to remember that those players have a greater historical success rate at overperforming their expectations, particularly relative to QMJHL, NTDP, or WHL players.