Paul Stewart: Lucy, Tell Your Statistics to Shut Up (AKA Ref Analytics Are Useless)

Paul Stewart: Lucy, Tell Your Statistics to Shut Up (AKA Ref Analytics Are Useless)
- Paul Stewart

Thank you!!

I'm caught in between age wise of the "old school" vs "new school" way of thinking about the game.

I am not a fan of the "money ball like" stats. I understand them I think some people use them to pretend to know more about the game and how its played, to come off as smart.

Find me a player that can skate, pass, score and I couldn't care less about his Fenwick and Corsi numbers.

Paul Stewart: Lucy, Tell Your Statistics to Shut Up (AKA Ref Analytics Are Useless)
- Paul Stewart

I wholeheartedly agree with you about refereeing statistics (even the so-called "French Canadian bias"), but I couldn't disagree more about the application of player statistics. The whole point of Moneyball was the fact that actual compiled numbers were proving that sometimes scouts didn't actually see the true value of a baseball player. Advanced hockey metrics work the same way.

Wins, losses, and goals are indeed the most important stats, but those don't exist in a vacuum. While you may not necessarily agree, teams that have the puck more generally register more shots on goal, and the teams that regularly outshoot their opponents by a large margin score more goals and, more often than not, win. Take a look at the possession stats of the last 6 Stanley Cup winners and you'll see that they follow this formula.

Now neither of us needs a Corsi chart to tell us that Pavel Datsyuk is a hell of a player, but we can use those metrics to determine just HOW good a player he is, relative to the rest of the league, and how big his contribution is to his team. Too often people want to look at statistics and point to whether or not that makes a player good, when the reality is that it's a combination of factors. If you play on a line with Patrice Bergeron, your Corsi will probably be pretty high, even if you aren't contributing as much to it by yourself. True analysis can only be done with both the eyes and the numbers.

As for officials, charting their statistics (other than missed calls or bad calls) is absurd, because they aren't actually playing the game. An official is reactive, and they can only call what happens on the ice.

As for officials, charting their statistics (other than missed calls or bad calls) is absurd, because they aren't actually playing the game. An official is reactive, and they can only call what happens on the ice.
- Sandus

An official is reactive, but that does not mean refs are objectively reactive (no one is entirely objective, biases exist no matter how hard you try to suppress them). So, yes, there is value in charting officiating. As the data sample grows, trends will emerge specific to officials. Yes, looking at an official in a single game is without statistical value ... but looking at an official's track record after 500 games? Absolutely has value. Maybe that ref is more disposed to call goalie interference ... maybe that official is more likely to be swayed by the home crowd's groan ... so on and so forth.

More importantly, lets look at it from a team's perspective. What if I know the two officials on a given night tend to call less diving calls. Maybe I'm a bit more free with the embellishment. Conversely, if I know an official is quick to call a dive, then you avoid going down that path.

The idea that sports cannot be statistically modeled is ... head in the sand time. Google analytics probably knows more about all of us than we do ... and every day life is far more complex than a hockey game. And, yes, that should be a bit scary.

An official is reactive, but that does not mean refs are objectively reactive (no one is entirely objective, biases exist no matter how hard you try to suppress them). So, yes, there is value in charting officiating. As the data sample grows, trends will emerge specific to officials. Yes, looking at an official in a single game is without statistical value ... but looking at an official's track record after 500 games? Absolutely has value. Maybe that ref is more disposed to call goalie interference ... maybe that official is more likely to be swayed by the home crowd's groan ... so on and so forth.

More importantly, lets look at it from a team's perspective. What if I know the two officials on a given night tend to call less diving calls. Maybe I'm a bit more free with the embellishment. Conversely, if I know an official is quick to call a dive, then you avoid going down that path.

The idea that sports cannot be statistically modeled is ... head in the sand time. Google analytics probably knows more about all of us than we do ... and every day life is far more complex than a hockey game. And, yes, that should be a bit scary.
- gollum

How is his track record over 500 games any different than his track record in an individual game? Goalie interference is goalie interference. If the interference call is a dubious one, that's worth paying attention to. But if it's a penalty, it's a penalty.

What if one official over 500 games calls more 4 minute high sticking penalties? Is that because he has bias that predisposes him to assess the double minor? Or is it perhaps a coincidence that he has been in a lot of games where there have been high sticks worth 4 minutes instead of 2?

How is his track record over 500 games any different than his track record in an individual game? Goalie interference is goalie interference. If the interference call is a dubious one, that's worth paying attention to. But if it's a penalty, it's a penalty.

What if one official over 500 games calls more 4 minute high sticking penalties? Is that because he has bias that predisposes him to assess the double minor? Or is it perhaps a coincidence that he has been in a lot of games where there have been high sticks worth 4 minutes instead of 2?
- Sandus

Individual games are highly volatile. Large samples are not. Over an individual game or two, it's entirely possible that a ref may have "experienced" more high sticks in the games he officiated than another referee. Over 500 games? Those examples of "statistical noise" will get washed out, and two independent refs working NHL games would be expected to have a similar sample of "officiating events" where they could have made a call one way or the other.

You're right, goalie interference is more subjective ... but a high stick (unfortunately) is subjective as well to a lesser degree. Perhaps that ref just doesn't notice stuff as much. Perhaps a ref is more responsive to the modern head snap, and, therefore, more likely to hand out a two minute minor for a weak high stick when a guy embellishes.

The fundamental problem here (and this is what makes Stewart's blog extremely weak as far as actually understanding statistical analysis) is that statistics are not, and NEVER are about one game. They are not based on a single game (or event), and they do not purport to have anything interesting to say about a single game (or event). Stewart noted that statistics have real value in baseball, but the best pitcher in the world could throw a meatball to the worst hitter in the world and watch it land in the left field seats in a single AB.

It's also worth noting ... Stewart is far from an objective commentator on officials.

Individual games are highly volatile. Large samples are not. Over an individual game or two, it's entirely possible that a ref may have "experienced" more high sticks in the games he officiated than another referee. Over 500 games? Those examples of "statistical noise" will get washed out, and two independent refs working NHL games would be expected to have a similar sample of "officiating events" where they could have made a call one way or the other.

You're right, goalie interference is more subjective ... but a high stick (unfortunately) is subjective as well to a lesser degree. Perhaps that ref just doesn't notice stuff as much. Perhaps a ref is more responsive to the modern head snap, and, therefore, more likely to hand out a two minute minor for a weak high stick when a guy embellishes.

The fundamental problem here (and this is what makes Stewart's blog extremely weak as far as actually understanding statistical analysis) is that statistics are not, and NEVER are about one game. They are not based on a single game (or event), and they do not purport to have anything interesting to say about a single game (or event). Stewart noted that statistics have real value in baseball, but the best pitcher in the world could throw a meatball to the worst hitter in the world and watch it land in the left field seats in a single AB.

It's also worth noting ... Stewart is far from an objective commentator on officials.
- gollum

This is the point I'm getting at. You can assess a bad call or poor performance within a game and document that, without documenting the rest of the game. The number of penalties of a certain type or the positions of the players or the home crowd or any of that is irrelevant. A bad call is a bad call, and you evaluate each bad call on a case by case basis and log the findings. Ditto with missed calls. You don't worry about any other statistics because they simply bog down the data. Then you can can evaluate officials' performance based on the total number of poor or missed calls per game. If there's a bias in there (e.g. 60% of missed calls were interference penalties), then you'll be able to find it.

This is the point I'm getting at. You can assess a bad call or poor performance within a game and document that, without documenting the rest of the game. The number of penalties of a certain type or the positions of the players or the home crowd or any of that is irrelevant. A bad call is a bad call, and you evaluate each bad call on a case by case basis and log the findings. Ditto with missed calls. You don't worry about any other statistics because they simply bog down the data. Then you can can evaluate officials' performance based on the total number of poor or missed calls per game. If there's a bias in there (e.g. 60% of missed calls were interference penalties), then you'll be able to find it.
- Sandus

You're missing the point ... it isn't about "good" calls and "bad" calls. It's about officials and tendencies, and officials have tendencies in their performance just like players (because, ya know, their human). While some calls are objectively "bad," the vast majority of controversial calls are subjective in some way.

Now, sure, if your purpose is to evaluate whether a ref is "good" or "bad," you can go through and try to assess each and every call a ref makes. But that is a very specific application, and, frankly, not terribly interesting. That's job evaluation stuff, and if they aren't doing that ... then they suck at evaluating the job referees are doing.

What is interesting is the other stuff, which you are dismissing. Lets just pull a number out of our a** as an illustrative example. Lets say, on average, goalie interference is called in a quarter of games played. So, we would *expect* a given referee to call approximately 10 goalie interference penalties per year (his partner calling another 10). But lets say we have a ref that called 15 in a give year. In a given year (82 games is a small sample), that may just be the luck of the draw ... but lets say that trend holds true for 5 or 6 years. The evidence begins to support the idea that the ref is more predisposed to calling goalie interference ... so tell your forwards to be careful around the crease.

It is true that baseball is an "easier" sport statistically due to its event-by-event nature. The problem, however, is folks that understand statistics and probability less than they should before going down this path have a tendency to make the specious argument that "oh, it works in baseball but it doesn't work in [fill in the blank]." This stuff works quite well in other sports, but they do tend to require greater nuance and care in their application and in how we talk about them. It's easy to understand that a .300 hitter slumping to .250 is going to make that up over his next 200 PAs ... it's not so easy to understand how statistics work within the flow of a sport like hockey, but they do.

Within Stewart's blog you see it in the casual dismissal of Corsi and Fenwick. Those stats are meaningless in an individual game. They have great value over a season, though. Essentially, those stats relate to possession. We know puck possession teams have great success, because we have all experienced the dominance of the Red Wings for the last 20 years ... a team that focused on puck control on a team-wide basis.

You're missing the point ... it isn't about "good" calls and "bad" calls. It's about officials and tendencies, and officials have tendencies in their performance just like players (because, ya know, their human). While some calls are objectively "bad," the vast majority of controversial calls are subjective in some way.

Now, sure, if your purpose is to evaluate whether a ref is "good" or "bad," you can go through and try to assess each and every call a ref makes. But that is a very specific application, and, frankly, not terribly interesting. That's job evaluation stuff, and if they aren't doing that ... then they suck at evaluating the job referees are doing.

What is interesting is the other stuff, which you are dismissing. Lets just pull a number out of our a** as an illustrative example. Lets say, on average, goalie interference is called in a quarter of games played. So, we would *expect* a given referee to call approximately 10 goalie interference penalties per year (his partner calling another 10). But lets say we have a ref that called 15 in a give year. In a given year (82 games is a small sample), that may just be the luck of the draw ... but lets say that trend holds true for 5 or 6 years. The evidence begins to support the idea that the ref is more predisposed to calling goalie interference ... so tell your forwards to be careful around the crease.

It is true that baseball is an "easier" sport statistically due to its event-by-event nature. The problem, however, is folks that understand statistics and probability less than they should before going down this path have a tendency to make the specious argument that "oh, it works in baseball but it doesn't work in
- gollum[fill in the blank]." This stuff works quite well in other sports, but they do tend to require greater nuance and care in their application and in how we talk about them. It's easy to understand that a .300 hitter slumping to .250 is going to make that up over his next 200 PAs ... it's not so easy to understand how statistics work within the flow of a sport like hockey, but they do.

Within Stewart's blog you see it in the casual dismissal of Corsi and Fenwick. Those stats are meaningless in an individual game. They have great value over a season, though. Essentially, those stats relate to possession. We know puck possession teams have great success, because we have all experienced the dominance of the Red Wings for the last 20 years ... a team that focused on puck control on a team-wide basis.

This is contrary to what officiating is about though. The goal is to enforce the rules as properly and uniformly as possible, not to have certain refs call the game in different ways. This happens in baseball all the time and it's infuriating (though the "old-timers" will tell you it's what makes the game good), like when pitchers have to adjust for a certain umpire with a quirky or atypical strike zone.

If every police officer had his own subjective interpretation of the law, how would you feel? Maybe one mph over the limit is speeding. Maybe it's 4 is speeding but 1-3 isn't. Isn't it better for every official to enforce the rules the same way and to correct those who don't do it right, rather than compile years and years worth of data about tendencies and have to adjust your own game?

This is contrary to what officiating is about though. The goal is to enforce the rules as properly and uniformly as possible, not to have certain refs call the game in different ways. This happens in baseball all the time and it's infuriating (though the "old-timers" will tell you it's what makes the game good), like when pitchers have to adjust for a certain umpire with a quirky or atypical strike zone.
- Sandus

You are describing the ideal ... not reality. In reality, officials have tendencies, weak spots, etc. We all have ideas about who is and is not a "good" referee in the NHL. We would love it ever every official was at that level, but they aren't. And they never will be.

If every police officer had his own subjective interpretation of the law, how would you feel? Maybe one mph over the limit is speeding. Maybe it's 4 is speeding but 1-3 isn't. Isn't it better for every official to enforce the rules the same way and to correct those who don't do it right, rather than compile years and years worth of data about tendencies and have to adjust your own game?

1) Where do you live? I want to move there, because the idea of a police force that isn't full of arbitrariness would be a utopia.

2) Yes, it would absolutely be better if every official was exactly the same, and called games exactly the same way.

3) No. 2 is utterly impossible to achieve with human officials. In any sport.

Yes, big data can identify lots of interesting things, but, as said, it doesn't apply to a single game. For instance, if you compared 2 officials over a 3 year, full season period, and one official 'A' averaging calling 3 penalties per game, and the other 'B' called 6 per game, it would be pretty clear that one official called the game more closely than the other.

The volume of data would negate the 'noise' of different teams, different parts of the season, etc. And it would be reasonable to expect ref 'B' to call more penalties than ref 'A', although, in any single game, 'A' may call more than 'B'.

You are describing the ideal ... not reality. In reality, officials have tendencies, weak spots, etc. We all have ideas about who is and is not a "good" referee in the NHL. We would love it ever every official was at that level, but they aren't. And they never will be.



1) Where do you live? I want to move there, because the idea of a police force that isn't full of arbitrariness would be a utopia.

2) Yes, it would absolutely be better if every official was exactly the same, and called games exactly the same way.

3) No. 2 is utterly impossible to achieve with human officials. In any sport.
- gollum

The use of the police as an example was meant to be a little ironic, but yes, we are in agreement. It's not the way things are, and it would be great if it was. I just believe it's the job of the officials to work on their games and become more consistent rather than the job of the players/teams to adjust to the weaknesses of the officials. I don't want to see teams play differently depending on who gets assigned to the game.

The use of the police as an example was meant to be a little ironic, but yes, we are in agreement. It's not the way things are, and it would be great if it was. I just believe it's the job of the officials to work on their games and become more consistent rather than the job of the players/teams to adjust to the weaknesses of the officials. I don't want to see teams play differently depending on who gets assigned to the game.
- Sandus

Sure, no one should remain static and not seek to "get better." But the reality is that they're never going to be perfect, and they're never going to be entirely uniform. Moreover, that is not indicative (necessarily) of being a "bad" referee. They're all out there doing the best that they can do. Teams should not sit around and wait for that to happen.

The reality is that this type of stuff happens naturally ... the issue people have is when things start to get quantified. It freaks 'em out for a variety of reasons, but one of the most important is that human beings are terrible at processing large amounts of information, and cognitive biases are all over our assessments. One of the great values of empirical data is that it cuts through some of those biases and if you are open minded, forces you to assess what you think with what actually happened. When that data becomes sophisticated, it can tell you a lot.

The concept: Directing more shots towards your opponents net than he directs at yours will increase your long-term chances of out scoring him and winning games.

Why this concept is so controversial for so many is mind-boggling to me.

The concept: Directing more shots towards your opponents net than he directs at yours will increase your long-term chances of out scoring him and winning games.

Why this concept is so controversial for so many is mind-boggling to me.
- BeanCountingHab

Because people don't like the sense of lost agency to individual players, and then teams.

What I find more frustrating is the idea that analytics are ignorant of the concept of distinguishing between team and individual in statistics.

Refs topic related..

First Read this:

http://www.cbc.ca/sports-...-in-outcome-of-games.html

Now Read this one:

http://www.thehockeynews....o-war-with-wayne-gretzky/

Where's the "I freakin' love this article" button?

Paul Stewart: Lucy, Tell Your Statistics to Shut Up (AKA Ref Analytics Are Useless)
- Paul Stewart

If you cant see the usefulness then you are not looking and being pigheaded....there is a lot of great info in todays advanced stats. Are they to be used as an end all be all? No. Just one more tool.

Paul Stewart: Lucy, Tell Your Statistics to Shut Up (AKA Ref Analytics Are Useless)
- Paul Stewart

The concept: Directing more shots towards your opponents net than he directs at yours will increase your long-term chances of out scoring him and winning games.

Why this concept is so controversial for so many is mind-boggling to me.
- BeanCountingHab

Controversial? Maybe for some. I think the hang-up for me is when folks spend so much time buried in a spreadsheet they stop using their own two eyes. It's not just in hockey. I work in a company that involves heavy use of analytics to study what we do. And it's definitely very useful. But some use analytics as the absolute final word in every decision rather than as a helpful guide. When you start looking only at analytics and statistics and stop looking at the actual thing your analyzing, you lose more than you gain.

The Neil Greenberg piece a couple of years ago about Tavares not being a top 25 player under 25 based on a whole slew of advanced stats is a perfect example. The disconnect is captured when he writes:

So how can such a talented young player, who has accomplished so much in so little time, be left off a list of the 25 best players under 25 years old? Simple. His performance is not as impressive as it looks.

Yes...it is as impressive as it looks, and when you take your head out of the spreadsheet and watch the game, it's as clear as day.

The shorter version of my point: Statistics in the hands of people who can put them in the proper context are very useful. Statistics in the hands of idiots who pretend they know how to use them can lead you very far off course.

I wholeheartedly agree with you about refereeing statistics (even the so-called "French Canadian bias"), but I couldn't disagree more about the application of player statistics. The whole point of Moneyball was the fact that actual compiled numbers were proving that sometimes scouts didn't actually see the true value of a baseball player. Advanced hockey metrics work the same way.

Wins, losses, and goals are indeed the most important stats, but those don't exist in a vacuum. While you may not necessarily agree, teams that have the puck more generally register more shots on goal, and the teams that regularly outshoot their opponents by a large margin score more goals and, more often than not, win. Take a look at the possession stats of the last 6 Stanley Cup winners and you'll see that they follow this formula.

Now neither of us needs a Corsi chart to tell us that Pavel Datsyuk is a hell of a player, but we can use those metrics to determine just HOW good a player he is, relative to the rest of the league, and how big his contribution is to his team. Too often people want to look at statistics and point to whether or not that makes a player good, when the reality is that it's a combination of factors. If you play on a line with Patrice Bergeron, your Corsi will probably be pretty high, even if you aren't contributing as much to it by yourself. True analysis can only be done with both the eyes and the numbers.

As for officials, charting their statistics (other than missed calls or bad calls) is absurd, because they aren't actually playing the game. An official is reactive, and they can only call what happens on the ice.
- Sandus

I obviously can't entirely discredit advanced hockey statistics but I think it's hard to apply "moneyball" style stats to hockey. So much in hockey revolves around intangibles. Sure advanced hockey metrics tell a story, but certainly not the whole tale.

If "advanced stats" are useless, please explain this image:

http://i.imgur.com/zi0yJGp.jpg

Why is it that great puck-possession teams (measured by 5v5 Fenwick Close) make the playoffs more often than bad possession teams, and that every team to win the cup since 07-08 has had a regular-season Fenwick above 0.500 (except Pittsburgh, who changed coach in the playoffs and promptly became the best possession team in the playoffs)?

Having the puck allows you to score and prevents the opposition from scoring. Scoring more than the opposition means winning. Therefore, coaches and GMs should prioritize systems and players that help puck possession. It's a pretty intuitive conclusion. This is the only way to have long-term success.

Controversial? Maybe for some. I think the hang-up for me is when folks spend so much time buried in a spreadsheet they stop using their own two eyes. It's not just in hockey. I work in a company that involves heavy use of analytics to study what we do. And it's definitely very useful. But some use analytics as the absolute final word in every decision rather than as a helpful guide. When you start looking only at analytics and statistics and stop looking at the actual thing your analyzing, you lose more than you gain.

The Neil Greenberg piece a couple of years ago about Tavares not being a top 25 player under 25 based on a whole slew of advanced stats is a perfect example. The disconnect is captured when he writes:

So how can such a talented young player, who has accomplished so much in so little time, be left off a list of the 25 best players under 25 years old? Simple. His performance is not as impressive as it looks.

Yes...it is as impressive as it looks, and when you take your head out of the spreadsheet and watch the game, it's as clear as day.

The shorter version of my point: Statistics in the hands of people who can put them in the proper context are very useful. Statistics in the hands of idiots who pretend they know how to use them can lead you very far off course.
- UIF

"Try watching the games!" is the ultimate defense of anti-stat proponents, and it drive me crazy. I appreciated you're not quite saying that here, and I agree it should be a tool to use when watching the game.

If I could watch a game with perfect recall and remember every positive and negative play every player makes on the ice through the course of game, and give all those plays equal weight in my mind I probably wouldn't see any need for stats. But I can't do that. Fortunately, someone is generally taking those kind of notes for me and putting them in a format to support what I'm watching.

Sometimes, those stats support what I'm seeing on the ice. Example: Douglas Murray is in fact as useless as he looks out there. (It's actually hard to understand how terrible until you look at the stats).

Sometimes those stats might show me I've had observation bias about a player while watching. For example, it's a lot easier to dismiss errors when they don't lead to a goal or scoring chance. Conversely, a player who plays a near perfect game, but who's mistake winds up in the back of your net will always stand out in your mind more.

At the end of the day though, I think the terms "fancy stats", "advanced stats", "Fenwick" and "Corsi" are throwing up a roadblock in growing their acceptance. People hear those and think we are trying to come up with some complex formula to evaluate teams and players. What these measure are actually giving us is very simple and so intuitive:

Help your team direct a lot of shots at your opponents net and minimize the shots directed at your net and you're a positive contributor. Teams that direct more shots at their opponents net win more games on average. The facts prove this is true, why would it illicit such skepticism? What is it that doesn't make sense?

Information is information. To dismiss it as meaningless just because you say so is ignorance.

Also to throw out random stats like +/- our corsi without any basis behind them is also just as ignorant.

All information is a tool. But knowing how to read the information and apply it to the game is the real tool. No one stat can be used on is own, all information in the hockey world requires context, usually in the form of multiple stats put together to build a case.

Yeah, advanced stats make things more complicated and are not always true indicators of a players performance. But if the property context is applied they can tell you everything you need to know about a player. But the best way to judge a player is to simply watch what a player does on the ice.