OT: The Tyranny of Metrics

A new book in which the author points out that society's increasing use of metrics to evaluate performance has consequences for the very behavior being measured; consequences that are sometimes good and sometimes bad.

As an example, the "performance" of surgeons is increasingly measured by metrics that count surgical success (avoiding death or negative outcomes). The result? Studies reveal that, increasingly, surgeons decline to offer surgery to patients with complex conditions and questionable outcomes, resulting in deaths because their conditions, while requiring surgery, involve outcomes that are not "safe" enough from the perspective of surgeons engaged in "risk avoidance" because they are now being measured by metrics.

The author says he is not against the use of metrics; he is only against the overuse of metrics; against metrics not being a tool for judgment, but rather metrics being a complete replacement for judgment (judgment based upon experience).

It got me to wondering: Has the increased reliance upon metrics in sports had unintended, unforeseen, negative impacts that has changed how baseball (or other sports) is played?

The Tyranny of Metrics

Forums: 
DISCLAIMER: Forum topics may not have been written or edited by The Key Play staff.

Comments

Has the increased reliance upon metrics in sports had unintended, unforeseen, negative impacts that has changed how baseball (or other sports) is played?

The short answer is yes.

Example: when teams break recruiting rules to get a better recruit in order to win. Winning being the metric they're interested in.

You have to be careful designing the metric you're interested in, because when you manage to the metric, something that's not being emphasized can get lost, and even the way you measure it can get distorted.

This quote from the book sort of summarizes the idea: Muller says 'anything that can be measured and rewarded will be gamed.'

Example: when teams break recruiting rules to get a better recruit in order to win. Winning being the metric they're interested in.

Morality and feeling good about yourself aside, if there are no impactful punishments for breaking the rules, do they actually exist?

More and more, it just seems like the NCAA operates in this world where the rules were thrown out a long time ago, and the only people who don't realize it are the fans.

"Some days you’re a horse and some days you’re a horse’s ass. I’ve been a horse’s ass for a little while." - Roy Halladay

I want to say that morality was a stronger driving force in the past, but then I'm reminded of the 1929 World Series...

Or how about the SMU football team?

What's
Important
Now

or just UNC Athletics in general

"Some days you’re a horse and some days you’re a horse’s ass. I’ve been a horse’s ass for a little while." - Roy Halladay

Wait, I said "in the past".

You really think they only started cheating in the 90's?

"Some days you’re a horse and some days you’re a horse’s ass. I’ve been a horse’s ass for a little while." - Roy Halladay

Well played.

Cheating is well ingrained in the American way. I have seen it in absolutely every business I have worked in.

What's
Important
Now

Wanting to get the best results for minimum, risk, effort, or capital is natural. Placing rules and laws and determining morality for a society is relatively new and unnatural

Recruit Prosim

I mean over 2000+ years of rules and laws (10 Commandments) is relatively new and unnatural?

What's
Important
Now

I mean, if morality, rules, and laws were "natural", you wouldn't have to make rules and laws telling people, "Hey. Don't do that!"

Side note: Fernley, the morality debate is back!

"Never half-ass two things. Whole-ass one thing."
-Ron Swanson

"11-0, bro"
-Hunter Carpenter (probably)

Compared to biological/evolutionary urges of self preservation? Yes

Recruit Prosim

Placing rules and laws and determining morality for a society is relatively new and unnatural

Rules, laws, and enforcement are what makes a society work.

That said, having transparency and promoting a social conscience is a step in the right direction, it just doesn't engage everyone.

I can verify the medical metric use and more procedures being refused by doctors.

Wet stuff on the red stuff.

Join us in the Key Players Club

How so, if you care to elaborate?

What's
Important
Now

My wife can lose pay, insurance company payments to hospital, or her job for a certain number of adverse reactions to procedures even one that could safe their life. If they meet so many risk factors she won't do procedure.

Wet stuff on the red stuff.

Join us in the Key Players Club

Of course metrics has changed the way sports are played. NBA, NFL, and MLB look like entirely different sports now than they did 10 years ago.

NBA is all pace-and-space, 3 point shooting around spread pick and rolls or iso-ball as teams realize that the most efficient shots are at the rim, open threes, or foul shots (see Houston)

NFL is way pass-heavy compared to past years as teams realize that changes to rules allow for a more efficient offense when the ball is thrown vs rushed. Coaches are more aggressive on 4th downs and in going for 2 than in years past as the numbers support those decisions.

MLB has seen pitchers adjust approach to be more strikeout heavy, and boom-or-bust hitters are no longer considered liabilities. A shift towards the three true outcomes for a hitter has seen OPS numbers rise while batting averages drop. Also look at pitchers... true workhorse pitchers who gives you 230 innings a year are all but gone and replaced by fireballing starters who throw 90 pitches in 4+ innings and then are replaced by a succession of fireballing one-inning-at-a-time relievers

The big thing with any metric is to figure out why it says what it says. If the "numbers" tell you something different than what your eye does, the question ought to be "why?" rather than making a decision to go with the numbers or gut.

Chem PhD '16

In terms of aggregate data, I'm inclined to say that its had a generally positive impact. For instance, CTE studies as it relates to position and playing time etc. Or injury studies as it relates to surfaces. Efficiency measurements make functional sense, and heatmaps for player movement helps training and positioning across many sports. I'd say its a safe assumption that stat tracking has greatly improved practical training.

For individual statistics, its a slippery slope for sure. Im very interested in sports science and usually when these questions come up I point to AC Milan. Like the moneyball days in Oakland and Boston, AC Milan uses a similar concept for soccer. However, its a historic team that has been pretty mediocre lately, certainly in comparison to a decade ago. Simply put, there is never a magical formula and stats only make sense in context. The combine has a preset amount of 'measurables' for a reason but that's only part of the story.

I know the validity of the CTE study in football, but part of me wants to call bullshit on it and for one reason alone. There are plenty of other sports where head injuries occur, but for some reason scientists and doctors refuse to do the same studies on sports like Hockey, and Soccer (For the headers) etc.

I get it, football has the most head to head collisions, but you need to see how different it is from sport to sport.

What's
Important
Now

I don't think the issue is that doctors and scientist are 'refusing' to cover other sports - they have limited funding, and limited research material. Right now they are focusing funding on high contact sports (Football, ruby, boxing, and there is a fair amount of hockey research actually, but it is being debated more heavily than the aforementioned sports), which makes sense, because these sports provide the most test data. Remember, right now almost all CTE research is from dead bodies, and soccer isn't producing as many cadavers as other contact sports.

All we know for certain is that those who have shown symptoms have had a lot of hits to the head over many years. Until we develop technology/methods to understand how CTE develops in living people, the medical community is stuck studying dead bodies. We do not know how many hits over how much time will result in CTE, but we do know that (to date) everyone discovered to have CTE has had many many years of head hits.

Twitter me

There are CTE studies for other sports, and there was a big one for soccer that found it was pretty common. Worth noting that there is a misconception about headers, the CTE study focused on concussion, which s most often stem from going up to head a ball (head-to-head collisions, elbows to head) and falling and hitting your head, not the act of heading the ball itself. It's possible for young players to get concussions from just heading the ball but not very common.

I've been browsing around here for nearly a year without actually posting, but as a recent master's grad who wrote a thesis about using big data, I feel like this topic is as good as any to actually post something substantive.

The sheer amount of data available now makes it so easy to justify 'x' decision because, essentially, there's bound to be enough data to back it up. I think it presents the opportunity to work backwards by figuring out what you can gather from data instead of deciding what you want to find and gathering the data to answer the question. In the context of sports, I think you see this type of thing happening when execs who don't necessarily understand big data use a small sliver of available data to justify signing/drafting a player who, instinctively, they think is good.

Like the author of this book, I don't think big data/advanced metrics should be the only thing on which decisions should be based by a GM or a coach, but if you're not at least considering them in your decision making process, you're wasting your resources.

Also - Everybody Lies by Seth Stephens-Davidowitz is an excellent book that discusses big data and its implications. Not really sports-focused, but a great read nonetheless.

It's important to use metrics as a tool to understand what is happening, not as the goal itself.

I work as a Manager in software development. We measure defects closely, a reasonable thing to do. When a defect is found, the person(s) responsible for the defect is called on the carpet. I had an extremely talented developer who I turned to frequently for the most challenging assignments. As a result, he had a higher defect rate than my average developers.

Part of our annual review process is a mandatory category of defect rate. So I was forced to give him a bad review in that category. I gave him top marks in every other category, but had to give him the lowest rating in the defect category. If any of the categories rated falls in the lowest rating, it automatically means no raise for that individual. He got mad and quit.

I lost my best developer because of the misuse of metrics.

I'm hoping you went to bat for this guy in upper management.

What's
Important
Now

Oh yes. To absolutely no avail.

We see this sort of thing all the time where I work. Management demands that we have 80% unit test coverage on code and there is a tool that verifies that.

It didn't take developers long to realize that you can build test cases that run the code but don't actually do anything. Management is all happy because they can brag that their team has 80% code coverage, but the tests themselves are useless, and perhaps worse than useless because the tests make you think the code is working when in actuality, the test is proving nothing.

@Test
public void testBuggyCode() {
try {
runBuggyCode();
} catch(Exception e) {
}
}

There, wasn't that easy? Don't you feel confident?

So yeah, you do enough of this and you can get pretty cynical about metrics.

I mean, I have to disagree with him with the knowledge that Time of Possession is an accurate predictor of the quality of a team.

What I think of when I hear about metrics

Never Forget #1 Overall Seed UVA 54, #64 UMBC 74

Wait... so my self-worth isn't determined by virtual turkey legs???

A better indicator is your average turkey legs per post because efficiency stats are so much better than counting stats

Chem PhD '16

Well your self-worth is determined by you, so it's up to you whether that metric supersedes all others...

The issue with metrics is that people measure for the sake of measuring and concentrate only on measuring. They don't know what to measure so they choose pointless metrics for analysis. You have to ask yourself why you're measuring something. What is the goal? What do you intend to do with the metric? A metric without adjustment or action is just a number.

The same point can be made when implementing frameworks. They're intended to provide a basis for doing something. Not intended to implement Carte Blanche by the book without consideration of how your business operates. Implementing a framework without this consideration and trying to fit a circle peg in an oval hole is ignorant and just shows incompetence. Implementing a metric just because "you have to measure something" shows incompetence.

Exactly. It's widespread.

if the data says that you win more nooners than you lose... should you play more nooners?
-ducks and runs-

Go Hokies

No, because we also win more night games than we lose.

I like nooners. Oh wait, we're talking football aren't we? Nevermind.

How about putting the hottest coach in College Football on the Hot Seat for not raising more money, even though he's won 19 games in his first 2 seasons.

Also being upset because he doesn't allow people to watch his practices yet he gets wins which is a good quantitative data to measure.

What's
Important
Now

I just saw this article about UVa using analytics to gain an edge in football.

While I don't think you should base all your decisions on metrics, it seems like VT's engineering school should be able to provide better data than UVa's can. I wonder if anyone at VT is working this...

UVa using analytics to gain an edge in football