Sunday, October 20, 2013

Getting an Edge: Breaking down the unbiased LCS numbers

Amid accusations of umpire bias during the League Championship Series, both National and American, the discerning fan turned to objective analysis: If the Cardinals truly were gifted umpire calls over the Dodgers, the numbers would show it, the same principle conceivably holding true for the Red Sox and Tigers.

For better or worse, the numbers show that the Cardinals experienced a net advantage of +20 pitches during the 2013 NLCS vs. Los Angeles; the Red Sox +12 during their ALCS vs. Detroit.

In the grand scheme of this type of analysis, league average theoretically and logically should be a net neutral (plus zero) result, that's what average is all about—yet St. Louis, for instance, came away with 20 calls that a simply complex piece of technology says should have gone the Dodgers' way based on the rules book definition of the strike zone. For the purpose of this discussion, we concentrate on the NLCS, ultimately concluding that bias did not factor into the series, despite St. Louis' clear pitch calling advantage.

At this point, one might be tempted to oust the overt "bias" inherent with a 20-call swing—especially if one bleeds blue. It certainly doesn't help that on Game 6 eve, the media churned out the report, "NLCS umpiring crew not pleased with Puig." Puig, of course, is a Dodger, as is 2012 ejections leader Don Mattingly.

Yet in this post-Tim Donaghy era of officiating to the tune of Pitch f/x and Zone Evaluation precision, could, as the Twitterverse invariably suggests every night, an umpiring agenda be to blame?

The first step to proving anything with numbers is inevitably to gain a thorough understanding of what those numbers represent—We know St. Louis had a net advantage of 20 pitches, but what does that mean?

It means simply that pitch f/x determined a pitch thrown by a St. Louis pitcher called a strike was located out of the strike zone OR a pitch thrown by a Los Angeles pitcher called a ball was located in the zone twenty more times than an LA pitch out of the zone was called a strike or a STL pitch in the zone was balled.

It has nothing—yet absolutely everything—to do with catcher framing analysis, which is our next step to understanding the +20 conundrum.

In 2013, the St. Louis Cardinals finished first amongst MLB teams with 24.07 real runs added due to catcher pitch framing (getting the extra strike on pitches out of the zone [+]). Meanwhile, the Dodgers finished 23rd with 14.51 runs lost due to catcher pitch framing (pitches in the zone called balls [-]).

To further the point, Dodgers ace Clayton Kershaw's "robo-ump" or league average catcher WAR increases from 6.5 to 6.7 when framing is neutralized while Cards ace Adam Wainwright drops from 6.2 to 5.8.

More to the point, catcher performance results compiled by Mike Fast—albeit to a bench mark of Fast's strike zone maps which deviate from the "rule book" zone—found that in 2011, the Molina brothers (namely Jose, but Cards catcher Yadi as a rising star) were the epitome for "best"—Yadi's net extra strikes-per-pitch rate was +.007 while LA catcher A.J. Ellis floundered near that "worst" benchmark, drawing a -.022 rate for those years studied.

Because the value of extra strikes per pitch considers the likelihood of an extra strike being called per thrown callable pitch and not just borderline pitches, similar to the overall UEFL accuracy method, we consider all 937 callable pitches during the Dodgers-Cardinals series.

Multiplying 937 by the net .015 rate (.022-.007)—the series was exclusively caught by Molina and Ellis—suggests the Cardinals should have netted an extra net 14.055 ball/strike calls, which is just six short of the +20 swing actually recorded.

To determine whether this difference of six is simply due to chance or is significantly significant, we create a 95% confidence interval. The interval is (8.34, 23.33), suggesting that the +20-pitch swing is not statistically significant and the deviation from 14-to-20 is due to chance.

In other words, the umpires' "missed calls" at the plate during the Dodgers-Cardinals series were not out of the ordinary and were due to chance—the numbers suggest bias did not play a role during that series.

All of that said, framing makes its appearance, most notably, in how a catcher receives a pitch. For instance, Xander Bogaerts' controversial 7th inning walk versus Max Scherzer on Saturday night resulted from a 3-2 breaking ball over the outer edge of home plate, at the knees, that catcher Avila received poorly—by (1) moving his glove and reaching out over the plate and (2) pulling the pitch back towards the center of the strike zone, Avila gave the impression that the pitch was worse than it actually was.

Similar poor receiving by a catcher occurred during Game 6 of the NLCS when Kershaw threw two consecutive pitches within the rules book strike zone that were balled after Ellis made the pitches look worse than they actually were. A rousing double followed the ensuing 2-0 (instead of 0-2) pitch.

There are two schools of thought: The prevailing attitude is to reward the proper pitches—both thrown and caught well—and to make the "expected call" when possible. The second is that umpires must call the rules book strike zone in spite of a catcher's potentially poor performance.

This is what happens when a butchered catch, located within the strike zone, is called a strike: Jim Joyce calls Wilson Ramos out on an inside corner pitch the catcher didn't see coming
Another instance of a catcher's too-eager sell job on a strike, Umpire Kulpa properly rules K3 regardless / Ted Barrett ejects Reyes after strike three at the bottom of the zone is pulled back up

More video: CB Bucknor ejects Fredi Gonzalez on a beautifully caught pitch off the plate called a ball
And a failure to hit a spot pitch on the corner improperly ruled ball four


Gil Imber said...

Good article and nice statistical analysis as well.

Gil Imber said...

your work continues to astound me! Thanks for scientifically backing me up!! You da man!!

Gil Imber said...

I'm not a professional umpire, but I do umpire and attended one of the Florida camps. My thoughts on pitches that are caught poorly: if the catcher reaches all the way across his body or pops all the way up its a ball no matter what; if it goes back to the screen its a ball no matter what; if it bounces off the mitt and its borderline its a ball; any other situation where he catches it funny I call it a strike if I'm pretty sure it caught a legit piece of the zone. The "ball to the screen" is just because its such a horrible baseball play to let that happen to a strike. The rest of it is because I'm not good enough to make those calls with accuracy. The Jim Joyce link was impressive.

Gil Imber said...

As much as I like to slam umpires for mistakes, there really haven't been any bad calls in the post season. They are all doing quite well.

Gil Imber said...

Except that the Joyce pitch was inside all the way. Check Pitch FX on Brooks.

Gil Imber said...

Paul Emmel got the series !! AWESOME !

Gil Imber said...

Along with Hirschbeck, DeMuth, Wegner, Joyce and Miller

Gil Imber said...

Why do you like to slam the umpires?

Gil Imber said...

I actually called a strike that the catcher missed and it went to the back stop. The catcher set up way outside (this is house league so not the best players) and the pitch went to the other side of the plate. The catcher missed the ball, but I saw that it was painted on the inside part of the plate. So I called a strike. To me, what is a strike is a strike and what is a ball is a ball. The catcher framing just makes it more obvious and easier to call.

Gil Imber said...

More of a concern to me than the +20 is that in 3 of the 6 games the plate umpire was below 92%. That includes Mark Carlson's performance in Game 2 which was, by far, the worst of any umpire in the playoffs. By this site's own admission, the standard this year for judging correctness causes more borderline pitches to be deemed "correct" this year when they were not "correct" last year. Looking through plots from last year versus this year the number of pitches that fall into this category result in approximately a 2% bump (give or take) versus last year

Gil Imber said...

Molina must have been really on in his "framing" that it's okay for him to miss all of those pitches

Gil Imber said...

I like your logic but let me be a realist for a second. At your level (and almost every level) - there are no instant replays and the pitch moves much faster than the catcher. What most fans and people on the bench will see is the catcher leaning, falling over, turning the glove, pulling the glove back into the zone or other "dead giveaways" that the pitch was a ball. Was the pitch a ball? Maybe, maybe not - but if you make it a habit of calling pitches that everyone can tell is a ball (because of the slower moving - more obvious catcher) strikes, you are going to make your life miserable and negatively impact the game as the focus will shift from baseball to your "really bad" strike zone.
Just my thoughts but thinking that way has helped me.

Gil Imber said...

This site also adds the diameter of the baseball (as opposed to the radius) to the edges of the plate even though pitch f/x data represents the center of the baseball... Not a huge difference because of the margin of error but I've never understood why that was the case here

Gil Imber said...

I agree, that pitches that don't look like strikes shouldn't be called strikes, I do higher level games as well, and for that I do the mlb style of calling strikes, but for the little kids, they have a hard time pitching and a harder time catching so do you expect me to call every pitch a ball because they can't catch?

Gil Imber said...

That's what I used to think, until I was doing a game once where I called one of those. My Dad who used to umpire was there, and he told me it missed by well over a foot. I think the optical illusion comes in when you're tracking the ball with your eyeballs, when your eyeball moves that much the muscle pulls it back to center a little bit. Try this: wherever you're sitting, look at an object in the room out of the corner of your eye without turning your head. Point at the object. Okay, now are you really pointing at it?

Gil Imber said...

I won’t get into too much detail the problems with aggregating data and rates, but
after reading this I decided to analyze the raw data myself. First off, I am a
Dodgers fan. I do not believe that the call differential was the result of any
malicious umpiring bias, but the pitch differential should be unacceptable
regardless. From the data on initial pitch release
(x0,y0,z0,vx0,vy0,vz0,ax,ay,az) and we can calculate pitch movement. Pitchf/x
data represents the center of the baseball as it crosses the front of the
plate. Combining these two allows for us to conclude that some called strikes
that were borderline out of the zone in the raw data likely entered the zone
after crossing the front of the plate. The Cardinals are the biggest beneficiary
of taking this into account, which led me to find the total series differential
to be +15 instead of +20.

Given all that, I found the results to be:

Ordered pair is: (Incorrect Called Strikes for, Incorrect Balls called

Game 1:

Dodgers (7,6)

Cardinals (3,4)

Net Result – Dodgers +2

Game 2:

Dodgers (3,4)

Cardinals (9,3)

Net Result – Cardinals +7

Game 3:

Dodgers (3,4)

Cardinals (2,2)

Net Result – Cardinals +1

Game 4:

Dodgers (3,6)

Cardinals (3,1)

Net Result – Cardinals +5

Game 5:

Dodgers (5,2)

Cardinals (2,0)

Net Result – Dodgers +1

Game 6:



Net Result –
Cardinals +5




Net Result –
Cardinals +15

Now, lets
determine the game by game extra strike rate for AJ Ellis and Molina

Game 1

Ellis: + .0097

Molina: - .0067

Game 2

Ellis: - .0167

Molina: + .0896

Game 3

Ellis: -

Molina: 0.000

Game 4

Ellis: - .0341

Molina: +

Game 5

Ellis: + .0411

Molina: + .0303

Game 6

Ellis: -

Molina: +




Clearly, the
call differential cannot simply be attributed to framing, if anything it shows
Ellis’ performance rose to the occasion of playing in a NCLS series. Not to say
he was never sloppy, but on overall his framing only resulted in a 3 strike
loss for the series. Molina netted 12 strikes, which is well above what would
have been expected based on a .007 extra strike rate (456 called pitches for
Cardinal Pitchers x .007 = 3.192 extra strikes). The obvious aberration was game
2, which was a critical point of the series, and was absolutely butchered by
Carlson, NOT superbly framed by Molina.

Bottom line
is that although I’m 99% sure this was not some intentional screw job by the
umps, this series was called poorly and the whole series but especially game 2
was not within the expected range because that confidence interval was based on
aggregate performance (together they performed as expected but individually
they both outperformed expectation, resulting in an expected aggregate value),
and with a questionable standard deviation value.

Gil Imber said...

I'm probably not getting any responses because by the time I posted all of you just moved on, but I thought I might get a response from whoever posted this in the first place, as that person has at least a basic familiarity to statistics and might understand my objection to the analysis here...

Maybe my wording is unclear when saying the aggregate was expected but individual performance was not. The net ~+14 pitches was based on the expectation that Yadi would net ~ +3 pitches, and Ellis would cost ~11 to the Dodgers. When you combine them that's ~+14 for the cardinals. But what actually happened was Ellis cost the Dodgers ~3 pitches, or about %27 of what would be expected, and Yadi netted +12, or about 400% of expected. Combining their rates made it look like it was reasonably close to the expected outcome. As an analogy, let's say you were making a 300 mile trip in an old Geo, which tops out at 60 mph, so you expect the fastest you can get to your destination is 5 hours. Unexpectedly you find yourself driving into 100 mph winds, but your Geo manages to inexplicably go 61 mph the entire way so you make it in just under 5 hours. The net driving time was in the expected range, but actually your car went much faster than expected, were it not for the wind also being much faster than expected.

Post a Comment