Sample size, Jamie Vardy, Paul Pogba, and player evaluation

Remember back in the summer, when Sign Jamie Vardy was both a thing that almost happened and a thing that a ton of Arsenal fans desperately wanted to happen? Well, as seen in this morning’s Cannon Fodder:

Before we start, I want to be clear on what this article is not. This article is not JAMIE VARDY SUCKS, and this article is not I TOLD YOU WE SHOULDN’T SIGN JAMIE VARDY, mostly because while I wasn’t desperate for it to happen, I wasn’t really opposed to it either. What this article is is a discussion of a concept that a lot of sports fans forget: sample size.

There is a great reticence to apply the statistical revolution that has taken over most other sports to soccer, and that’s understandable. Soccer, with its almost complete lack of discrete game states, is really, really hard to quantify accurately, and it is also very hard in soccer to separate play from production, which makes any attempt to place an objective value on production extremely difficult.

That’s another thing this article isn’t, though - a condemnation of statistical analysis. As a sports fan and alleged writer, I come from a background of baseball analysis. I wouldn’t call myself a sabermetrician, necessarily, because I don’t actually further the work of those who do that sort of thing at all, but I embrace what they’ve done and I understand the ways in which what they’ve done have deepened people’s understanding of what matters in baseball and why.

And, while I’m not going to try to do that here - for the above reasons and others, I’m not sure I’m ready to embrace statistics in soccer in any significant way - I am going to try to reinforce one concept that is portable to soccer, if only in an anecdotal way.

One of the cardinal sins of the baseball analyst, or of any analyst in any field, is looking at too small of a pile of data when drawing or attempting to draw a conclusion. The concept of ‘sample size’ is critical, because using enough data filters a lot of noise out from the signal that’s trying to get through.

For example, in baseball, assuming good health and every day play, a typical offensive player will get approximately 650 plate appearances in a season, and depending on the statistic, it can take anywhere from 250-400 PA’s for a statistic to be meaningful.

The same applies to entire teams and seasons. Baseball stats people calculate something called “park effects”, which, since every ballpark is different, tries to express whether a particular ballpark is better for pitchers or for hitters. This effect is measured over, at minimum, three seasons, because there are so many variables that come into play when determining that (home team roster construction, away team roster construction, weather, etc) that a single season’s worth of games just isn’t enough to draw a meaningful conclusion.

It’s with that in mind that I turn back to Jamie Vardy. And to Paul Pogba. And to any number of players who, after one fantastic season, are the object of many fans’ PLEASE SIGN HIM RIGHT NOW pleas.

Let’s look at Jamie Vardy’s career, shall we? For purposes of this, I’m limiting that look to his time with Leicester, because I’m only thinking about Premier League- or Championship-to-Premier-League level transfers and performance; he spent three years in the Conference, but those aren’t really germane to talking about top-level performance so I’m excluding them.

In his time at Leicester, Vardy has played two seasons in the Championship and three (including the current season) in the Premier League. In that time, his appearances and goals were:

2012-13, Championship: 29 appearances, 5 goals
2013-14, Championship: 41 appearances, 16 goals
2014-15, Premier League: 36 appearances, 5 goals
2015-16, Premier League: 38 appearances, 24 goals
2016-17, Premier League: 17 appearances, 3 goals (extrapolates to ~7 goals in 40 appearances)

In and of itself, those five seasons say “I’m inconsistent!”, which I don’t think is necessarily inaccurate, or necessarily damning or bad. But, the other thing that it says is that, in five seasons, Jamie Vardy has been scoring at a rate of just over three goals per season (3.22, to be precise).10 goals per season; using the above numbers, Vardy has averaged 37 games/season and 10.6 goals/season (thanks to everyone who has pointed out how bad at math I am. You’re not wrong).

(I won’t repeat the whole analysis for Paul Pogba here, but I will say that extrapolating his complete 2016-17 as I did for Vardy, he’s averaged 4.8 9.2 goals/season in the last five seasons.)

So, the question then becomes: Do you value the Jamie Vardy that scored 24 goals in a single season more than you value the Jamie Vardy that averages three 10 goals per season over his career?

The reason I tend to lean more towards the latter view than the former is also the reason I can’t really get into serious statistical analysis of soccer. Jamie Vardy’s 24 goals last year were the result of a completely serendipitous set of circumstances. He’s a good-not-great player on a team full of guys who played out of their minds all year, and who for the most part stayed healthy, and he and they rode that health and good form to a championship. Don’t misunderstand; I’m not damning him with faint praise, that’s not nothing and the season LCFC had was one for the ages.

But that’s also why it’s hard for me to pull Jamie Vardy out of 2015-16 into a new team this season and expect him to plonk in 24 goals again, in a different setup with a different team. I mean, he’s in largely the same team he was last season, with the exception of N’Golo Kante, and he’s already eight goals behind where he was production-wise last year.

Does that make him a bad player? Absolutely not. But it highlights how important the team a player surrounds himself is when trying to evaluate a player, and how it may be wise to take the long view of a player’s skills when deciding whether you want him to sign for your favorite team after one good season. There’s so much that goes into that “one good season” that isn’t necessarily isolatable to an individual player that I hesitate to ever want a player just because they had that blind squirrel of a season.

Clearly, owners prize recent form, and are willing to pay great sums of money for it; and in a league with no realistic salary constraints that’s not necessarily a huge issue. I just get a little frustrated when a player - again, this is about any player with one good season, this isn’t just about Jamie Vardy - becomes the latest recency bias fan fetish without the fans looking at the whole picture of his career.