How to separate luck and skill

My notes from the book "The Success Equation" by Michael Mauboussin.

These are some of my notes from the book "The Success Equation" by Michael Mauboussin. This book was spotted on Warren Buffett's desk in this tour of his office. There's lots more interesting stuff in the book, but these notes in particular answer the question "How do you separate luck and skill?" We'll start off with some definitions:

Luck is a chance occurrence that affects a person or a group (e.g., a sports team or a company). Luck can be good or bad. Furthermore, if it is reasonable to assume that another outcome was possible, then a certain amount of luck is involved. In this sense, luck is out of one’s control and unpredictable. Randomness and luck are related, but there is a useful distinction between the two. You can think of randomness as operating at the level of a system and luck operating at the level of the individual. Luck is a residual: it’s what is left over after you've subtracted skill from an outcome.

The definition of skill depends on how much luck there is in the activity. In activities allowing little luck, you acquire skill through practice of physical or cognitive tasks. In activities incorporating a large dose of luck, skill is best defined as a process of making decisions. Here, a good process will have a good outcome but only over time. Patience, persistence, and resilience are all elements of skill.

Separating luck and skill

At the heart of making this distinction lays the issue of feedback. On the skill side, feedback is clear and accurate, because there is a close relationship between cause and effect. Feedback on the luck side is often misleading because cause and effect are poorly correlated in the short run.

In most cases, characterizing what’s going on at the extremes is not too hard. As an example, you can’t predict the outcome of a specific fair coin toss or payoff from a slot machine. They are entirely dependent on chance. On the other hand, the fastest swimmer will almost always win the race. The outcome is determined by skill, with luck playing only a vanishingly small role.

On the far right of the luck-skill continuum are activities that rely purely on skill and are not influenced by luck. Physical activities such as running or swimming races would be on this side, as would cognitive activities such as chess or checkers. Marion Tinsley, the greatest player of checkers, could win all day long, and luck played no part in it. He was simple better than everyone else.

On the far left are activities that depend on luck and involve no skill, like roulette or the lottery. Also on the left side are things like investing, business strategy, and poker. It doesn’t mean that skill doesn’t exist in those activities. It does. It means that we need a large number of observations to make sure that skill can overcome the influence of luck.

There is a quick and easy way to test whether an activity involves skill: ask whether you can lose on purpose. In games of skill, it’s clear that you can lose intentionally, but when playing roulette or the lottery you can’t lose on purpose.

Reversion to the mean

Consider the relationship between where the activity is on the luck-skill continuum and the size of the sample you are measuring. A small number of results tell you very little about what’s going on when luck dominates. And in an activity where the results are nearly all skill, you don’t need a large sample to draw reasonable conclusions.

An understanding of where an activity is on the luck-skill continuum allows you to estimate the likely rate of reversion to the mean, and vice versa. Any activity that combines skill and luck will eventually revert to the mean. If what happens is mostly the result of skill, then reversion to the mean is scant and slow.

The position of the activity on the continuum defines how rapidly the outcome goes toward an average value, that is, the rate of reversion to the mean. In activities that are all luck, there is complete reversion to the mean. So if you estimate the rate of reversion to the mean, you can help place an activity on the luck-skill continuum.

Estimated true average = Grand average + shrinking factor (observed average - grand average)

The shrinking factor is the rate of reversion to the mean. For activities that are all skill, the shrinking factor is 1.0, which means that the best estimate of the next outcome is the prior outcome. For activities that are all luck, the shrinking factor is 0, which means that the expected value of the next outcome is the mean of the distribution of luck. So we can assign a shrinking factor to a given activity according to where that activity lies on the continuum.

Francis Galton’s insight was that reversion to the mean and correlation are two elements of the same concept. So in the above equation, shrinking factor usually approximates correlation. A high correlation suggests weak reversion to the mean. The most obvious hazard is that correlations are not stable in many fields. In unstable and nonlinear activities, relying on past correlations won’t work in the long run.

Placing Activities by Answering Three Questions

1. Cause and effect. First, ask if you can easily assign a cause to the effect you see. In some activities, the relationship of cause and effect is clear. You can repeat behavior and get the same result. If you can easily identify the cause of a given effect, you’re most likely on the skill side of the continuum.

Consider two elements of a manufacturing business. The first is the actual manufacturing process. This is an activity that falls near the all-skill side. A proper process using statistical control yields a favorable outcome a very high percentage of the time. The second element is simply deciding which products to manufacture. We call this strategy, and even a well-conceived strategy can fail catastrophically. So even with the same company, some activities will rely mostly on skill and others on luck.

2. The rate of reversion. Very simply, what is the rate of reversion to the mean? To answer this question you need some way to measure performance.

3. Where prediction is useful. Where can we predict well? In other words, where are experts useful? Areas that have high predictability include engineering, some areas of medicine, and games such as chess and checkers. Experts are notoriously poor at predicting the outcomes of political, social, and economic systems. Complex adaptive systems effectively obscure cause and effect. You can’t make predictions in any but the broadest and vaguest terms.

Variance (observed) = Variance (skill) + Variance (luck) Skill = Observed outcome - luck

If you can calculate the standard deviation of the observed outcomes in an activity, and what the standard deviation would look like if luck were the only factor, you can estimate the relative amount of skill and luck involved (luck/observed = amount of luck involved).

Performance measurements

Useful statistics have two features. (1) Persistence, which means what happens in the present is similar to what happens in the past. This is also called reliability—if luck is more important, than you would expect the reliability to be low. (2) Predictiveness of the goal you seek. This is also called validity, meaning it is valid to conclude that one measurement causes another. It compares two values. In basketball, an example would be percentage of shots made and total points scored.

Statisticians asses persistence and predictive value by examining the coefficient of correlation. The process of determining which statistics are useful begins with a definition of your objective. Knowing your objective is important because it’s hard to chart a course without knowing the destination. Next, you have to determine what factors contribute to achieving your objective.

You can plot a measurement on a chart that has the luck-skill continuum as the horizontal axis and the predictive value on the vertical axis. The most predictive and persistent measurements are best. (Examples include on-base percentage for baseball, or active share for investing.)

But while a skill is persistent, it is not always the case that persistence reveals skill. A careful consideration of what elements of performance are within the command of an individual or a sports team is essential.