The biggest challenge with Key Metrics-Goodhart's law

As managers, one of the most important jobs we have is to measure things. We need to measure code quality, test coverage, team velocity, team morale, and much more. Most of the time we deal with things that can never be measured, but we keep finding ways to measure and improve.

“What gets measured, gets improved”, Peter Drucker

It is crucial for your Team’s success to define, track and act on Lead Measures to achieve your goals. There are challenges in defining Lead measures. Generally, they are not intuitive and it is not easy to collect data for Lead Measures. But one of the key challenges with Lead Measures is defined as Goodhart’s Law[1], according to which, “when a measure becomes a target, it ceases to be a good measure”.  When you define some metrics to measure the success of a System, people tend to optimize for the metric instead of a goal. Here are some of the examples from history:

When the British ruled India, bureaucrats in Delhi grew concerned about the proliferation of cobras in the city. To get the problem under control, authorities offered a bounty on cobra skins. However, soon they found out that several enterprising Indians had started breeding cobras to eventually kill them and sell their skins.[2]

Intending to increase the number of accounts sold, Wells Fargo in 2016 introduced overly ambitious sales goals to be met by their employees. As result, facing the threat of losing their careers if such quotas were not met, some employees began to open large numbers of unauthorized accounts.

In building the first transcontinental railroad in the 1860s, the United States Congress agreed to pay the builders per mile of track laid. As a result, Thomas C. Durant of Union Pacific Railroad lengthened a section of the route forming a bow shape unnecessarily adding miles of track.[3]

If you set a goal to add more Tests, you will see lot more tests being added. Not necessarily improving the test coverage. If you set a goal to reduce P0 bugs, you will see a lot of P0 bugs getting downgraded to P1, P2s, and never getting fixed. Here are some of the strategies you can use to guard against Goodhart’s law:

Double Level Thinking: When you define metrics to track your goals, go one step further and purposefully think “How these metrics can be hacked?”. Based on your analysis improve your metrics. To improve test coverage, instead of measuring the number of tests being added, you might want to track the number of defects escaping your pipeline.

Multiple Metrics: In most cases defining a single metric that can not be hacked will be really hard. You can define more than one metric to track the same gaol. Hacking multiple metrics is much harder than a single metric. To improve product quality you can track code coverage, escaped bugs, and customer issues.

Collect additional data points: Define processes to track additional data to prove the validity of your metric. It will be helpful to track the bugs being downgraded from P0 to P1. Reviewing this list at a later point can help you find if your metric is being hacked.

Retrospect: Retrospect and question your metrics on regular basis. Review your data, make sure the metric is still relevant and be flexible to change it.  Define a specific cadence to review your metrics. A quarterly review of metrics works well in most cases.

The most important thing that you can do to guard against Goodhart’s law is, to make sure every member of the team understands, is aligned and feels passionate to achieve the ultimate goal. It all comes down to creating a culture of high values and maintaining it when your team grows. You can use all the above strategies to keep your metrics meaningful but people will eventually find a way around these. Unless you work on changing the culture and shifting the mindset to get to your ultimate goal.

“Instill the Goal in each team member”,  WhiteBeltThinker

[1] Goodhart, Charles (1975). “Problems of Monetary Management: The U.K. Experience”. Papers in Monetary Economics. 1. Sydney: Reserve Bank of Australia.

[2] “The Cobra Effect: Good Intentions, Perverse Outcomes”. Psychology Today. Retrieved 29 March 2021

[3] Mark Zwonitzer, writer, PBS American Experience documentary “Transcontinental Railroad” (2006) “Program Transcript. Transcontinental Railroad. WGBH American Experience”