Over the last few weeks, our little friend Stick Child has been doing a really interesting school project about a thing called variation. Not everyone knows about variation. His project is called:
â€œUnderstanding Variation (For the Very Young or Pathologically Resistant)â€.
Stick ChildÂ has learnt some pretty cool stuff during his project,Â which he thinks might be useful for grown-ups who struggle to use numbers properly when it comes to trying to understand performance information.
Hereâ€™s what heâ€™s been up toâ€¦
Armed with clip-boards and pencils, Stick Child and his friends have been standing outside their school and counting the number of red cars that drive past on different days. First of all, Stick Child ended up with a nice tally chart like this…
…then he used the daily totals to make an even nicer control chart like the one below:
As you can see, there were different amounts of red cars each day. Stick Childâ€™s teacher showed the class where the special dashed lines belong on the chart, and explained that in this case, every â€˜Xâ€™ between them is completely normal because of this thing called â€˜variationâ€™. This means it’s a mistake to assume there’s any meaning behindÂ different individual values, as well as aÂ big waste of time trying to find out why one is different from another.Â (If you have difficulty sleeping and want to know how the lines are calculated, there is a step-by-step guide in Stick Childâ€™s favourite bedtime book).
Next, Stick Child did some experiments with his chart. First of all, he randomly picked one of the numbers between the lines and called it a target. Then he tried to work out why sometimes the number of red cars hit the target and sometimes it didnâ€™t.
Next, he tried to make the target influence the number of red cars by shouting at the chart and / or the cars as they drove past, but that didnâ€™t work either.
Then, he said he would give 10p to one of his friends if she was able to make the target work. At first this didnâ€™t make any difference, but laterÂ his friend said the target had been met. Stick Child lookedÂ closely at her chart and discovered that she hadÂ altered some ofÂ the numbers on it, so he told the teacher and kept his 10p.
Next, Stick Child picked random previous daysâ€™ totals on the chart and drew arrows between then to try and work out if the number of red cars was increasing or decreasing. Unfortunately, this just caused confusion because he got a different result every time; he quickly determined that making such binary comparisons was rubbish, so stopped doing it.
Finally, Stick Childâ€™s teacher timed how quickly the children had drawn their charts, then ranked them in a league table. Then she told the children that half of them were below average. None of the children could understand why she would do this, as they had worked very hard on their projects. It made them feel sad.
After a minute, the teacher told them this was actuallyÂ just part of the lesson and that really she was very pleased they had done their best, because this is what really matters. Stick Child and his friends were glad that this silly way of assessing performance would never actually happen in real life.
What Stick Child Learnt
The project taught Stick Child and his friends lots of useful things. He learnt that there is no point worrying about why theÂ total numberÂ of redÂ cars was different on different days â€“ this happens because of that thing called variation. HeÂ found outÂ that unless systems conditions change (e.g. due to a road closure), those little â€˜Xâ€™s will continue to appear anywhere between the dashed lines.
In addition to this, Stick Child and his friends learnt thatÂ targetsÂ don’t make any difference to the amount of red cars that drive past his school. This is because variation doesn’t pay any attention to man-madeÂ follies, suchÂ as numerical targets. He remembered his Dad always says that numerical targets are arbitrary and likely to cause dysfunctional behaviour.
Stick Child also discovered that there is no point drawing arrows between two isolated numbers because it gives the impression of trends that simply do not exist. Finally, heÂ learnt that league tables are a poor way of assessing performance. For his efforts, the teacher sentÂ a nice letter to his parents.
Even though he is only nine, Stick Child knows that control charts do notÂ just apply to red cars or school projects.Â They can also be used by grown-ups, for things like crime figures, response times, or almost any other set of numbers you might want to learn about. (That’s if the grown-ups really want to learn about these things).
If he understands this, so can you.