Statistics will be at the heart of the new CCGs. But how far can you rely on the information you are being given? Here’s a mathematics-free guide. All you need is logic.
Firstly, we need two important definitions: data are the raw facts and figures; information is knowledge derived from interpretation of those data.
Good statistical practice depends on having data which is representative and accurately measured; from which we get information which is correctly derived and logically inferred; which is then appropriately presented.
Are the data accurate?
Clearly, each piece of data has to be measured carefully and precisely.
When using a sample, how accurately does it reflect the larger group from which it is taken? Clearly, the bigger the sample the more accurate it is likely to be – but even so, if it has inadvertently excluded certain groups of people, any conclusions drawn from it may be completely false.
It’s obvious that your sample has to include all social groups, all ethnic groups and all ages. But has your sample been subtly biased because of the way it was selected – for example, only measuring those who attend a specific clinic (thus excluding the housebound); or excluding those on night shifts (because it was done during the daytime)?
Does it inadvertently omit those without internet access; who have no fixed abode, or were in hospital at the time? What about those who can’t read or speak English, or too busy to respond? Have you unintentionally excluded a complete group – for example, by choosing a survey date that is a major holy day for a particular religion?
If you are trying to measure all contacts, rather than just a sample (for example, everyone coming to a particular clinic) how complete is your survey? Are telephone or email requests going unmeasured?
If you are conducting a survey, are the questions slanted? Are they ambiguous? Do they allow contrary opinions? Can ‘Don’t know’, ‘Inapplicable’ or ‘I don’t want to answer’ be recorded? And – the biggest problem of the lot – those with an axe to grind are more likely to complete the survey, which will therefore return a skewed result.
Is the information correctly deduced?
Obviously this starts with having good data. Remember the GIGO principle — Garbage In, Garbage Out. You can never extract good information from duff data.
Never use data for a purpose other than for which they have been collected, especially when trying to create scores or rank people or institutions. It’s a golden rule of informatics, but frequently ignored.
QOF is an excellent way of paying practices for work done, but it can’t be used to rank them. Why not? Because some practices have no patients in relevant cohorts (e.g. Down’s syndrome), and so cannot earn those points. The playing field is not level.
Is what you are actually measuring a true proxy for what you want to measure? As Einstein said, ‘Not everything that matters can be measured: and not everything that can be measured matters.’ Just because we have data doesn’t of itself signify that this data means anything.
We want to identify ‘the good doctor’. But no parameter truly measures this concept, even though we all know what it means. Instead we have to rely on proxy measurements – patient satisfaction ratings, referral rates, prescribing costs, number of episodes of unscheduled care – without questioning whether these figures actually measure ‘the good doctor’. (I can have high satisfaction ratings by being smooth and personable, even if I practise execrable medicine. I will have high prescribing costs if I use expensive medicines to keep my patients out of hospital. I will have high referral rates if I’m alert and spot more disease.)
Is like being compared with like? That poorly scoring practice in your locality – it may not be the clinicians’ fault. Does it have more deprivation, a higher number of non-English speaking patients, an inability to find locums, no room to expand, or three partners on long-term sick leave?
Be wary of percentages
Be wary of percentages. Firstly, you can’t average averages – mathematically it’s completely erroneous.
When assessing percentage changes, consider the baseline. With small populations a single event may have a disproportionate effect. A practice may be unfairly red-rated because immunisation of 2-year olds is only 50% – but what if it’s a tiny practice with only two children in the age group, and one mother refuses permission?
Check the baseline. Similarly, if you want to look good, pick a particularly bad moment for your baseline. If you only did two routine health checks this month that’s a great time to start! You can then proudly show three checks the next month as a whopping 50% improvement (even if you really should have been doing 30…)
Watch your interpretation of averages and percentiles. As the joke goes, ‘I wont rest until all my employees are above average.’ (How?) Similarly, there is always someone in the lowest centile, even in a universally high-scoring group. Don’t make the mistake of assuming that ‘below average’ means bad. It may do, but it doesn’t have to.
Is the information being presented appropriately?
For a great take on this, read ‘How to lie with statistics’ by Darrell Huff (Penguin Business). See how ‘gee-whiz graphs’, which cut off the blank space at the bottom, magnify the apparent gradient of the graph, making improvements look disproportionately good.
Discover how to choose the type of average (mean, mode or median) to support your own arguments ‘statistically’. Use percentages and baselines ‘creatively’. Once you’ve learned how to do it maliciously you won’t be fooled so easily in the future.
I promised you a maths-free encounter with statistics. You need only one talent — razor-sharp logic, together with a lateral-think approach that seeks out each and every possibility that the data and the information might not represent quite what they purport to. And that’s the way to look at healthcare statistics.
- Dr John Lockley is a GP in Ampthill, Bedfordshire and Clinical Lead for Informatics at Bedfordshire CCG