I’ve made any number of mistakes in my time as editor of the JC. But the one I am most ashamed of was when I asked reporters to stand in the street and ask passers-by to answer a series of questions. We spoke to around 150 people and then presented their responses as a “straw poll”, with percentages reported for each answer.
Let me be charitable to myself: I was clueless. The news report was rightly savaged as misleading and wrong. But it was an important lesson to learn, because it taught me just how dangerous a spurious poll can be — and how important it is always to check the methodology of any poll result.
Last week, a poll by researchers at Oxford University examining conspiracy theories around the spread of coronavirus was published, generating all sorts of lurid and worrying headlines. Newsweek headlined its story: “One Fifth of English People in Study Blame Jews or Muslims for Covid-19.” You probably saw a similar story and, like me, were jolted when you did.
Except the ‘poll’ is pure, unadulterated rubbish. It is utter nonsense from start to finish.
The Newsweek report began: “Nearly half of people in England believe in conspiracy theories about Covid-19, according to a survey carried out by psychologists at the University of Oxford”.
In one sense that story was entirely accurate. It was indeed what the survey found — that a fifth of respondents agreed with the statement that “Jews have created the virus to collapse the economy for financial gain.”
But in a more meaningful sense the story was bunkum — because the survey which it reported was bunkum.
The findings, “Oxford Coronavirus Explanations, Attitudes, and Narratives Survey”, put together by the National Institute of Health Research’s Oxford Health Biomedical Research Centre at the University of Oxford, were from a self-selecting online questionnaire carried out between May 4 and 11. Around 2,500 people replied to it.
A total of 5.3 per cent said that they “agree a little” with the statement about Jews; 6.8 per cent said that they “agree moderately”; 4.6 per cent said that they “agree a lot”; and 2.4 per cent said that they “agree completely”.
There is a basic rule of thumb with polling — if a result is startling it is usually wrong. The findings that almost half of people in England believe in conspiracy theories about Covid-19, and that nearly a fifth believe ‘the Jews did it’ are indeed startling, not least because no serious and properly conducted poll comes even remotely close to replicating this.
The reason is that the questionnaire could have been designed to come up with such outrageous findings. At the most basic level, respondents were self-selecting. In other words, those people most likely to have fringe views on the subject were those most likely to respond, because they were especially interested in it.
Indeed, the organisers confirmed that one of their recruitment methods for responders was “word of mouth”. So if you think ‘the Jews did it’, you told your antisemitic friends about the survey.
The researchers say they adjusted the results to reflect population quotas but this is risible: the fact remains that self-selection cannot other than bias such a questionnaire. Or, to put it more basically: nutters’ views are always going to be over-represented.
On a more technical level, for most of the questions the sliding scale of responses involved 4 positive responses and only 1 negative (ie agree a little, agree moderately, agree a lot, agree completely, do not agree). This looks like a survey determined to produce a finding that shows support for conspiracy theories. Any serious survey balances out positive and negative responses (usually two positive, two negative and one neutral).
Does it matter what one silly survey says? It matters a lot. I would bet my mortgage on the spurious statistics from this poll being cited for years. The results may be drivel but who stops to check out the methodology behind such statistics? Only people who know about statistics, and those like me who have — ahem — been bitten themselves by a fake poll.
