Friday, July 29, 2016
Dear CNN and every other lover of non-scientific polls:
I beg you to please realize the degree to which you undermine your mission as organizations committed to objective reporting with such meaningless and un-interpretable data.
The sampling required for a scientific poll requires a remarkable art. When someone wants to take the pulse of a group, be it eligible voters, likely voters, neo-Nazis, or parents of two year olds, sampling statisticians go to great lengths to deploy their umpteen degrees and massive intellects to ensure that their results are as valid and reliable as possible.
What that means is in layman’s terms is that when a scientific poll is performed two days in a row, the two results would not vary much and can therefore be trusted to reflect the views of the targeted group. That requires the sample to include a mix of people fully representative of the target. Sampling statisticians deploy a wealth of tools, databases, statistical weighting processes, and sampling frames to ensure that their estimates are valid.
A non-scientific poll will produce wildly different results when it is re-administered a few hours or a few days later. It provides the opinion of the people asked, but that opinion cannot be generalized any further. You know nothing from such a poll about the larger population or the general opinion on the matter. When you stare at the results from such a poll what you see is a Rorschach blot, a nonsensical conversation starter that enables the viewer to state their opinion on something. The inferential ability from such a sample is zero, and yet presenting it to an audience and talking about it makes it seem otherwise.
When a network says, "we conducted a non-scientific survey" the only rationale responses are:
1. Ignore the response as a vapid attempt to lend credence to one person’s opinion or discredit another’s and give the talking heads a launching pad for whatever opinions they already had.
2. Switch to PBS or NPR, which make an erstwhile attempt to avoid such malpractice.
4. Cancel the network's Survey Monkey account and demand all staff who will ever comment and/or utter the word “survey” attend and pass college courses in basic statistics, sampling techniques, and common sense.