| 0:36 | Intro. Surprising aspects of polling, scientifics of polling. Telephone polling arose in the 1930s. From about the 1930s when polling started till the 1970s polling was done in person. Gallup sent people door-to-door. Method used was area probability sampling. With telephone polling it turned out there was no good listing of people. Random digit dialing. Worked pretty well. In last 10-15 years it began to run into trouble. People don't like being called; they do not like being called on their cell phones by strangers. Reduced cooperation. No longer feasible to get 70% of the people you call to be polled. Typical response rates now are 20% or maybe even less. Automatic dialers: typically 10-20 times as many phone numbers are selected as the number of interviews. Start to wonder if the people you are getting are representative; if you look at it they are not. Known that if you just call you get more women than men. End up with people with higher education, higher income, households where there are more people at home. Some people have a natural suspicion of polling saying that you don't sample everyone. Representative sample is statistically reliable even much smaller. A 1000-person random sample if there are no non-response problems. Doesn't depend on how big the population is. Ask 1000 people, get 95% confidence interval. If 50% are for Barack Obama, can be 95% confident that there are between 47% and 53%; still a 5% chance that it is outside that range. That's with a perfectly executed sampling plan. To be 99% confident, sample about 4000. |
| 6:55 | What's the problem? Magic of probability theory enables you to say, first, that if you take a big enough sample you'll get the right answer: Law of Large Numbers; and second, to quantify how big the error is: the Central Limit Theorem. Problem is the execution. You don't get 100% cooperation, predictable skews. People screen their calls; people who don't have phones; people with multiple phones who could get oversampled. Weighting helps address the skews, but if you don't weight on the right set of variables it can lead to systematic over- or under-estimates. Even if the weighting does correct the skews, it adds variability. Typical margin of error reported in the newspaper ignores the variability from weighting. Here's the idea. Suppose that in the population about 11% of adults are African American. In a typical telephone sampling, 4, 5, 6% is more typical. That means you have to double-weight the blacks in the sample to make the sample fraction about the same as the population. If the weighting is relatively small it doesn't add much variability. There are groups that are underrepresented by factors of 5-10. If you have a weight of 10 in a sample of 1000 that means 1 person is representing 10 people. If that one person's answer is recorded incorrectly it moves the whole sample by 1%, which will kill your margin of error. Two basic problems: weighting adds a lot of variability, probably about twice what the normal margin of error would suggest, twice what the sampling error would be with a simple random sample. Second, skews in the sample that are not taken out by weighting. Adds component that doesn't go down with increasing the sample size. Sampling over and over tells you size of sampling error but nothing about the skews. Skews that probably account for 80-90% in the polls represent the skews in the sample rather than the sampling errors. Poll in advance of New Hampshire primary, 100% got it wrong this year. If you have a lot of polls, you expect some to get it right. Approximately thirty polls in the week before the primary, not a single one had Hillary Clinton ahead of Barack Obama. Clearly something systematic was affecting all the polls. The people doing them were professionals using standard methods that usually work. Why didn't it work? We don't really know, but there appear to be a few things. Obama does very well with people with college degrees and graduate degrees, overrepresented by a factor of 2-3 and not enough weighting corrects for that. Hard to predict who will vote in a Presidential primary. About 90% of people called say they will vote but actual percentage is much smaller than that. Reported margins of error: given that African Americans, less educated people, higher educated people who don't want to bother, you have to weight. In a sample of 1000 one person represents too many. Don't the polls correct their margins of error for that distortion? No. Of 1300 polls in 2008 Presidential primaries, fewer than 50 reported anything other than the margin of error for a simple random sample with no weighting. Intellectually bankrupt, scandalous. |
| 15:19 | In a small state such as New Hampshire being sampled over and over harder to get people to answer the phone. Response rates are typically not reported and quite low. Some polls do well with low response rates, but you need a model. You can only correct for the factors you know in advance and for which you know the distribution. Age, race, gender, sometimes education; rarely income. Use weighting methods that are 60 years old, well behind better methods of weighting. For a long time there was a belief that random digit dialing was good enough. In the last 20 years most polls have gone to imposing quotas by gender and and primitive weighting to fix the other problems. Problems exacerbated in recent years and pollsters have not kept up. Likely voters: pollsters don't report actual responses because they want to weight them. What about likely voters? Standard is to ask questions about likeliness of voting. Problem is when people tell you they are going to vote, a lot of them won't vote. Turn-out rates in the mid-50% range for Presidential elections, under 40% for Congressional elections. Are polls fudged to look like other polls? Wouldn't say it's fudging. You have a choice of what weights to use. If you look at final polls, they are too similar relative to what we know their sampling variability is. That suggests that pollsters look at each other's weighting schemes. Roper Center, U. of Connecticut project: look at unweighted samples. Polls report their data to the Roper Center archive. People doing the polls, would not accuse them of dishonesty. It's certainly an art. People wish to be similar rather than different. But sometimes you want to stand out. Most of the New Hampshire samples were fairly small. One had a little over 100 Republicans, too small a sample so said let's ignore it. Wake up call for polling. |
| 22:55 | Internet polling: ESPN, David Ortiz, people vote multiple times. Disclaimers often say it's not scientific. Needn't be unscientific. People are used to automatic tellers at banks, punching in numbers on phone, etc., new technology is opportunity, with internet don't have to have a roomful of people on the phone. Problem ten years ago was that relatively few people on the Internet. With colleague Norman Nye, started company called Knowledge Networks, recruited a panel by telephone using random digit dialing, equipped them with hardware so that they could be covered. Reaction: not much bang for the random selection; and as time went on, more people on the Internet anyway. Good opportunity to have a different approach than using randomness. Randomness is great if it's feasible, but it's not truly random if you have a small response rate or if the people who respond are different from the population in ways that are not observable. Take a rigorous approach and use some things that didn't exist 25 years ago. Data bases, consumer files, that give you information on many dimensions about most people in the country. If you dial a phone number randomly, you don't know who you have missed. Could use these data bases. For political polling, start with a registered voter list--don't have to rely on self-reporting. Match that to a consumer file, age, gender, race from voter file; from consumer file income, home value, and so forth. That's the population we'd like to sample from. Target sample. Take a pool of about a million people who have opted in, agreed to be part of your Internet panel. Because it's so large, can find close matches to people selected off the voter list. Not a random sample, but insofar as the set of characteristics matched on is representative. Can often beat a random digit dialing with minimal response rate. More representative of the population: not just same number of women, young, blacks, but matched on more characteristics. In 2006 the average error was quite a bit less than the typical phone polling. Could have just been lucky: but then lucky 45 times. Had removed the biases better. Right now, if in telephone poll someone hangs up, they just call another random number. Effectively substituting another draw from the same skewed population; but should substitute someone with same characteristics as the person you missed. You don't know who you missed. |
| 32:37 | Every week for Economist magazine YouGov (Rivers's poll) runs a 1000 person poll of this style. For the election this year, running a number of large projects, sample sizes of 30,000-40,000 nationally, enabled to get state-by-state results. Pollster.com, Real Clear Politics report all the polls being done out there. Interactive Voice Response (IVR) does robo-calls, automated calls at random, answer questions by punching phone numbers, probably 80-90% of polls being done this year. Response rates are low but their record is not bad. Probably paid closer attention to questions of weighting. Can't even tell if respondent is male or female, older than 11. [Date of podcast recording: July 16-17, 2008, pre-conventions.] Who else does kinds of polls? Zogby, IVR, not good in 2006. Harris Interactive, but most Harris polls are telephone polls. Why does anybody run telephone style polls anymore? Associated Press and NY Times will not report a poll that uses a non-probability sampling method--i.e., something other than random digit dialing. Denial, being conservative. Full explanation put out there, newly developed methodology. Revealing weights is something of a trade secret; plus, if people saw they they might be horrified. |
| 40:01 | Real Clear Politics likes to average a bunch of polls. That may not be a more effective way of washing out errors if people are making the same systematic errors. If two polling methods used the same methodology of 500 people, averaging two would be better, essentially 1000 people. If they are not using the same methodology then averaging is no more reliable. If you have a reliable poll it should be weighted more than an unreliable one. But how can you get anyone to agree who should be weighted more than anyone else. Fivethirtyeight.com, 538, baseball statistics, historical accuracy weighting. Gallup poll shifts likely voter weighting, bump in methodology reflecting shift. Folklore, what happens the week before an election; when pollsters make errors they make stories ex post explaining themselves. "It's just a snapshot." There are cases where there is obvious movement but more frequently the polling methodology is off. Bradley effect: Tom Bradley running for governor of CA after being Mayor of L.A., number of cases where black candidates were ahead in the polls and then ended up losing or winning by less than the polls showed them. In 2008 Obama ahead in New Hampshire pre-election polls. Also called the Wilder effect, Doug Wilder who ran for Governor of Virginia in the mid-1980s, ahead in polls but won only narrowly. Do people tell you they will vote for a black candidate but then not do so? Study of many black candidates; conclusion that there was a systematic over-reporting but it disappeared about 10 years ago. Also looked at women candidates, Whitman effect after Christie Whitman. Is this anything other than an urban legend? A number of involved pollsters vigorously deny that there was an overstatement in the Bradley vote, but that it was a difference between the present and the absentee vote and Bradley lost the absentee. Democratic voters are pretty liberal, hard to believe they are averse to voting for a black candidate. Some experiments at Presidential level, darkening skin color; hypothetical Congressional election race against black and white candidates didn't really find any effects. Age, education effects are probably more important in explaining the New Hampshire polling problem. New Hampshire was unusual, and polls in other states not as far off. Election exit polling are better, but uniformly overstate Democratic vote. |
| 51:29 | Florida in 2000 was a bit of a debacle, networks predicted Gore; eventually retracted when it became clear that election was too close to call. Erroneous data feeds: data entry error. No network has made a miscall at the state level since 2000. Staff decision desks with a variety of inside and outside experts. Let's say a poll comes out 46-40 with a margin of error of 3%. Could be a dead heat by margin of error, with each getting 43%. Called a statistical dead heat. But your best guess is still that the leading candidate has a 6 point lead. If you are holding yourself to a 95% margin of error, you shouldn't say that one candidate is ahead of the other. You are equally likely to overstate as understate. Could equally well be 49%-37%. Many state polls are done with 400-500 voters. A lot are based on the robo polls, probably dial through almost everybody in a small state by the time an election occurs. Hard to believe that this method is reliable, but in fact they appear to have performed about in line with telephone polling. Convention bounce: phenomenon that almost always occurs after the convention of one party is that party's candidate gains in the polls. Because the out party always holds its convention first--that would be the Democrats this year--that will mean that the Democrats will have a gain till the Republican convention. Doesn't always happen, but could be bounce of 4-5 points, so net swing of 10 points. Love fest? Videos of candidates? More likely that those of the convening party is happy to participate and answer the phone, those of other party hide away. In the methodology of the Internet panel, controlling for the characteristics: reward people for participating in those polls. Send out an email: what percentage follow up and answer poll. Some people very active; others just 10% response rate. Try to control who gets the email invitation so that people who want to answer more polls get more without skewing the sample. Compensation given for people's time; in Internet world you have name and address. Using incentives might increase cooperation; but have to be careful. Are you an owner of x, y, or z? There are probably people out there trying to make a living off of filling out polls--can't earn much. Can do some validation. |
| 1:01:47 | Weekly survey for The Economist: what's surprising? Shift in partisanship. From the New Deal through the 1960s, Democrats had a massive lead in party identification, 18 points through 1970. Starting in the 1970s and going through the Reagan years, 2004, Republicans closed that lead. 80% of those Republican gains have been erased since 2004, including half in the fall of 2007 and spring of 2008. In principle should make this a terrible year for Republicans. Republicans nominated a candidate who is the least associated with the Republican administration than anybody you could think of; will be interesting to see if he gets associated with the administration over time. Also interesting this spring: last fall initially people didn't know Obama was black. By last December, blacks knew he was black and massively supported him. Among whites, Obama had the most amazing positives in years. In spring as Hillary Clinton ramped up the rhetoric those positive numbers went away. McCain's strategy. Polling on economics issues. Caplan at George Mason, Myth of the Rational Voter, claims that system works pretty well but voters aren't particularly well informed about economics. Standard views, trade, price ceilings, average voter wants to pass a law. |
READER COMMENTS
Unit
Jul 21 2008 at 9:26pm
I have to say that this discussion was creepy.
Here is a scary neologism: “opiniocracy”? It’s truly disheartening that there’s so much information out there that pins us down into categories of human sentiments and from which political action is then derived. “Opinion” poll is not the right term, better say “emotion” poll or “superstition” poll. When a pollster calls you up and asks you to choose among 4 different scape-goats which is the most likely to be causing the higher gasoline prices (talk about leading questions) people end up choosing the coolest sounding one. It’s not even a matter of economic ignorance, it’s not opinions that are being polled, but dreams and fantasies. It’s as if in Ancient Greece pollsters asked “What do you think caused the current war? A. The crows that flew over the temple, B. Zeus is mad, C. etc…”
In the market people have to give up something to add their choice to the general demand (yet an earlier interviewee talked about the “tyranny” of the market). Here we have elaborate scientific methods to record people’s wishful thoughts which are then made sacred by politicians, and participants are remunerated for their time? I say if you want a truly unskewed picture of people’s *opinion* you have to make them pay to record their choice, not the other way around.
Mark Selden
Jul 22 2008 at 7:28pm
Why do exit polls overstate the Democratic vote?
(Forgive the question if it’s just the Republican-heavy absentees; I’m assuming they are describing an overstatement of the Democratic vote at the polls, not total vote with the absentees included)
jayson
Jul 23 2008 at 11:31pm
This podcast makes me think that Mark Twain was correct there are 3 kinds of lies.
TM
Jul 24 2008 at 8:27pm
How do any of the polling techniques compare with:
1. each other
2. eventual results and
3. with election markets.
Based on my undertanding of the podcast, it seems (despite the process concerns,) the outcomes do not vary much for the first two comparisons and were not discussed for the 3rd.
Thanks for the links to this podcast. Maybe I’ll find my answer there.
TM
NB
Jul 25 2008 at 11:58am
Now yougov.com might have to adjust their techniques so they don’t end up oversampling all the economists that just signed up for the database.
Devon Steven Phillips
Jul 26 2008 at 4:26pm
I enjoyed this podcast, as I always do. Thanks to the host and guest.
I have always thought it’d be cool to get exit polling grouped in non-standard ways (not sex/income/race/religion). For instance: Which candidate is most popular amongst illegal drug users, or legal drug users? Which one appeals most to baseball fans, third shift workers, or cable news junkies? Do people who vacation in the Caribbean vote noticeably different than those who prefer going to Europe? etc…
Many of these questions would be very hard and expensive to explore but couldn’t the database spoken of in the podcast tell us which candidate smokers prefer, and even which brands are associated with each candidate? Do purchasers of sugary cereals make different choices than those who buy bran cereals? etc…
What’s truly amazing is that so many people vote at all, given the microscopic chances that one vote will influence the outcome. If enough people realize in the future maybe we will have to offer rewards like those of yougov just to get people to vote.
Luke
Jul 27 2008 at 2:19pm
On the episode “Boudreaux on Law and Legislation” (on which I can no longer comment), Boudreaux mentions the work of “Bob Cooter,” which is relevant to Hayek’s work on the emergent properties of law.
Is that THIS Bob Cooter?
http://www.law.berkeley.edu/faculty/profiles/facultyProfile.php?facID=24
And, which of his articles are relevant to that discussion? I was hoping to see some in the reading list. (BTW, the reading lists are great; thanks for providing so much additional material related to each podcast.)
SteveO
Jul 28 2008 at 11:26pm
Great podcast. “Unit”, the first commenter is right baout the bizarre nature of the polls. His analogy to greek polling is funny, and depressingly accurate.
To Mark Selden, my cynical answer is that basically all of the major networks, and most of the print media are overwhelmingly run by liberal minded people. I used to work in the student journalism field, and I can tell you the national conventions indoctrinating the high school and college kids are one step away from Nazi Youth Rallies. I’m not kidding.
My one feedback about the cast itself, I was stunned that Russ didn’t ask about election markets. (I just finished Surowiecki’s “Wisdom of Crowds”)
Russ Roberts
Jul 29 2008 at 11:49am
SteveO,
You’ll find election markets discussed in the first Robin Hansen podcast as well as the Sunstein Infotopia podcast.
Russ
T L Holaday
Aug 2 2008 at 10:36am
I was interested that Rivers’ statement at 47.56:
Is it so accepted that non-liberals would be expected to be averse to voting for a black candidate? Is non-liberal equivalent to “favoring non-blacks”?
jtc
Aug 2 2008 at 10:48am
There is a great old movie staring Jimmy Stewart/Jane Wyman called “Magic Town”in which “An opinion pollster finds a town which is a perfect mirror of U.S. opinions”… Fits well with this podcast…
SteveO
Aug 3 2008 at 11:57pm
Thanks Russ. Believe me, I’ve already listened to them. =)
Which is to say, I’ve listened to all the EconTalk episodes.
TSowell Fan
Aug 5 2008 at 11:57pm
“Doesn’t depend on how big the population is. Ask 1000 people, get 95% confidence interval.”
By this, I took Rivers to mean that it didn’t matter what the USA’s population was, sampling just 1,000 of its inhabitants on their opinions about any topic would yield an estimate of the entire population’s beliefs with 95% confidence.
Umm, not what I learned in statistics.
drtaxsacto
Aug 8 2008 at 1:16am
This was one of the best in the series (in what is a consistently good series of podcasts). Rivers presented some absolutely fascinating ideas about the potential errors in polling from the perspective of someone who seems to be committed to thinking about how to make it better. Bravo!
bill raynor
Aug 19 2008 at 11:35am
Are there any technical references to the matching approach that Rivers suggested? His www page at Stanford does not include any references.
Comments are closed.