- Published on Thursday, 30 October 2008 00:00
- Written by Carl Bialik, WSJ.com
According to most polls, Sen. Barack Obama holds a formidable lead over Sen. John McCain among the youngest voters. Among 18- to 29-year-olds, Obama leads by 32 points in the latest Gallup poll, by 36 points in the latest CBS/New York Times poll and by 39 points in the latest Pew poll. However, one daily tracking poll consistently has shown McCain leading Obama among a similar group, 18- to 24-year-olds - an anomaly that provides a lesson about the dangers of slicing polling data too thin, especially among voters who are hard to reach.
The unusual results came from a poll conducted for Investor's Business Daily by TechnoMetrica Institute of Policy & Politics, the polling arm of the market-intelligence company TechnoMetrica. In the 11 daily results of the IBD/TIPP poll released between Oct. 16 and Oct. 26, Obama led McCain just once, by two points, among 18- to 24-year-olds. Nine times McCain led by eight points or more. His average lead among these young voters during that time span was 15 points. One IBD/TIPP poll last week showed McCain trailing by 1.1 points overall - generating discussion among blogs and in a Pittsburgh Tribune-Review editorial - and leading by 52 percentage points among 18- to 24-year-olds, evoking incredulity from political stats-head Nate Silver.
Raghavan Mayur, president of TechnoMetrica, told me he was equally surprised by the results, saying the widespread perception that Obama is leading by a large margin in that group "is my perception, too." He blamed the result on a small sample size. Each daily tracking poll includes about 1,000 interviews spread over the prior five days; each day a new set of survey respondents is added and the oldest set is discarded.
Ideally, Mayur would like to have 75 of all those respondents fall into the youngest age range. Some pollsters would have preferred more; this age group makes up 13% of the adult population, though its voting rate historically has been lower than average. His sample fell far short even of his lower goal, typically including just 25 to 30 respondents from age 18 to 24 - meaning just five or six new interviews with these young voters were being conducted each day. "We are not able to get to speak to as many as we would like to in that group," he said.
He blamed that on several factors. For one thing, nearly one-third of adults in that age range lack landline phones, and Mayur's pollsters don't dial cellphones. (He points out that when calling cellphones, the chance that the person who picks up lacks a landline and is in the relevant age range is quite low.) Furthermore, among those who do live in households with landlines, young people may be away at school or in the military, Mayur said.
This small sample size at first didn't trouble Mayur, as Obama led among these voters in the first three tracking polls. But when the results started to break McCain's way as suddenly and dramatically as they did, Mayur began to question his own methodology. On the day McCain's lead widened in this group to 52 points, Mayur added a footnote to the 18-to-24-year-old group: "Age 18-24 has much fluctuation due to small sample size." He says he didn't add a similar one to the Jewish subgroup, with just half the sample size as the young voters, because the Jews in his sample consistently stated a preference for Obama, as he expected.
This week Mayur took two steps to expand the sample size of young voters. His pollsters are asking those who answer the phone to put the youngest member of the household on. They are also dialing households expected to have young voters, found by cross-listing white-pages listings with drivers' registration data. "We have always preferred using random-digit dialing sample," Mayur said. "I am adapting to this new challenge with these modified tactics." In the two polls since the change, the trend has reversed itself: Obama now leads by double digits.
It's not unusual for a pollster to take measures to boost representation of hard-to-reach groups, but it is unusual to do so only when the results for those groups are unexpected. "I try to do the best I can," Mayur said. "I don't have an agenda."
Other pollsters may be facing similar challenges finding enough 18- to 24-year-olds, but they don't report results for this narrow age group, instead bundling them with 25- to 29-year-olds. That makes it difficult to say how far the results consistently favoring McCain missed the mark. Unless this slightly older group (25-29) favors Obama overwhelmingly, Mayur's problems appear to have extended beyond sample size to selection bias. In other words, perhaps unintentionally his original tactics disproportionately included McCain supporters.