Fundamentals of (Digital) Survey Design

MACS 30000: Perspectives on Computational Analysis
University of Chicago

Asking Questions 1.0

Why ask questions?

Because we can
Measure internal states

Total Survey Error

Census vs. Sample
Sources of error
- Representation errors
- Measurement errors

Relationship to Bias and Variance

\[ \begin{eqnarray} \text{Total Survey Error} &=& \text{Representation Errors (Bias)} \\ &+& \text{Measurement Errors (Variance)} \\ \end{eqnarray} \]

Relationship to Bias and Variance

Representation Errors

When the survey does not accurately represent the population of interest
Example: 1936 Literary Digest poll (10 million ballots send and 2.4 million received)
- Target population vs. Frame population
- Sample population vs. Respondents
  - Non-response bias

Mentioned that REs \(=\) When the survey does not accurately represent the population of interest
Example (that students read about for today): 1936 U.S. Presidential Election and Literary Digest poll (Roosevelt vs. Landon) – poll failed to make an accurate prediction (Predicted Landon would win, but Roosevelt won in a landslide); while its sample was massive (2.4 million respondents – reducing variance of estimate), it was not representative of the voting public:
- The target population was voters in the 1936 presidential election
- The frame population was the 10 million people whose names came from predominantly telephone directories and car registration records (and also magazine readership)
  - [Ask Reasons this might cause representation error?]
  - Because the target and frame populations were systematically different from one another, extreme coverage bias was introduced into the estimates of support for Alf Landon and Franklin Roosevelt
Sample population are the people the researcher will attempt to interview from a frame population (in Lit. Digest example – a census – contacted everyone and didn’t try to sample probabilistically or otherwise – will talk more about this later)
- No sampling error (poll was a census of frame population) – but this will be a problem we’ll talk about in a bit and another potential source of bias
- Respondents are the people who actually participate in the survey
  - Even if the target and frame populations perfectly match, if some members of the sample population systematically opt-out of the survey and fail to become respondents, the sample will become biased – called nonresponse bias
In the case of Literary Digest poll – only 24% responded, leading to additional nonresponse bias in addition to coverage bias (potentially only certain individuals – not even relevant sample of voting public – are taking the time to respond)

So, having a large sample size not necessarily better here – reduce variance of estimate, but bias great enough to produce faulty prediction.

At the beginning of your reading for today, they mention that non-response bias is becoming even more of an issue in surveys today. [Ask] What factors cause non-response bias to be a problem in traditional survey research today? [e.g. telephone/mailed surveys]
- Very few people respond to surveys
- Those that do skew older
- Older individuals likely have different attitudes than younger individuals
- Failure to correct for these differences can lead to incorrect estimates
Some options for addressing:
- Weighting/post-stratification (we’ll talk more about this later)
- Provide incentives for individuals to respond
- Oversample young individuals (call more cell phones)
- Pre-test to make sure it is not too long so that individuals drop-out part way through
- Send reminders to individuals to participate (Pew Research calls up to 7 times to try and reach a respondent)

Sampling Methods

Introducing sampling error as another source of representation error

Probability Sampling

Simple random sampling
Oversampling and undersampling
Post-stratification and weighting

Probability sampling

Simple random sampling
- Each person in the population has an equal probability of inclusion
- When respondents are selected via SRS with perfect execution, then estimation is straightforward because the sample will - on average - be a miniature version of the population
- Only source of representation error is sampling error (random statistical noise)
Oversampling and undersampling
- Current Population Survey (CPS) – by Census Bureau administered monthly
- Want state-level estimates of unemployment (among other measures), as well as a national estimate
- Need to oversample smaller states and undersample bigger states via stratified sampling strategy
  - If we take the raw arithmetic mean, our national estimate will be skewed by the small states
  - Instead, we apply weights to adjust the estimate with the knowledge that some individuals had a higher probability of appearing in the sample versus others
Post-stratification and weighting
- Giving more value to certain respondents over others
- To calculate a group (or cell) level estimate, average over all the responses in that group
- Becomes relevant especially when have systematic bias in nonresponse

Non-probability samples

Homogenous-response-propensities-within-group assumption
Response-propensity assumption
- But: potential issues with data sparsity

In the case of a sample that was collected in a non-probabilistic way (e.g. a convenience sample in an online poll), can also use post-stratification to yield good estimates. However:

Post-stratification only works if everyone in each group has the same probability of participating (homogenous-response-propensities-within-group assumption) …
or if there is no correlation between response propensity and the outcome being measured (response propensity assumption)
- Frequently this is not true in large groups
  - Example: (imagine we’re surveying individuals about unemployment and grouping by state) Everyone in Alaska should have the same probability of participating in a survey to meet the HRPWG assumption + no correlation between response propensity and unemployment status to meet the RP)
- But as you chop the data into smaller and smaller groups, assumptions can become more valid
  - All 18-29, female, college graduates living in Alaska have the same probability of responding (so, people that you don’t see in the population are like those that you do see in your survey – reducing systematic differences in response propensities within the group + decreasing chances of correlations between response popensity )
Data sparsity - unfortunately as you chop the data up, there are smaller numbers of respondents in each group
- Produces larger uncertainty around estimates
- If there are no respondents in the group, you cannot form an estimate (directly at least – without a predictive model)

Remember that these are assumptions - assumptions are inherently untestable, and frequently necessary to conduct inference

Goal should be to minimize the needs for assumptions, or at least consider their plausibility

Multilevel regression and post-stratification (MRP, e.g. Wang et al. 2015)

Collect your sample (probability or non-probability)
Divide the sample into many groups (cells) based on known characteristics
Estimate a multilevel regression model
Average over each group (cell) to calculate the estimate of interest
Post-stratify the results based on each group’s prevalence in the known population

Adjusting estimates for non-probability samples requires estimating the appropriate weight for each respondent. One method for doing so has gained in popularity in recent years, called multilevel regression and post-stratification, or MRP or Mr. P. The Xbox election survey you read about for today uses this methodology to predict the outcome of the 2012 U.S. presidential election and capture voter opinions leading up to the election.

The basic process is this:

Collect your sample (probability or non-probability)
- Non-probability sample of Xbox users who opt-in to the survey
- Collect almost 350,000 responses over a 45 day period
- We know this is not a representative sample - see Figures 1 and 2
Divide the sample into many groups (cells) based on known characteristics
- Sex
- Race
- Age
- Education
- State
- Party ID
- Ideology
- 2008 vote
- Divide into each unique combination of these variables (176,256)
Estimate a multilevel regression model
- Fitting a line to the data
  - Draw a scatterplot with data points
- Least squares or GLM assumes observations are independent from one another
  - Add a linear trend line
  - Add another example using a probability curve
- Here, we can think of this as pooling surveys from each state to form one massive survey
  - This is fine, as long as we can assume that the relationship between sex and vote preference is the same across all states
  - However we know that is unlikely to be true. For instance, women in Texas are probably more likely to support Trump than women in New York
  - The data is not exchangable - it doesn’t all come from the same “distribution”
- One way to control for this pooling issue is to include a fixed effect - add dummy variables for each indicator
  - Problem is that would result in 176,255 variables in the model - extremely complex
  - Also ignores the fact that there is some pooling across groups
  - That is, overall men and women lean towards Clinton
- Beyond the scope of this class, but multilevel modeling allows for partial-pooling
  - It accounts for both within-group variation (differences in presidential support within a single state) and between-group variation (differences in presidential support across states)
  - After all, there are probably some similarities between a white, female, 25 year old college graduate liberal Democrat from New York who voted in 2008 and a white, female, 25 year old college graduate liberal Democrat from Texas who voted in 2008
  - Multilevel modeling accounts for this in our estimates of the strength in relationship between the independent variables and the outcome of interest
Average over each group (cell) to calculate the estimate of interest
- What percentage of white, female, 25 year old college graduate liberal Democrats from New York who voted in 2008 support Clinton?
- What percentage of white, female, 25 year old college graduate liberal Democrats from Texas who voted in 2008 support Clinton?
- Etc. etc. etc.
Post-stratify the results based on each group’s prevalence in the known population
- Weight each group given its size in the population of interest
- Typical sources of population size
  - Census data
  - Current population survey (CPS)
  - Past presidential exit polls

Source: Bit By Bit Figure 3.8 (based on Wang et al. 2015, Figure 2 + 3)

As seen in the paper (with more technical details on the MRP), these adjusted estimates are much more accurate and give a pretty good estimate of actual support for Obama in the voting public in comparison to the original unweighted results.

Keep in mind, though, that some of the estimates, like predicted electoral vote share (which need in order to estimate how much of this support actually translates into votes), come with lots of variance and uncertainty (model could have been fine-tuned or calibrated more for state-specific estimates, but the main goal was predicting national vote-share).

Nonetheless this is a powerful tool for correct some bias – either in probability samples where response rates are uneven, or in non-probability samples, where we have systematically biased participation.

Representation Errors

Source: Bit By Bit Figure 3.2

Key Takeaways

Having a large number of respondents will often decrease the variance of estimates, but it does not necessarily decrease the bias
Researchers need to account for how their survey data is collected in making estimates - how might the population of respondents be skewed?

Measurement Errors

Answers we receive and the inferences we make can depend on exactly how we ask a question.

If you could choose between the following two approaches, which do you think is the better penalty for murder - the death penalty or life imprisonment, with absolutely no possibility of parole?

Are you in favor of the death penalty for a person convicted of murder?

Have discussed representation errors at length…

Measurement errors on the other hand can influence how we make inferences from the answers that respondents give to our questions. After all, Question wording matters to overall responses. For instance, one study asked the following questions [Reveal fragments]:

If you could choose between the following two approaches, which do you think is the better penalty for murder — the death penalty or life imprisonment, with absolutely no possibility of parole?
- 53% death penalty, 44% life imprisonment
Are you in favor of the death penalty for a person convicted of murder?
- 77% yes

Intended to assess the same thing, but lead to different measures.

[Ask] what might cause this?

Bottom line: what you learn from a survey often depends on how you ask your questions!

Measurement Errors

TIL that only 34% of American Millenials think that “America is the greatest country in the world” while that number is at almost 70% for Baby Boomers. from r/todayilearned

Avoiding Measurement Error

Copy questions from high-quality surveys (e.g. GSS, ANES)
Survey experiment some of your questions
Pilot test your questions (pre-testing)

Mental Exercise: Failure of US presidential election polls (2016)

Most public polls prior to election day predicted Hillary Clinton would defeat Donald Trump in the 2016 Presidential Election. Whoops!
Use the total survey error framework discussed today to assess what could have gone wrong. Consider the following:
- Representation error
- Measurement error
Suppose a pollster asked you for suggestions on how to avoid making a mistake predicting future winners of US presidential elections. Provide specific two recommendations and explain why you think they would reduce error in the poll.
Group activity: Meet in groups on (or before) Thursday and come up with EXACTLY ONE best recommendation that you can present to the class.

Discussion Groups for Week 4

	Week 4 Discussion Groups
0	Zhuojun / Yue / Tian
1	Andy / Ertong / Max
2	Anny / Abbey / Kuang
3	Daniela / Jiazheng / Yuhan
4	Huanrui / Pritam / Kexin
5	Lorena / Agnes / Tianle
6	Emma / Cosmo / Thomas

Survey 2.0: New Ways of Asking Questions

Typical Survey Modes

Interviewer-Administered: In-person or phone interview
Question types
- Closed-ended questions
- Open form

Computer-Administered Surveys

Benefits

Reduce costs
Reduce social desirability bias
Eliminate interviewer effects
Increase respondent flexibility, timing

Drawbacks

Cannot clarify confusing questions
Lose rapport with respondent
Interviewer can’t help maintain respondent engagement

Wiki surveys

Figure 3.9 in Bit by Bit: Results from a survey experiment (Schuman and Presser 1979)

One example of good digital-native survey type is a “wiki” survey…

Combines aspects of closed and open-ended surveys (in that users can actually contribute to the survey – hence the “wiki”)
Identifies rank-ordered preferences based on randomly selected responses to a question
The process: walk through example survey “Which do you think is better for creating a greener, greater New York City?”
- Start with a seed list of responses
- Ask respondents to compare two randomly selected responses
  - Continue asking them with randomly generated pairs until they stop
  - With enough responses, you can use this information to rank order the preferences of all the options
- Ask respondents to contribute their own ideas for responses, which are put into the pool for future respondents to choose from (subject to moderator approval – so constantly growing and not just limited to the original plan of the survey designer)
- Results: estimated chance that an idea will win against a randomly chosen idea (100 = estimated to win each time, 0 = never wins) – a rank-ordered list of preference within a population
What are the benefits of this approach?
- Uses the strengths of both open and close-ended questions (have certain seed questions, but users can add more – in NYC example 8 of 10 top scoring ideas were uploaded by participants and this actually informed Mayoral decision making when this was live in October 2010)
- Cheap, fast, easy
- Can still analyze in a systematic way - create an opinion matrix of all respondents and possible responses
What are the drawbacks of this approach?
- No absolute measures of preferences
  - We don’t ask about individual options, only preferences relative to other options
  - If the respondent doesn’t repeatedly answer these option pairings, we don’t know their overall preference structure
- Consistency
  - Respondents don’t all receive the same questions and option pairings
  - Is it better to maintain random pairings throughout the protocol or adapt the options given prior feedback? That is, once one option has been voted down repeatedly minimize or stop asking respondents to consider it
- Validity
  - How do we know these responses are capturing the respondents’ true attitudes?
  - No good comparison method - internal states are always hard to measure and quantify
- Can overcome a lot of these problems using statistical methods to pool and summarize individuals’ responses

Wiki surveys

Combines aspects of closed and open-ended surveys
Identifies rank-ordered preferences based on randomly selected responses to a question
Which do you think is better for creating a greener, greater New York City?

Gamification

Figure 3.11 in Bit by Bit: Interface from Friendsense study (Goel et al. 2010)

Can also solicit information through games – Goel et al. wanted to esitmate how much people think they are like their friends and how much they are actually like their friends (e.g. real vs. perceived attitude similarity).

So, need to survey a person about their opinions and then survey their friends about their opinions + a guess about their friends opinion (e.g. estimate how similar their perceived attitude similarity is)
Turned survey into a Facebook app that was fun to play:
- App selected random friends from respondent’s Facebook account and asked a question about the attitude of each friend (after which they were told whether they were correct or not; if friend hadn’t participated, there was an option to try to coax the friend into participating)
- Also intermixed questions about respondent
- Questions were adapted from General Social Survey – so quite good – along with more lighthearted questions (like this one in picture)
- Findings: even close friends disagreed on ~30% of questions, but friends overestimated their agreement with friends (most of the diversity of opinions was not noticed – whether on lighthearted issues or politics)
Standard attitude survey that was fun and became viral way of getting to know friends
Perhaps some concerns about data quality (given feedback between friends and selves + fact that it’s on social media and tightly controlled)

Ecological Momentary Assessment

Take long surveys, chop them up, and incorporate into lives
Major benefits of EMA
- Collection of data in real-world environments
- Assesments that focus on individuals’ current or very recent state or behavior
- Assessments that may be event-based, time-based, or randomly prompted
- Completion of multiple assessments over time
Cornwell and Cagney (2017) – We will discuss it on Thursday.

Involves taking traditional surveys, chopping them up into pieces, and sprinkling them into the lives of participants (in contemporary world where we all use smartphones and other wearable devices, lots of potential uses)
Major benefits of EMA
- Collection of data in real-world environments
- Assesments that focus on individuals’ current or very recent state or behavior
- Assessments that may be event-based, time-based, or randomly prompted
- Completion of multiple assessments over time
- Compared to traditional survey methods, all of this is really useful!
  - Typically ask questions months after the fact
  - Respondents have bad memories
  - Respondents drop out of the study
Cornwell and Cagney (2017)
- RQ: How much does residential neighborhood define older adults’ activities? A lot of previous studies assume that getting older implies that people are more anchored to their neighborhood – operationalized as census tracts (and this is thus the primary factor shaping health and well-being). To what extent is this actually true?
- Method: gave iPhones to 60 older adults in 4 NYC neighborhoods
  - Captured GPS location every 5 minutes for a week + administers 17 EMAs at different times over 4 days to assess activities they were performing
- Find that they’re much more mobile than typically assumed – activity spaces are much bigger than the residential (census) tracts they’re normally confined to from an analytical perspective
  - some socioeconomic variation, though: e.g. respondents with less education and lower incomes spent more time outside of their residential tracts than high-status respondents + low poverty in residential tract seems a good proxy for overall exposure to poverty outside
  - activities inside tract: housework, personal care, leisure, eating
  - outside tract in >40% observations involving socializing, shopping, exercising

Linking Surveys to Other Data

Amplified asking

Using digital traces to extract more value from survey data
Feature engineering + machine learning
Predicting poverty and wealth from mobile phone metadata (Blumenstock et al. 2015)

…for instance, via an “Amplified asking” research design, like you read about in the Blumenstock et al. reading for today.

Using digital traces to extract more value from survey data than it gives on its own
Here, the digital data is not of any direct benefit - it is only useful in conjunction with asking questions
Predicting poverty and wealth from mobile phone metadata (Blumenstock et al. 2015)
- Attempts to estimate poverty and wealth at a granular geographic level
- Cannot survey all Rwandans - too expensive
- Instead, surveys less than 1000 to measure indicators of wealth (e.g. owns a bike, has electricity, has indoor plumbing)
- Pairs the survey data with digital traces of phone logs
  - Call records are incomplete since they cannot directly measure wealth
  - However Blumenstock can use the digital traces to predict how wealthy someone is based on their survey responses
  - Uses machine learning approaches - we will discuss these in much more detail next quarter
- After training the model on the 1000 survey respondents, the authors can apply the model to all 1+ million cell phone subscribers to predict wealth levels
- Because the researchers also had the subscribers GPS data, they could then map these wealth measures geographically to identify clusters of wealth and poverty

Overall – amplified our capacity for asking questions by surveying a small subset, fitting a model to predict responses from digital trace data, and then extrapolating results at scale on remaining digital trace data.

Discussion: Amplified Asking in Policymaking

Blumenstock, Cadamuro, and On (2015) use call detail records (CDRs) from mobile phones to predict poverty and wealth in Rwanda. Other studies have used CDRs to predict aggregate unemployment rates. Do you think CDRs and other measurements generated through amplified asking techniques should replace traditional surveys, complement them, or not be used at all for government policymaking? What evidence would convince you that CDRs can completely replace traditional measures?

Enriched asking

Combine digital trace with survey data (1:1)
Record linkage \(\rightarrow\) messy

Discussion: Predicting traits from Facebook likes (Kosinski et al. 2013)

Role of enriched asking (e.g. why can’t we just use observational data alone?)
Source of data
Ethical concerns

Good example of this in the Kosinski et al. paper you read for today…

[Discuss in groups: 5 min in groups, 10 min as a class; can either solicit responses as a full class or as groups depending on time]

Uses enriched asking to enhance a predictive study of private traits and attributes based on Facebook likes
- Need to conduct a survey in order to merge with observed Likes to evaluate the predictive accuracy of the model
- Great example of enriched asking
Obtained survey data via an app created for the study - opt-in for participants (not just taking data from Facebook without asking)
- Some attributes can be tested strictly using observational data from Facebook
  - Age
  - Gender
  - Relationship status
  - Sexual orientation
  - Partisanship
- Others cannot be directly revealed with digital trace
  - Personality - Five Factor Model (OCEAN)
    - Openness
    - Conscientiousness
    - Extraversion
    - Agreeableness
    - Neuroticism (emotional stability)
  - Intelligence
  - Satisfaction with life
Ethical concerns – thoughts on ethics of this sort of data linking approach?
- Can predict private details of people who didn’t even sign off on the survey!
- Shortly after publication, Facebook switched Likes to be private by default (previously were public)

Fundamentals of (Digital) Survey Design

Asking Questions 1.0

Why ask questions?

Total Survey Error

Relationship to Bias and Variance

Relationship to Bias and Variance

Representation Errors

Sampling Methods

Probability Sampling

Non-probability samples

Multilevel regression and post-stratification (MRP, e.g. Wang et al. 2015)

Representation Errors

Key Takeaways

Measurement Errors

Measurement Errors

Sometimes measurement error is itself an interesting social behavior

Avoiding Measurement Error

Mental Exercise: Failure of US presidential election polls (2016)

Discussion Groups for Week 4

Survey 2.0: New Ways of Asking Questions

Typical Survey Modes

Computer-Administered Surveys

Benefits

Drawbacks

Wiki surveys

Wiki surveys

Gamification

Ecological Momentary Assessment

Linking Surveys to Other Data

Amplified asking

Discussion: Amplified Asking in Policymaking

Enriched asking

Discussion: Predicting traits from Facebook likes (Kosinski et al. 2013)