Observing Behavior: Digital Trace (Big) Data

MACS 30000: Perspectives on Computational Analysis
University of Chicago

What is digital trace data?

In aggregate: often called “Big Data”

We have finally found our telescope!

[J]ust as the invention of the telescope revolutionized the study of the heavens, so too by rendering the unmeasurable measurable, the technological revolution in mobile, Web, and Internet communications has the potential to revolutionize our understanding of ourselves and how we interact … . [T]hree hundred years after Alexander Pope argued that the proper study of mankind should lie not in the heavens but in ourselves, we have finally found our telescope. Let the revolution begin.

Watts, D.J. (2011:266). Everything is Obvious: Why Common Sense is Nonsense. United Kingdom: Atlantic Books.]

Observational data

Any data that results from observing a social system without intervening in some way

Digital trace

Digital data that captures “traces” of human behavior

Social media sites
Internet search data
Blogs and Websites
Administrative records
Historical archives
Geospatial data
and much more…

Digital trace comes in many forms and across many different platforms. There is no singular system collecting and collating this data. It is generated when humans interact with a digital system and that interaction is recorded and saved. Oftentimes, digital trace is created by governments and companies for purposes other than research. Researchers will re-purpose this data (an example of readymades that I talked about last week) to answer the questions they care about – and much of this unit is focused on thinking about research design strategies for effectively doing this.

Not only does digital trace come from different sources, the type of data collected varies significantly. Traditional social scientific data was readily quantifiable and could be stored in a tabular format. But contemporary data sources store much more complex data sets. So, we might work with:

text, audio, visual data from social media sites
textual logs of what people search in aggregate in search engines
all of the HTML, images, text, and so on that are encoded in company and personal websites
Raw administrative records
Historical archives of digitized historical/archaeological materials from the past, as well as archived digital artifacts that can no longer be accessed on the web in a “live” form
Geospatial data from apps and satellite images from government satellites
and much more… Given the always-on nature of many digital platforms, we generate a lot of digital trace every day.

For instance, just thinking about the data that I generate on an average day…

Geography of intergenerational mobility

Chetty, R., Hendren, N., Kline, P., & Saez, E. (2014)

Another groundbreaking use of digital trace relies on a pretty traditional source: IRS tax returns. When these records are digitized over several decades, they become a powerful tool for measuring income mobility across generations.

In this paper, Raj Chetty uses income tax returns to describe intergenerational mobility in the United States, to examine the relationship between a parent’s income percentile and their children’s income as an adult. Because tax returns include the residential address of the income filer, his team was also able to examine how upward mobility – operationalized as the chance of a child having a higher income than his or her parents – differs based on geography.

This type of study benefits from the rich detail of the source data. It includes every tax return filed in the United States, not needing to rely on a smaller sample of families. Since the data is collected directly from the IRS, we also have fewer concerns about false or incorrect measures of income. Without the richness of this dataset, the researchers would have more concerns about validity and reliability.

Traits of “Big Data”

The 3 Vs of Big Data

Volume
Variety
Velocity

However, it is descriptive of certain components of the production and form of digital trace data that we’ll need to grapple with as we design research using it and programs to actually handle it computationally. Specifically, one common approach is to think of big data in terms of the 3 V’s; e.g. relative to other data sources, this is data that has:

Volume: a lot of observations
Variety: a variety of data types and file formats
Velocity: moving so fast that it’s hard to collect all in one place – dare I say “real-time”

If you stick around for MACS 30113/123, you’ll see that as a result of these V’s, it’s often hard to even use traditional computing methods to work with this data, so you need to use special programming strategies and systems (clusters, cloud resources, special programming approaches) instead.

Some examples of data that participates in the 3 V’s: Twitter, Facebook, and other social media companies, as well as search engine logs and other data generated by private companies. Also, important data generated by the government in the form of administrative records. These include things such as tax records, school records, vital statistics, and more. While governments have historically collected this data for many decades, modern collection procedures digitize and automate much of this information. All provide data that is big, heterogenous, and sometimes moving fast enough that it can cause problems for traditional methods of study.

Lots of potential for reuse in social scientific research, but regardless of the source, we need to be cautious in repurposing data (don’t want to adopt data source blindly even if we have the skills to do so). Always strengths and weakness in using any data source in designing research.

Let’s think more explicitly about some strengths and limitations of using big data sources in social research…

Strengths

Big

Source: Bit by Bit

Big means there are a lot of values in a data set. This could be a lot of observations, or a lot of variables measuring behavior or characteristics of different observations. As this figure shows, information storage and computing power has been increasing at an exponential rate. There is just a lot of data in the world which we can draw upon to conduct research.

Contrary to public opinion, bigger is not inherently better. Bigness is beneficial in certain contexts, though, because it enables types of research not previously possible. One example is that we can study rare events. In a traditional sampling procedure, we may only see one or two individuals exhibit the rare behavior out of a total sample of one thousand. It is hard to generalize from these two individuals. But if we examine an event that occurs in point zero one percent of the population and could observe the entire population of one million individuals, we would have ten thousand individuals in our dataset where the event did occur. This is much easier to generalize from.

Another example is studying heterogeneity in a population. The larger the sample size, the easier it is to detect differential patterns of behavior in a sample.

Finally, we can also detect small differences. The smaller the effect size, the larger the sample size needs to be in order to have a reasonable chance of observing it. Power analysis can be used to calculate the necessary sample size. If we have an expected effect size of zero point one percent, we may need over a million observations to have a chance of detecting that effect. But with big data, observing this many observations can be trivial.

Always on

Another beneficial feature of big data is that it big data systems are constantly collecting data. This longitudinal feature has important implications for research.

Researchers can study unexpected events in ways not otherwise possible. Consider many social movements which quickly burst onto the scene with little notice or preparation. If you want to study these movements, you need to know where and when they will occur. But with always-on data sources, you can retroactively examine these events as if you observed them in real time.

Additionally, always-on systems allow us to produce real-time estimates which can be useful for policy makers. This could include using social media data to guide emergency responses to natural disasters, or producing real-time estimates of economic activity or disease occurrence.

Non-reactive

See also: Stephens-Davidowitz, Seth. 2014. “The Cost of Racial Animus on a Black Candidate: Evidence Using Google Search Data.” Journal of Public Economics 118:26–40. https://doi.org/10.1016/j.jpubeco.2014.04.010.

Big data systems are also non-reactive, which means individuals do not react to observation by researchers. If individuals participated in a research study and know they are being watched, they may adjust their behavior. This is known as reactivity or the Hawthorne effect.

Since big data involves repurposing data captured for other reasons, individuals are not expected to modify their behavior in response to scientific study. This is especially important if you want to study a behavior that is hard to measure accurately. For instance, many doctors and medical professionals fail to follow common hygiene protocols such as washing their hands prior to contact with patients. But if a nurse or other individual is watching the doctor, she is more likely to follow proper procedures and wash her hands. Here the Hawthorne effect is desirable, but in other situations we need to avoid it to accurately measure behavior.

In a widely cited study, Seth Stephens-Davidowitz used the prevalence of racist terms in search engine queries to measure racial animus in different regions in the United States. He could not observe this directly through a survey due to social desirability bias - respondents know racism and racist attitudes are frowned upon by society, so they would not be honest answer questions related to racial animosity. However they have no such constraints using racially charged terms in their search queries online.

Weaknesses

Inaccessible

One of the biggest limitations to big data is access. Most big data is generated by private companies and governments with limitations on using that data for research. Many of these restrictions are driven by legal, business, and ethical barriers that prevent data access. For one, companies are liable for breaches of sensitive information such as customer data. They will be very cautious about granting external researchers access to their data without a sufficient justification.

Some companies and governments have developed special programs to allow research to apply for access to data sources which contain personally identifiable information, but typically these partnerships rely on personal connections at the company or a strong reputation in the academic community to open that door. And even if your request is granted, it may come with limitations on how you can use the data, share it with collaborators or other researchers, the types of questions you can answer, or even when and where you can publish results.

Non-representative

Another concern is who is represented in the data. Social scientists typically work with probabilistic random samples from a defined population, such that the sample is a reasonable approximation for the population. This is not guaranteed when using big data sets generated by digital platforms which have skewed demographics. For instance, most of the popular social media platforms skew towards younger individuals. If you only study these users without accounting for that bias, your inferences will probably not generalize to a population including all adults, for instance.

Non-representative data can still be useful depending on the context.

If the population of interest was young adults ages 18-24, Snapchat or TikTok could be representative of that segment of the population
Questions that focus on within-sample comparisons as opposed to out-of-sample generalizations can use non-representative data.
We will see statistical techniques which can be used on non-representative data to correct for its non-representative nature. When used appropriately, we can minimize the impact non-representativeness has on our research.

Positivity bias

Drifting

Longitudinal data sets such as big data are useful for studying change over time. But in order to reliably measure change, the measurement system needs to be stable. If the measurement procedure changes, then we cannot reliably compare earlier measurements with later measurements.

Unfortunately many big data systems change all the time in a process termed drift. This change is driven by three factors.

Population drift is change in who uses a system.
Behaviorial drift is change in how people use a system.
And system drift is change in the system itself.

Think of the dominant social media platforms since the development of social media in the early two thousands. The first dominant platform was Myspace, a system that no longer exists. Then Facebook rose up as a widely used platform, but for many segments of the population that has shifted towards other platforms like Instagram, Snapchat, TikTok, Twitter, and more.

If we used Facebook to study a long-term process, we have to remember that the userbase has changed significantly from the early days where only college students could create an account.

Algorithmically confounded

See also: Ugander, Johan, Brian Karrer, Lars Backstrom, and Cameron Marlow. 2011. The Anatomy of the Facebook Social Graph. arXiv:1111.4503 [Physics], November. http://arxiv.org/abs/1111.4503.

Big data sources are highly engineered platforms designed to induce specific behavior such as clicking on ads or posting content. These design choices can introduce patterns into the data in a process of algorithmic confounding. These patterns would not exist if it were not for these design choices.

A classic example comes from a study using Facebook friendships and social graph techniques to understand group formation in a digital social network. The authors analyzed friendship data directly from Facebook and discovered a significantly high number of users had approximately 20 friends. Why would 20 be such a magical number? Is there a feature of human behavior that causes us to naturally connect with just 20 individuals?

In fact, no. The authors were familiar with Facebook’s platform and algorithms designed to help users connect with friends on the platform. What happened is that Facebook users with just a few friends received prompts and encouragement to generate more connections with other Facebook users. These prompts stopped once they reached 20 friends. There is no natural law of social connections to explain this outcome — it is an artifact of the company’s decision to implement this nudge.

Unstructured

Sensitive

https://aboutmyinfo.org/identity

A very significant concern with big data is that it often contains sensitive or private information about individuals. This information may be very useful for certain research questions, but could lead to significant harm if the data were exposed to the public. This is a major reason why many big data sets are inaccessible to the public.

Sometimes it is easy to tell if a dataset contains sensitive information. In many situations, companies will try to anonymize the data before releasing it so that no observations can be traced back to a real human being. However many of these anonymization efforts are insufficient such that the data can be re-identified through any number of techniques. For example, most Americans can be uniquely identified using just their date of birth, gender, and ZIP code. If a dataset contains these variables, many of the observations could be linked back to the actual person.

This is an issue we will revisit when we discuss ethics and digital privacy later this quarter.

Incomplete

The final characteristic we will discuss is incompleteness. No matter how big your data set may be, it probably is missing at least some of the information you want. This is a common problem when we re-purpose data sets for research (the people who decided what to collect had different goals for this data!). One significant problem we already examined is creating an operationalized variable based on the theoretical construct we hope to study. This requires careful thought to ensure the thing you measure in a repurposed dataset is actually the thing you care about.

Other incompleteness problems are more readily solved. One is having incomplete demographic information about the users, while the other is having incomplete information on behavior on other platforms. This could be solved by collecting the data directly using a survey. Another technique is to impute the missing data using information researchers have on some people to infer attributes for other people. A third possibility is combining multiple data sources through record linkage.

Characteristics of Big Data that are…

…generally helpful for research: big, always-on, and non-reactive
…generally problematic for research: incomplete, inaccessible, nonrepresentative, drifting, algorithmically confounded, dirty, and sensitive

The future of digital trace data

Source: https://commons.wikimedia.org/wiki/File:Gartner_Hype_Cycle.svg

Clark et al. 2016 “Vaporous Marketing: Uncovering Pervasive Electronic Cigarette Advertisements on Twitter.”

Collected 850,000 tweets that used e-cigarette-related keywords from January 2012 through December 2014
80% of tweets were from bots (i.e. non-humans)
Tweets from bots were more positive than organic tweets (i.e. higher average sentiment score)

It turns out that Clark et al. did this exact study!

Collected 850,000 tweets that used e-cigarette-related keywords from January 2012 through December 2014
Upon closer inspection, they realized that many of these tweets were automated (i.e., not produced by humans) and many of these automated tweets were essentially commercials. They developed a human detection algorithm to separate automated tweets from organic tweets. Using this human detection algorithm they found that 80% of tweets were automated.
- ASK: How might this influence any findings about public sentiment of e-cigarette-related tweets? – They’re advertisements so they’re quite a bit more positive than human sentiment!
When they compared the sentiment in organic and automated twets, they found that the automated tweets were more positive than the organic tweets

So, good to be cautious about using big data for something it was not originally designed to be – lots of caveats to consider – otherwise highly misleading results.

#In next class, we’ll begin to more explicitly talk about research design strategies for operationalizing concepts when we’re working with big data. Can get complicated because we’re effectively using this data for something that it was not designed for, but lots of potential value!

#Be sure to read the readings ahead of class; we’ll discuss them heavily and you should be prepared to talk about them! [Hint: think through the reading worksheet that I shared with you last week.]

Review

A Theory: People who are more intelligent earn more money.

Two studies that make this claim based on their findings …

Study 1: People on Twitter who use longer words are more likely to mention luxury brands
- Longer words \(\stackrel{?}{\sim}\) more intelligent
- Luxury brand \(\stackrel{?}{\sim}\) earn more money
- Weak construct validity
Study 2: People who score well on the Raven Progressive Matrices Test have higher reported incomes on their tax returns
- Raven Progressive Matrices Test \(\sim\) Intelligence (Carpenter, Just, and Shell 1990)
- Reported Income on Tax Returns \(\sim\) Earnings
- Much stronger construct validity!

Let’s say that we have two groups of researchers who are testing the theory that “people who are more intelligent earn more money.” They both claim that their findings lend support for this theory.

Using Twitter data, one group of researchers performs a study that shows that people who use longer words are more likely to mention luxury brands.
- [walk through 2 relations] Weak construct validity – theoretical constructs seem only loosely connected to data. You can perhaps make the argument that they’re related, but only a pretty weak one.
We perform one study that finds that people who score well on the Raven Progressive Matrices Test (a well-studied test of analytic intelligence) have higher reported incomes on their tax returns.
- [walk through 2 relations] Much better construct validity

Both studies can make this claim based on these findings, but second one is much better construct validity

Example digital traces used to operationalize theoretical constructs

Table 2.2, *Bit by Bit*
Data source	Theoretical construct	References
Email logs from a university (meta-data only)	Social relationships	Kossinets and Watts (2006), Kossinets and Watts (2009), De Choudhury et al. (2010)
Social media posts on Weibo	Civic engagement	Zhang (2016)
Email logs from a firm (meta-data and complete text)	Cultural fit in an organization	Srivastava et al. (2017)

Looking at some real operationalizations of theoretical constructs for use with digital trace data: * email logs \(\sim\) social relationships (via network analysis – linking nodes together via messages between one another) * social media posts \(\sim\) civic engagement (post content reflective of increased civic concern – as per the studies at hand) * email logs from a firm \(\sim\) cultural fit in an organization (here evaluated the content of emails and were able to assess cultural fit via the similarity of language used)

So, bottom line is that we need a clear link somehow from the theoretical constructs we’re working with to the data we have. This is especially important to deeply consider when we’re working digital trace data that we didn’t custom design.

Once we have a good link can start to think about how to design our research in such a way to truly assess our social scientific theories…

Counting things

How frequently does \(x\) occur?
How does that frequency compare to that of \(y\)

Farber 2015: Comparing competing theories in labor economics

Do NYC taxi drivers work more hours on days with higher hourly wages, or on days with lower hourly wages?
Overall: drivers work more hours on days with higher wages
But: this is learned behavior

For instance, Henry Farber asked the simple question: “Do NYC taxi drivers work more hours on days with higher hourly wages, or days with lower hourly wages?”
- Important as an operationalization of two competing theories in labor economics:
  - neoclassical models predict that taxi drivers would work more days where they have higher hourly wages.
  - behavioral models predict the opposite (once target wage is met, then drivers will end up working fewer hours on days they’re earning more)
- Obtained data on every taxi trip taken by NYC cabs between 2009-2013 (which is publicly hosted in the AWS Cloud now) that contained info about each trip – start time/location, end time/location, fare, tip.
Overall: drivers work more hours on days with higher wages (as per neoclassical model)
Because of the size of data, Farber was also able to bin by different subgroups and found that newer drivers actually didn’t follow this neoclassical model strategy – however, they learned it over time and as they became more experienced, they gradually became neoclassical agents. Those who followed the “target-earner” strategy tended to quit driving a cab!

Counting things 2.0: deriving measurements

Measurements: Operational definition of a theoretical construct
Example: Measuring “Happiness” of members of the U.S. Congress as a function of multiple variables (Wojcik et al. 2015):
- Sentiment of tweets (number of positive vs. negative words)
- Intensity of smile in professional photographs

Measuring things using digital data is a slightly more advanced application of “counting” observational data. Rather than using the raw count or frequency of events, one combines these counts to derive a measurement, or an operationalized definition (typically in the form of a regression model, or the like). Remember that constructs are abstract ideas; we need to create operational definitions that accurately and validly measure that construct using real data. If we cannot do this, digital data will not inform our research.
Example: Measuring “Happiness” of members of the U.S. Congress as a function of multiple variables (Wojcik et al. 2015):
- Conservatives self-report being happier than liberals do
- To test this in using digital trace data, Wojcik et al. created a derived measure that incorporated the sentiment of tweets + intensity of smile –showing that liberals display, greater happiness (despite self-reporting disparity)
- So, instead of just counting number of positive words, etc. – combine this metric with another image-based metric to assess “happiness” Can be challenging to derive, but can sometimes be more convincing than a more simple metric (and then compare relative “happiness” between the two).

Reading Exercise

Chan et al. 2021 “Can psychological traits explain mobility behavior during the COVID-19 pandemic?”

What is the primary research question?
What data do the authors use to answer this question?
How do they attempt to bridge the gap between the theoretical constructs in their research question and the data they use in the paper? Assess the construct validity of this study.
How do the authors answer their research question? What methods do they use?
What are the authors’ main findings?
What are some strengths and limitations of this research (design)? Consider both the data and the methods used. (In other words, why did this paper get published?/What is its main contribution (special ingridient)? What is left unanswered?)

In-Class On Thursday

Shi et al. 2017 “Millions of online book co-purchases reveal partisan differences in the consumption of science”

What is the primary research question?
What data do the authors use to answer this question?
How do they attempt to bridge the gap between the theoretical constructs in their research question and the data they use in the paper? Assess the construct validity of this study.
How do the authors answer their research question? What methods do they use?
What are the authors’ main findings?
What are some strengths and limitations of this research (design)? Consider both the data and the methods used.

[~25 minutes: 15 min. in groups, 10 min. full class; write answers on board]

I had you read one such paper for today’s class that used counting as the prime metric for evaluation – Shi et al. 2017.

Let’s dissect this study using some of the principles we’ve been talking about this week with regard to the strengths and weaknesses of “Big Data” as well counting/measuring.

Reading Notes

Studies political polarization in the context of science
- Is science apolitical or nonpartisan?
Disconnect between survey analysis and observational data
- Asking people is not always the best strategy
- Misleading/incorrect answers
- Don’t remember
- Implicit bias/people are not actually aware of their polarization or true preferences on science
Uses book co-purchase data to reveal the network of consumption of science
- Novel research strategy
- Leverages digital trace data without requiring individual-level purchases
Counting strategy
- Count co-purchase links between scientific books and political books to identify the probability of a linkage given info on the nodes
- Calculates probabilities for a Binomial distribution
- Uses Bayes’ Rule(!) to calculate these values
Nonreactive
- Studying people in their natural state
- Free to purchase and consume the science they actually care about and are invested in
- Doesn’t require interference in the system
Inaccessibility
- Amazon and Barnes and Noble have no incentive to provide individual-level purchase data
- Instead, authors just scrape publicly accessible data from their websites
- Overcomes access issues
- Also limits the types of questions that can be answered
  - Can only describe the state of the network/co-purchase links
  - Undirected network analysis
  - Cannot assert causality
    - Does purchasing conservative books cause individuals to purchase specific books in the sciences?
    - Does purchasing scientific books which are more central lead to moderation in political ideology?
    - None of these questions are answerable with the existing data
Algorithmic confounding
- Should we trust the co-purchase data? It is derived from Amazon’s algorithms which are intended to sell more books
- Authors argue that is not a problem here because the algorithm’s intended purpose coincides with the research project - recommended books should be things individuals are already likely to purchase
- Do you buy this argument?
Open access data
- Site hosting code and data
- Not all source data is publicly available
- Analysis code and supplementary tables open for inspection and further exploration

Mental Exercise

Group Assignment

	Week 2 Discussion Groups
0	Andy / Abbey / Kuang
1	Jiazheng / Cosmo / Kexin
2	Pritam Rashtrapal / Agnes / Tianle
3	Huanrui / Tian / Max
4	Daniela / Ertong / Zhuojun
5	Emma / Lorena / Yue
6	Anny / Yuhan / Thomas

Read Stephens-Davidowitz (2014) and understand how the author used Google’s search records to measure racial animus in the United States. Critically evaluate the strengths and weaknesses of the measure.

Play around with Google Trends to see what functionalities it offers and try to figure out Google’s methodology by checking out information that Google publicly releases (such as its FAQ page).

Then, consider the following scenario:

You were asked to conduct a study on how the U.S. public’s attitude toward mask-wearing varies across regions during the COVID-19 pandemic. You thought maybe you could obtain some good measurements by making use of Google’s search records. How would you construct your measure(s)? What search queries (or combination of search queries) would you use? What research question(s) does your measure best answer? (i.e. what public attitude/interest does your query exactly measure?) What does it NOT measure? What are possible measurement errors and biases?

For future weeks, I will announce group assignments ahead of time through emails.
On each Thursday’s class, I will give you 10 minutes to decide on your group response. Hopefully, we will be able to have more time for presentations and Q&A.
I encourage you to start your group discussions before class.