Running Digital Experiments

MACS 30000: Perspectives on Computational Analysis
University of Chicago

Assignment 1: Any questions?

Where does this experiment fall on the continuum of lab/field?
What are the strengths and limitations of the choice to design the experiment in this way?
Do the authors identify causal mechanisms in this study?

Is the experimental method particularly suited for the research subject and question? Or is it not?
What is its relative strength/weakness in comparison with an obversatinal design?

How would you study the same problem using an obserational design?

Salganik et al. 2006 - Inequality and Unpredictability in an Artificial Cultural Market.

What is the RQ
What is the treatment? How do the authors operationalize and randomize the treatment?
What is the outcome? How do the authors measure the outcome?
What are the findings?
Where does this experiment fall on the continuum of lab/field?

van de Rijt et al. 2014: Success-breeds-success experiments

What is the RQ
What is the treatment? How do the authors operationalize and randomize the treatment?
What is the outcome? How do the authors measure the outcome?
What are the findings?
Where does this experiment fall on the continuum of lab/field?

RQ: Is success a result of hard-to-observe differences in ability or is it arbitrary (e.g. one fortunate success triggers another, and so on)?
Experiments:
- Hard to observe these types of effects when in a sterile lab environment – helps to run experiments on existing platforms to observe actors in natural environment (e.g. in a digital field experiment!)
- Recruitment: randomly select members of web communities and assign to control and treatment groups
- Treatment: Early success, as determined by platform
  - Kickstarter: randomly funded 100/200 unfunded projects (success \(=\) both source and recipient financially affected)
  - Epinions: randomly sampled unrated reviews that they deemed “very helpful” and awarded them a “very helpful” rating (success \(=\) recipient benefits materially, but no cost to source)
  - Wikipedia: Sampled from the top 1% of most productive editors and gave status award to randomly chosen subset (success \(=\) social status)
  - change.org: Sampled 200 early-stage campaign petitions and granted 12 signatures to a random subset (of 100) (success \(=\) ideological support)
- Measurement: kept record of subsequent indications of success (as indicated by treatment) by third parties after the treatment
Findings:
- Early success \(\rightarrow\) greater subsequent success
  - On Kickstarter there’s a 2x difference between treatment and control; others not as dramatic, but still statistically significant
  - Also an effect that holds over time – not just an initial boost, but a lasting cumulative advantage over control
  - In SI, they also measure possible processual mechanisms (e.g. enthusiasm on social media after a Kickstarter has donation – which is higher with treatment) – but emphasize that there are many unobservables and this prohibits them from fully assessing causal pathways
- Findings: Increasing the magnitude of early success does not produce greater subsequent success, though (no statistically significant effects – although some are close). The authors also suggest that this might also have implications for understanding social inequality – perhaps cumulative advantage alone is not as powerful as though (although, this ignores the larger systemic factors that go along with cumulative “success” outside of isolated web environments).

Discussion

Compare and contrast Salganik et al (2006) and van de Rijt et al (2014)
What are the relative strengths/weaknesses of the lab design vs. field design in terms of
- statistical validity
- construct validity
- internal validity
- external validity

Discussion Groups

0	Daniela / Tian / Max
1	Pritam / Tianle / Yue
2	Emma / Anny / Zhuojun
3	Jiazheng / Cosmo / Yuhan
4	Andy / Abbey / Ertong
5	Huanrui / Kuang / Kexin
6	Lorena / Agnes / Thomas

Running Digital Experiments

Assignment 1: Any questions?

Discussion: Bail et al. 2018 - political polarization on social media

Salganik et al. 2006 - Inequality and Unpredictability in an Artificial Cultural Market.

van de Rijt et al. 2014: Success-breeds-success experiments

Discussion

Discussion Groups