Measuring Polarization in High-Dimensional Data: Method and Application to Congressional Speech
We study trends in the partisanship of Congressional speech from 1873 to 2009. We define partisanship to be the ease with which an observer could infer a congressperson's party from a fixed amount of speech, and we estimate it using a structural choice model and methods from machine learning. The estimates reveal that partisanship is far greater today than at any point in the past. Partisanship was low and roughly constant from 1873 to the early 1990s, then increased dramatically in subsequent years. Evidence suggests innovation in political persuasion beginning with the Contract with America, possibly reinforced by changes in the media environment, as a likely cause. Naive estimates of partisanship are subject to a severe finite-sample bias and imply substantially different conclusions.