Sunday, July 31, 2016

Research Matters

One of my first blog posts was about the fantastic book Between the World and Me, by Ta-Nehisi Coates. At that time, I said I was looking forward to writing about some academic research about racism and the use of force by police officers. As with many things in the blog, it took a while. In this case, it wasn't for lack of trying. Over the last six months, I found myself returning over and over again to google scholar, but was unable to find any compelling research in this area.

Then, the exact week that officer involved shootings became a major news story again, with two high-profile incidents, rallies across country, and then a shooting against police officers, a relatively high profile piece of economic research came out. A working paper, The Empirical Analysis of Racial Differences in the Police Use of Force by Roland G. Fryer, Jr  was posted on the website of the National Bureau of Economic Research (NBER). The paper examines whether African Americans, and other minority groups experience disproportionate amounts of force, after being stopped or encountered by the police.




Before I get into the analysis, two things really stood out about this paper in comparison to most other economics journal articles. First, was the introduction and the conclusion of paper were more passionate and emotional than those found in most rather bland papers. Read just the first paragraph and you'll see what I mean:
From “Bloody Sunday” on the Edmund Pettus Bridge to the public beatings of Rodney King, Bryant Allen, and Freddie Helms, the relationship between African-Americans and police has an unlovely history. The images of law enforcement clad in Ku Klux Klan regalia or those peaceful protesters being attacked by canines, high pressure water hoses, and tear gas are an indelible part of American history. For much of the 20th century, law enforcement chose to brazenly enforce the status quo of overt discrimination, rather than protect and serve all citizens. The raw memories of these injustices have been resurrected by several high profile incidents of questionable uses of force. 
It's surprising to find this type of language, words like"brazenly" and "raw memories" are usually few and far between. The paper ends with a simple sign-off "Black Dignity Matters." This  topic clearly and rightly inspires emotion even among highly respected and 'rational' economists. The emotion cannot be fully separated from the research, and it's easy to see why. What does it say that there is not much of a literature about this in the first place? And unless the results are clear that no racial bias exists, is there any amount that is acceptable? I don't think an author can begin an area of research like this, without emotionally engaging those questions.

Second, while researchers are generally hesitant to present their own work as the final source of anything, I have found they are often bold and authoritative about their findings. Papers are always a strange mix of caveating the imperfections and hyping the significance of what they did find. This research was visibly humble. It recognized that it was the beginning of an area of inquiry, and is by no means the last word on the topic. After reading the paper,  I learned it came under some criticism for its findings and messaging (which should be no surprise as this was a hot-button issue). This media  coverage totally missed the important point that this is the beginning, not the end, of a body of research.

On to the research itself: I am actually only going to briefly describe the study, as it has been well covered (this was a pretty decent summary). However, as I read the paper, in some weird weigh I found myself question the point of the research design. I felt uncomfortable about the study, but in an interesting way that is worth examining.

The author looked at 4 different data sources about police use of force, each with its own merits and deficiencies. Two were public data sources:  A dataset documenting the New York Stop and Frisk program, and the Police Public Contact Survey. Neither of these capture officer-involved shootings, but do capture various levels of reporting about use-of-force, such as pat-downs, hand-cuffs, pepper-spray, and batons. The other two data sources were culled from police-reports (sometimes hundreds of pages long) about police-public interactions where either a weapon was discharged, or for calls where it would have been likely that a weapon was discharged. This second set of datasets were hand-coded by researchers actually reading the reports and interpreting the free-form text into a series of structured variables about incident and its context.  But, they only came from a small number of city' police-reports, all of whom were willing to share the data; an obvious source of bias.

These data-sources only consider incidents where there is police-public contact. All the analyses measure is whether race is a significant factor in the use-of-force conditional on contact. It knowingly doesn't measure whether race is a factor in the amount of contact. However, at first-blush, this seems like a reasonable question. I would think we should have a police force that treats all individuals the same,once they are interacting with the public regardless of race.

The top level results are that race is a significant factor in all use of force, except for lethal shootings. For example, the paper reports the following result
In the raw data, blacks are 21.3 percent more likely to be involved in an interaction with police in which at least a weapon is drawn than whites and the difference is statistically significant. Adding our full set of controls reduces the racial difference to 19.4 percent.
This result appears to be roughly consistent across the data sources, varying levels of force, and different model specifications. Some versions of the analysis try, for example, to control for context or control for a "lesser" level of use of force being drawn initially. Reading the paper, or one of its summaries, can help break down the exact effect-sizes, differences between the data-sources, and explain some interesting empirical approaches the author used.

As any good economics paper does, the analysis continually layers on more and more controls. One of these controls really got my head spinning: "precinct-level fixed effects".

To understand what this jargon means, it's worth diving into a very little bit of econometrics. In any standard regression analysis, the researcher would predict the outcome of interest (the dependent variable) using the variables the influence it (the independent variables). So in this case, the research predicted a variable "use of force" using "race." This estimates the effect-size, called a coefficient. If coefficient is statistically different from zero, economists say there is an effect.

But obviously that simple analysis is not enough. If, for example, minorities are more likely to pull a weapon an officer, but no "pulling a weapon" independent variable is included, the weapon's effect would be captured in the "race" independent variable. So "pulling a weapon" should be included as a control.

A lot of data is naturally clustered, or related to other observations. Think of students going to the same school or officers working in the same precinct. Something would likely be similar about those individuals. They have the same teachers, or work in the same neighborhoods. But rather than quantifying every unique thing about each cluster, the regression can just include a "fixed effect" variable, that just tells the regression the individual belongs to a specific cluster. Then anything unique would be captured in this fixed effect. In the case officers and precinct, this may reflect the management style of the captain or the particular mix of crimes they are more likely to observe.

In this paper, some of the analysis includes precinct-level fixed effect and some don't. It gets layered on, as the author expands the analysis. To make sure I understood the paper as I was reading, I started asking myself "how do I interpret the results if coefficient on race is no longer statistically significant once precinct fixed-effects are included" (generally, this is not what happened by the way). It would mean that all variation use-of-force, that initially seemed to be due to race, was actually due to precinct. This means the some precincts use force more than others. But it also means the precincts that have more use-of-force are the ones that have more interactions with minorities.

But, wait, wouldn't that be the exact same as saying police officers use more force on minorities? The fact that the coefficient didn't lose significance (though it became a little less pronounced) when these controls were included means all precincts use more force when interacting with minorities. But the fact that the effect is significant when these controls are not included, suggests there is a differential, no matter what.

Similarly, the author lamented the lack of officer-level data in the stop and frisk data, which prevented including officer fixed effects. If included, and the coefficient lost its significance, the interpretation are just some officers that use more force, and they interact with minorities the most.

From a policy perspective, I guess the question isn't really "is there more use of force against minorities?". It is pretty clear from, all evidence, there is, at the very least because police interact with minorities disproportionately more frequently. The question is "at what level is the disparity use-of-force being injected into the system?". Is it at strategic department levels, when they decide where to allocate policing resources? Is it at precinct level, where captains are making choices about how to manage officers? Is it at the officer level, where some officers are just more likely to use force than others? Or is something that each officers brings to the table, because of their previous experiences and biases? By the way, the answer may be all of the above.

To put this in context, it means the author could have found no effect at all, but wouldn't be saying that there is no disparity in the use-of-force. It would mean that the disparity lives at the level where decisions about whether police will contact with the public. Instead it's attributing at least some of the effect to a lower level.

One other thing I noticed in this paper was a unique opportunity for data-science and econometrics to come together. The author lamented the large amount of work to turn hundred page documents describing a police-public encounter to a structured dataset with variables about the context of interaction. To me this is an opportunity. He has a dataset of memos, a schema of data he extracted, and a training set of labelled data. Somebody could machine-learning and natural language processing to do the extraction for him.

No comments:

Post a Comment