How Algorithms Make Exceptions

Organizations have long used individuals’ pasts to predict those individuals’ likely future behaviors. Banks look at our credit histories to predict the likelihood of on-time mortgage payments; states look at criminal histories to predict the likelihood of recidivism when making parole decisions. These pasts tell stories, narratives of behavior that, some argue, produce essential knowledge about the future so organizations can make smart decisions today. At the same time, relying too much on our pasts threatens autonomy, makes us prisoners of our worst mistakes, and entrenches discrimination.

One notable difference today is that these and other predictive decisions are now made using algorithms, computer programs, and tools collectively called “artificial intelligence” (AI), all of which can introduce many wrinkles into the age-old story of predictive policymaking. In a masterful article in the American Journal of Sociology, the field’s most respected journal, Barbara Kiviat (Columbia), Sara Sternberg Greene (Duke), and Hesu Yoon (CREST, ENSAE Paris) focus on one wrinkle: the problem of exceptions.  When predictive decisions were made exclusively by humans, exceptions were easy to make. Social workers or bankers could look at the numbers and appreciate changes in circumstances; the human touch made that possible. The conventional wisdom is that the algorithmic turn erases that discretion.  Kiviat, Greene, and Yoon argue that the algorithmic turn instead complicates discretion, moving it upstream and changing it from an individualizing mechanism to one that problematically relies on cultural archetypes.

To some, the erasure of discretion was supposed to be a good thing. Discretion can be the gateway drug of discrimination; algorithms reduce everyone to numbers and “ostensibly treat everyone the same”. Kiviat, Greene, and Yoon nevertheless show “how algorithms treat some people as special”. Using a case study of tenant screening algorithms and relying on interviews with landlords and tenant screening executives, among others, they demonstrate that exceptions to predictions based on the past can be systematic, “within the rule systems themselves” rather than “ad hoc, one-off.” Exception-making is essential here because exceptions are sometimes the only ways those with otherwise disqualifying pasts can access loans, housing, and freedom from incarceration. The question is not whether exceptions are possible, but rather who is sufficiently legible to systems of power to benefit from exception-making that is baked into the design of machines that make probabilistic predictions about the future.

Ordinarily, landlords have no interest in renting to someone who is likely not to pay their rent, steal from their neighbors, or bring criminal activity to the property. But there are exceptions to this rule—namely, when they would lose even more money not renting to certain people or groups. If demand for housing is low, landlords might relax their standards. If the applicant pool is comprised of people who, for structural reasons, are overpoliced and overincarcerated, it might be hard for landlords to find anyone who meets overly strict background check standards. So exceptions are made. These exceptions comprise the starting point for Kiviat, Greene, and Yoon. They seek to uncover “how landlords overlook some background check blemishes but not others.”

For many smaller landlords, decisions about the potential risks posed by tenant applicants are made informally (the authors call these people “judgmental users of data”). If a smaller landlord finds a blemish on a record, they may call up the applicant and ask them to explain why they missed a payment or why their car was repossessed. Algorithmic decision-making, perhaps surprisingly, doesn’t erase that discretion; it retains that discretion but pushes it upstream. Kiviat, Greene, and Yoon found that larger landlords work with tenant screening companies to identify which blemishes should be disqualifying and which should not. The authors called these landlords “algorithmic users of data”. Medical debt, they found, is often ignored; a landlord might not “like a that a person was driving drunk, but how does that affect them living at your community?”

These insights generally align with existing research on the sociotechnical process of algorithmic decision-making. Where the authors’ work really shines is when it reveals trends in the reasons all landlords tend to overlook problematic pasts. Kiviat, Greene, and Yoon found that when landlords “construed would-be tenants as not at fault, reformed, or engaging in unrelated behavior, they more easily overlooked problems.” Small landlords asked questions of applicants, trying to get at underlying motivations. Bigger landlords who used algorithmic screening used their own assumptions and existing cultural narratives to generalize about the faultlessness of certain problems, like student debt or medical debt. For example, landlords mused that maybe those with student loans just hadn’t yet found the right job yet or maybe these applicants should be honored for getting an education and trying to improve their position in life.

There is a big difference, however, in these narrative explanations. The first focuses on a particular applicant: What happened in this circumstance? What was going on in this person’s life to explain this knock on their credit history? This approach to exceptions could be described as ad hoc. It could also be described as more tailored and more generous. Indeed, Kiviat, Greene, and Yoon found that those landlords who looked holistically at single applicants rather than use tenant screening algorithms were more likely to forgive even seemingly serious criminal histories as long as the explanation the applicant offered was persuasive.

The second explanation for this forgiveness narrative ignores the individual. Landlords “imagin[ed] circumstances” that might explain away something that would normally disqualify an applicant. To do this, executives drew on “cultural archetypes” rather than real life, such as “the student overwhelmed by the debt that paid for her education, the one-time drug dealer who had turned his life around.”

When decision-makers rely on cultural archetypes to determine which record blemishes to ignore, some blemishes are systematically ignored more than others. Narratives that are more common in the cultural zeitgeist—like student loan debt—led to systematic exceptions at the design stage. This has a dark side. Corporate executives at tenant screening companies and large landlords may be clued into a narrow slice of much broader and more diverse cultural experiences. Those that regularly surfaced in their world were more likely to benefit people like them.

Kiviat, Greene, and Yoon ultimately found that all types of landlords had record blemishes in mind that would not disqualify applicants, but the manner in which landlords assessed applicant risk affected how they were able to apply those exceptions in practice. The mostly smaller landlords who made their decisions judgmentally could almost always act on their views. The mostly larger landlords who made their decisions algorithmically and who had to bake in their exceptions into rules had to either consistently overlook a blemish or never overlook a blemish. This feature changed which data points would be considered for exceptions. For example, in the case of past evictions, algorithmic decision-makers would not make exceptions for the innocent tenants lest they make exceptions for the worst ones.

So, which record blemishes could be algorithmically and systematically ignored? First, those that had “come to be associated with an exculpatory cultural narrative”. Medical debt is paradigmatic. Millions of people from all walks of life have medical debt, and many understand how even wealthy families can suddenly go into debt when a loved one gets sick in a country that has no national health service. This means that storytelling, something we might assume would only play a role in judgmental decision-making, affected algorithmic screening as well.

Second, algorithmic screeners could only ignore those blemishes that were technically possible to ignore. Credit reports only make certain aspects of people’s lives legible, and someone at some point had to make a decision to write code that would enable a tenant screener to separate out certain types of infractions from others. If that categorization capability did not exist, algorithmic decision-makers were out of luck. They simply couldn’t, by virtue of system design, make a systematic exception even if they wanted to. This is yet another example of how the algorithmic society does not erase discretion but rather shifts it earlier and often to those with technical expertise.

Lawyers tend to think that rules and exceptions go together, especially in contrast to flexible standards. That is Law 101 for most first-year students. The algorithmic age complicates our traditional understanding of rules and exceptions. Kiviat, Green, and Yoon offer a master class, both methodologically and analytically, of this new world of coded rules and designed-in exceptions.