Algorithms don't define "creditworthiness" or "high-risk individuals". Data scientists do. Shouldn't citizens have a say?
From TechCrunch
We have one eye in A/UK open for the emancipatory potentials of radical technologies - the possibility that their smartness and productivity can deliver resources of time and space into people’s lives. But we have another one open for how that politically and structurally comes about. Who shapes and designs these automations, for whose benefit and aimed at what outcomes, is crucial to talk about.
One tech we must keep both eyes on - and have done throughout this blog’s existence - is machine learning, otherwise known as “weak AI” (strong AI - fully simulating humanity - is, as yet, the stuff of SF fantasy).
Weak AI holds out the promise that something can make sense out of, and see patterns in, the tsunami of data that swirls around us in society - or that its algorithms and computations can help us in our own interpretations. (Our friend Vinay Gupta promises, for example, that software can fully monitor and indicate our carbon-emissions in our purchases and transactions).
Yet there is a gathering critique of the trustworthiness of these systems - particularly as they function in areas like criminal justice, or consumer finance, or health. The “Garbage In/Garbage Out” cliche from the earlier years of computer science still seems to be pertinent.
A piece from the Boston Review journal by Annette Zimmermann, Elena Di Rosa, Hochan Kim - somewhat dense, but certainly worth a patient read - calls out the technologists’ claims that they can program these systems to account for their mistakes.
Their core critique is here:
Developing algorithmic systems entails making many deliberate choices. For example, machine learning algorithms are often “trained” to navigate massive data sets by making use of certain pre-defined key concepts or variables, such as “creditworthiness” or “high-risk individual.”
The algorithm does not define these concepts itself; human beings—developers and data scientists—choose which concepts to appeal to, at least as an initial starting point.
It is implausible to think that these choices are not informed by cultural and social context—a context deeply shaped by a history of inequality and injustice.
The variables that tech practitioners choose to include, in turn, significantly influence how the algorithm processes the data and the recommendations it ultimately makes.
Making choices about the concepts that underpin algorithms is not a purely technological problem. For instance, a developer of a predictive policing algorithm inevitably makes choices that determine which members of the community will be affected and how.
Making the right choices in this context is as much a moral enterprise as it is a technical one. This is no less true when the exact consequences are difficult even for developers to foresee.
New pharmaceutical products often have unexpected side effects, but that is precisely why they undergo extensive rounds of controlled testing and trials before they are approved for use—not to mention the possibility of recall in cases of serious, unforeseen defect.
And of the examples they give - of the starting prejudices and assumptions of the instigators of these processes, fatally damaging their assessments - the criminal justice ones are the worst:
… Consider the machine learning systems used in predictive policing, whereby historical crime rate data is fed into algorithms in order to predict future geographic distributions of crime.
The algorithms flag certain neighborhoods as prone to violent crime. On that basis, police departments make decisions about where to send their officers and how to allocate resources.
While the concept of predictive policing is worrisome for a number of reasons, one common defense of the practice is that AI systems are uniquely “neutral” and “objective,” compared to their human counterparts.
On the face of it, it might seem preferable to take decision making power out of the hands of biased police departments and police officers. But what if the data itself is biased, so that even the “best” algorithm would yield biased results?
This is not a hypothetical scenario: predictive policing algorithms are fed historical crime rate data that we know is biased.
We know that marginalized communities—in particular black, indigenous, and Latinx communities—have been overpoliced. Given that more crimes are discovered and more arrests are made under conditions of disproportionately high police presence, the associated data is skewed.
The problem is one of overrepresentation: particular communities feature disproportionately highly in crime activity data in part because of how (unfairly) closely they have been surveilled, and how inequitably laws have been enforced.
It should come as no surprise, then, that these algorithms make predictions that mirror past patterns. This new data is then fed back into the technological model, creating a pernicious feedback loop in which social injustice is not only replicated, but in fact further entrenched.
It is also worth noting that the same communities that have been overpoliced have been severely neglected, both intentionally and unintentionally, in many other areas of social and political life.
While they are overrepresented in crime rate data sets, they are underrepresented in many other data sets (e.g. those concerning educational achievement).
Structural injustice thus yields biased data through a variety of mechanisms—prominently including under- and overrepresentation—and worrisome feedback loops result.
Even if the quality control problems associated with an algorithm’s decision rules were resolved, we would be left with a more fundamental problem: these systems would still be learning from and relying on data born out of conditions of pervasive and long-standing injustice.
Discriminatory garbage in… Is there only a Luddite response to such “computer says no” behaviour? Zimmerman, Di Rosa and Kim suggest that they could be subject to a “democratic agenda” - where communities get a change to discuss and decide whether these algorithms should be introduced into an area.
San Francisco has just passed a “Stop Secret Surveillance” ordinance, on an 8-1 vote, banning facial recognition in public areas. The ordinance states:
Decisions regarding if and how surveillance technologies should be funded, acquired, or used, and whether data from such technologies should be shared, should be made only after meaningful public input has been solicited and given significant weight.
At the heart of Silicon Valley, that’s a good precedent. But it may require a degree of localist militancy in the UK over the next few years, as the installation of facial recognition equipment by the Metropolitan police is rolled out in London.
See the resources behind this blog’s categories on technology, and tags on civic tech, automation, artificial intelligence and design thinking.