Bad Robots – Racial Bias

Bad Robot Outcome:
Algorithms determine that ‘black’ offenders twice as likely to reoffend as ‘white’ offenders

The Story

An AI software application called COMPAS, (Correctional Offender Management Profiling for Alternative Sanctions), owned by Equivant, Is used as a case management tool that assists US based courts make decisions about the likelihood of a person up before them in court, re-offending or becoming a recidivist. The AI has been used in courts in California, Florida, New York and Broward County amongst other states and places.

During 2016 there was an investigation into the COMPAS AI, named the ProPublica assessment. The team working on the investigation found that “blacks are almost twice as likely as whites to be labelled a higher risk but not actually re-offend.”

The algorithm also was only successful 20% of the time in predicting recidivism. Interestingly, a further study found that the COMPAS algorithm only had the same level of accuracy, around 65%, that people with little or no criminal justice experience had.

Propublica’s analysis also notes that “The COMPAS system unevenly predicts recidivism between genders. According to Kaplan-Meier estimates, women rated high risk recidivated at a 47.5 percent rate during two years after they were scored. But men rated high risk recidivated at a much higher rate – 61.2 percent – over the same time period. This means that a high-risk woman has a much lower risk of recidivating than a high-risk man, a fact that may be overlooked by law enforcement officials interpreting the score.”

Read more on this topic here:

https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

Our view

AI used in justice and policing is fraught with potential problems as demonstrated by the COMPAS example. And the implications of the determinations of the AI will have significant impact on people’s lives as they influence decisions judges make in sentencing or general policing.

As such there can be no errors, biases or other problems with the data, algorithm or decision making of the software when used for essential services like policing or in courts. Hence we believe that AI is still too immature for significant usage in courts and much greater development work on judge’s AI and algorithms needs to be performed before this type of application becomes commonplace.