Bad Robots: Amazon Recruitment Software Discriminated Against Female Applicants
Bad Robot Outcome:
Amazon’s AI-enabled recruitment software tool “downgraded” resumes of job seekers that contained the word “women” or that otherwise implied the applicant was a woman.
The Story
Beginning in 2014 Amazon, the Seattle-based technology behemoth, began experimenting with ways in which they could automate their recruitment process. Their solution was an Artificial Intelligence-enabled system that assessed applicants’ resumes and then evaluated such resumes, giving them a numerical score ranging from zero to five.
In fact, the computer models were trained using resumes that had been submitted to Amazon over the previous ten years. During that period, a significant majority of all resumes were submitted by men.
As such, the system concluded that male applicants were preferable. Resumes containing the word “women” (e.g., “women’s soccer team”) were given lower rankings. The system even downgraded resumes of applicants who had graduated from two all-female universities.
The Fall-Out
Reuters first broke the story in October of 2018. Amazon immediately went on the defensive, stating that the tool was “never used by Amazon recruiters to evaluate candidates.”
Certain Amazon employees, speaking anonymously, had a slightly different story. According to these insiders, company recruiters did look at the rankings but did not solely rely on them.
Following Reuters’ first story, today a quick Google search will lead you to endless articles, blog posts, and opinion pieces about this now widely-publicized incident.
Our View
Artificial Intelligence is only as good as the data it receives. This story echoes the same pattern as many of our past “Bad Robot” posts. Bad data = bad robots.
Hindsight is 20/20, but the past is nothing unless a building block onto which we learn and grow. The team responsible for creating and implementing this particular AI system should have been more analytical regarding the information being uploaded into the system. Even a quick scan of the resumes would have indicated that the vast majority of them were submitted by male candidates. Perhaps engaging an expert in Diversity & Inclusion, alongside the technical team, would have prevented this from happening.
At the end of the day, robots are made by people. People are imperfect. People are biased. We at the Ethical AI Advisory hope that this example serves as a warning for all of those in the field of Artificial Intelligence to be thoughtful, analytical, and discerning. How might your data contain the biases of your organization, its practices, and its people? These are certainly not easy questions to ask, but they are important and they are necessary.
