Photo: Getty
For years, a team at Amazon reportedly worked on software that vetted the resumes of job applicants in an effort to surface the most likely hires. It gradually became clear that no matter how hard engineers tried to fix it, the recruitment engine found a way to discriminate against women, Reuters reports.
On Wednesday, the outlet cited five sources familiar with the automated resume review program that began in 2014. According to those sources, a team that consisted of around a dozen engineers was tasked with building a program that would utilize machine learning to review a decade’s worth of resumes submitted to Amazon and its subsequent hiring practices. The goal was to teach an AI how to identify the most likely hires to streamline the list of potential recruits that would have to be subsequently vetted by human recruiters. From Reuters:
In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.
Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.
Gizmodo reached out to Amazon for comment on the report and to ask if the tool was ever used in its own hiring process, but we did not receive an immediate reply. Reuters’ sources claimed that recruiters at Amazon did look at recommendations flagged by the system but didn’t rely on its rankings.
The algorithm’s gender discrimination issues became apparent about a year into the project’s lifecycle and it was eventually abandoned last year, the report said. It appears one of the primary issues was the dataset that Amazon had to work with. Most of the resumes submitted to the company over the previous decade came from men, and the tech sector has been controlled by men from its earliest days.
Another issue cited in the report was the algorithm’s preference for language that was often used by male applicants. Common words and phrases like a proficiency in a certain programming language would be ignored and verbs like “executed” and “captured” were given more weight.
After 500 iterations that were each trained to understand 50,000 unique terms, the team just couldn’t get the tool to stop reverting to discriminatory practices, Reuters reported. As time went on, the models often spiraled into recommending unqualified applicants at random.
The team’s effort highlights the limitations of algorithms as well as the difficulty of automating practices in a changing world. More women are joining the tech sector and all of the major tech giants have diversity initiatives in some form or another. But change has been painfully slow. Machines simply do what we tell them to do. If a machine is learning from example and we can only provide a sexist example, we’ll get sexist results.
According to Reuters, a new team has been assembled at Amazon’s Edinburgh engineering hub to take another crack at the “holy grail” of hiring.
[Reuters]
Share This Story