When algorithms failed us.

The first example shows the results of an experimental setting by researchers at CMU. They found out that male browsers are four times more likely than female browsers to see a job-related Ad promising high salaries. Hence, the algorithm used by Google Ads is biased towards men. But why? Unfortunately, as previously discussed, algorithms are also prone to biases that render their decisions unfair.

The data was collected by creating 10,000 fresh browser instances and randomly assigning them to treatment (namely, “female”) and control (“male”) groups. The browsers visited 100 employment websites (Indeed, Glassdoor, and Monster, among others) and finally visited the Times of India to capture the Ads shown to these browsers.

The following example shows the pronouns used by the Google Translate API when translating different job positions from English to non-gender-neutral languages such as Spanish. For example, take a look at the results above; for job occupations like “engineer” and “CEO,” the algorithm chooses a male pronoun in Spanish, whereas for “nurse” and “librarian,” it chooses a female pronoun. This is very problematic for reasons that go beyond the scope of this work. However, I will insist on the following: algorithms should mitigate these biases to avoid eliciting their users to unfair and discriminatory associations—between job occupations and sex, in this case.

In the example on the right, we see that the women’s appearance percentage in the top 100 images shown by Google Images strongly correlates with the Bureau of Labor Statistics estimates of women’s participation in the workforce. And what’s interesting (but not in a positive way) is that for job occupations in which women are overrepresented (model, flight attendant, among others), Google Images is more like to show images of women. This is also problematic because it reinforces gender stereotypes that some organizations and individuals are trying so hard to fight[viii] [ix].

Should AI and ML show the world as it is or as it should be?

In the literature, there is some discussion around whether algorithms should show the world as it is (discriminatory and sexist, unfortunately) or should take a step further and apply affirmative actions to help historically disadvantaged populations. Going back to the Google Images on the right, some would say that the algorithm just replicates what it’s seeing in the world, thus not introducing any bias. Still, some would argue that it’s not enough, and researchers and practitioners should introduce debiasing techniques to show, at a minimum, a 50/50 distribution for each occupation between women and men, for example.

The examples that I showed before might be subtle and don’t directly affect people’s lives. But some others can have deadly consequences. In my final example, we’ll see an algorithm used to manage the population’s health that exhibits racial bias; sicker Black patients are assigned the same level of risk as healthier White patients. According to the researchers[x], the bias arises because the algorithm predicts health care costs rather than illness. However, the algorithm does not consider unequal access to care —the Black population is widely known to be at a disadvantage—which, again, translates into Black patients not receiving the care they need. Finally, the researchers argue that “remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.”