Automation Bias as a Legal Problem

The human decision-making process is based on a limited rationality. In a world that is becoming ever more complex, considering all every single relevant factor when making decisions is becoming increasingly difficult. Providing support for human decision-making processes in the form of algorithm-based systems is thus ubiquitous and irreversible. The focus of (scientific) discussions, however, remains primarily on visions of “artificial intelligence,” robotics and fully automated decision processes. The risks inherent in algorithm-based decision support have not yet been sufficiently considered from either a legal or an institutional standpoint, but are nonetheless actually widespread.

Automation bias is a phenomenon, proven in behavioral psychology, that people, despite their own better knowledge, tend to defer to suggestions and decisions made by machines.Thanks to the widespread use of such systems, e.g. in education or benefits administration, these kinds of biases give rise to a leverage effect that increases the number of those affected. The risk of automation increases particularly in areas in which people are regularly required to make a large number of similar decisions. Expert knowledge and experience does not protect against this.

Organizations, managers and clerical workers all need to be made aware of these risks. Safeguarding quality and results and protecting fundamental rights through procedures are already established legal concepts. The risks inherent in algorithm-based decisions, however, have so far only been reflected in law in a few specific areas; the standard defined in Art. 22 of the GDPR, the “AI provision,” explicitly does not cover decision-support systems.

The project develops solutions to counter automation bias from a legal perspective with a strong focus on interdisciplinary findings, particularly in the fields of work ethics and psychology.