Assisted Scoring of Learner Answers through Normalization

Deutsch

ASLAN is a DFG-funded CATALPA project. Project number 563947383.

In addition to imparting knowledge, assessing learning success is one of the most important tasks in any teaching and learning process. Free-text assignments, i.e., assignments that require a short, freely formulated text as an answer, play a central role in many subjects. The researchers are investigating how they can use AI to support the correction of such assignments and are taking a novel approach.

Project goals and research questions

Due to the variance in learner responses, fully automated assessment using machine learning methods has so far only been possible to a very limited extent. In particular, the lack of transparency and explainability of the models stands in the way of practical application.

We therefore propose an alternative paradigm for the use of artificial intelligence and language processing to support assessment, in which the actual assessment is not automated (with all the associated acceptance problems and ethical challenges), but rather the assessment continues to be carried out by humans with the aid of text normalization methods.
Our aim is to make the assessment proposals more explainable, easier to adapt to new tasks, and at the same time significantly reduce the burden on teachers in terms of time-consuming assessment tasks.

In order to achieve these goals, a central focus of the project is on the further development of text normalization procedures so that the linguistic variance of the answers can be significantly reduced.

Another focus is on practical integration into Moodle as an eAssessment system and the implementation of usage studies under realistic conditions.

In particular, we are investigating how evaluators interact with the system and how this approach influences the accuracy and time required for evaluations compared to conventional human evaluation on the one hand and a fully automated evaluation model on the other.

The research community will benefit from the project in many ways beyond the research results achieved: An annotated data set will be created for research into text normalization procedures, which will be freely available beyond the project and will stimulate further investigation of the underlying linguistic phenomena. Normalization will be organized in part as a shared task with the community, thus stimulating further research in this area. All tools developed, in particular a prototype for AI-supported evaluation, will also be made available as open-source projects.