News

The Common Thread: How Do We Measure "Correctly"?

Both keynote speeches at the main conference revolved around the same fundamental challenge: "Essentially, they questioned which gold standard we’re actually using to evaluate and whether it even reflects the right thing," summarizes Marie Bexte, who attended the conference as a co-author. The topic is closely linked to bias research: When evaluation data is deliberately designed to isolate a specific bias, it becomes clear just how much the measurement tool itself influences the result.

Dr. Josef Ruppenhofer’s presentation demonstrated exactly how this problem manifests in his own research. At the main conference, he presented a paper from the now-completed DAKODA project, which focuses on the analysis of language learner data. Specifically, the paper addressed the automatic detection of word order. "Our evaluation faced challenges similar to those mentioned in the keynote speeches: Sometimes there is no ‘ground truth’ that everyone can agree on, and one should allow for multiple analyses to be considered plausible," said Ruppenhofer. With learner language (interlanguage), an additional layer comes into play: The annotators are classifying a language variety that they themselves do not speak. Methodologically, however, there is no way around this. Learner data must be interpreted in relation to the target language, even if that is precisely what creates the ambiguity one actually wants to avoid.

The Second Theme: Making Language Accessible

A second theme ran through the Klartext workshop, at which CATALPA was represented by Jeanette Bewersdorff’s presentation, "Linguistic Features of German Easy and Plain Language: A Cross-Dataset Analysis." Taken together, the workshop presentations and the keynote on wahl.chat – which demonstrated how election platforms can be made more accessible – made it clear that text simplification and barrier-free access to information are of the utmost importance.

Beyond KONVENS: Handwriting, Documents, Hieroglyphs

Photo: private

The fact that the Research Professorship in Computational Linguistics has a broader methodological scope than a single conference can demonstrate is evidenced by three additional papers by Said Yasin on automatic handwriting and document analysis, which were submitted and accepted independently of KONVENS:

• a journal article in IJDAR (International Journal on Document Analysis and Recognition),
• a paper presented at the main ICDAR conference, the world’s leading conference on document analysis,
• and a paper presented at the ICDAR DAS workshop on the automatic recognition of hieroglyphs – a further development of his master’s thesis.

Take-Aways for Other Researchers

For those who weren’t in Hamburg, this KONVENS conference offers a clear methodological guiding principle: Before evaluating a language model or an analytical method, it’s worth scrutinizing your own gold standard just as critically as the system itself. This applies to automated language diagnostics just as much as it does to bias analyses or the evaluation of text simplification. After all, wherever there can be more than one plausible "correct" answer, transparency regarding the limitations of one’s own evaluation data is not a secondary consideration but rather part of the research result itself.

Impact on Education and Everyday Life

What may seem highly technical and theoretical at first glance reveals, upon closer inspection, a real impact on teaching and learning and illustrates some of the areas on which the Research Professorship in Computational Linguistics at CATALPA is focusing:

• Digital Inclusion and Accessibility: Research on easy and plain language concretely demonstrates how language technology can make teaching and learning more accessible – and thus more equitable – in light of the growing diversity of students.
• Trustworthy, Verifiable AI: A critical examination of gold standards and bias in evaluation provides precisely the foundational work needed to make statements about the quality of AI systems reliable. Especially in the sensitive field of education, such an assessment is essential,
• Language Support and Educational Equity: The work in the DAKODA project on the automated analysis of learner language lays the groundwork for better diagnostics in the acquisition of German as a second or foreign language.
• Digitization of Handwriting: Handwriting research shows how the grading of exams, for example, can be made more efficient, as the need to decipher handwriting could be eliminated.

Four KONVENS presentations, one workshop paper, and three additional international publications on document analysis within just a few weeks – that’s more than just a strong conference showing. It offers a concise snapshot of how CATALPA at the FernUniversität in Hagen combines basic research in natural language processing with very concrete societal benefits.