Educational Natural Language Processing (EduNLP)

Deutsch

The junior research group "EduNLP" is part of the research center CATALPA.

Automatic evaluation of free text answers and automated feedback for learners and teachers - to make this work very reliably in the future, our junior research group investigates how language processing methods can be used here.

Goals and research questions

This project investigates how natural language processing methods can be used to automatically score free-text answers and provide learners and teachers with automatic feedback.

A core research question of the project is how learners can be provided with formative feedback about their essays. This can cover different aspects of writing such as syntactic or lexical variance, structure and argumentation, coherence, thematic fit with the topic or use of figurative language.

A number of sub-questions arise from this overall goal, such as:

  • What is the performance of existing scoring algorithms for a certain phenomenon and how can they be adapted for a specific use-case?
  • What are properties of useful formative feedback from the learners’ perspective and how can datasets with such feedback messages be collected?
  • How can we automatize such feedback, for example by training a decision tree to select an appropriate feedback message from a pool of human-created messages or by using natural language generation methods to produce the feedback?
  • How do humans judge such feedback, e.g. in terms of understandability and naturalness and how does the feedback influence the learning outcome?

  • Prof. Dr. Andrea Horbach

    • Yuning Ding (Doctoral Researcher, start date: 01.01.2022)
    • Viet Nguyen (student Assistant, start date: 01.02.2022)
    • Finn Brodmann (student Assistant, start date: 01.02.2022)
    • Joey Pehlke (student Assistant, start date: 01.02.2022)

  • December 2021 – November 2024

  • 2024

    Talks and Poster Presentations

    • Wehrhahn, F., Ding, Y., Gaschler, R., Zhao, F., & Horbach, A. (2024, June 26–28). Argumentative essay writing practice with automated feedback and highlighting. [Poster Presentation].

    2023

    Journals

    • Zesch, T., Horbach, A., & Zehner, F. (2023). To score or not to score: Factors influencing performance and feasibility of automatic content scoring of text responses. Educational Measurement: Issues and Practice, 42(1), 44–58. https://doi.org/10.1111/emip.12544

    Conferences

    • Bexte, M., Horbach, A., & Zesch, T. (2023). Similarity-based content scoring - a more classroom-suitable alternative to instance-based scoring? Findings of the Association for Computational Linguistics: ACL 2023, 1892–1903. https://aclanthology.org/2023.findings-acl.119
    • Ding, Y., Bexte, M., & Horbach, A. (2023a). CATALPA_EduNLP at PragTag-2023. In M. Alshomary, C.-C. Chen, S. Muresan, J. Park, & J. Romberg (Eds.), Proceedings of the 10th workshop on argument mining (pp. 197–201). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.argmining-1.22
    • Ding, Y., Bexte, M., & Horbach, A. (2023b). Score it all together: A multi-task learning study on automatic scoring of argumentative essays. Findings of the Association for Computational Linguistics: ACL 2023, 13052–13063. https://aclanthology.org/2023.findings-acl.825
    • Ding, Y., Trüb, R., Fleckenstein, J., Keller, S., & Horbach, A. (2023). Sequence tagging in EFL email texts as feedback for language learners. Proceedings of the 12th Workshop on NLP for Computer Assisted Language Learning, 53–62.
    • Mousa, A., Laarmann-Quante, R., & Horbach, A. (2023). Manual and automatic identification of similar arguments in EFL learner essays. Proceedings of the 12th Workshop on NLP for Computer Assisted Language Learning, 85–93.

    Proceedings

    • Kochmar, E., Burstein, J., Horbach, A., Laarmann-Quante, R., Madnani, N., Tack, A., Yaneva, V., Yuan, Z., & Zesch, T. (2023). Proceedings of the 18th workshop on innovative use of NLP for building educational applications (BEA 2023).

    Talks and Poster Presentations

    • Zehner, F., Zesch, T., & Horbach, A. (2023a, February 28–March 2). Mehr als nur Technologie- und Fairnessfrage: Ethische Prinzipien beim automatischen Bewerten von Textantworten aus Tests [Paper Presentation]. 10th GEBF Annual conference, Universität Duisburg-Essen.
    • Zehner, F., Zesch, T., & Horbach, A. (2023b, February 28–March 2). To score or not to score? Machbarkeits- und performanzfaktoren für automatisches scoring von textantworten [Paper Presentation]. 10th GEBF annual conference, Universität Duisburg-Essen.

    2022

    Conferences

    • Bexte, M., Horbach, A., & Zesch, T. (2022). Similarity-based content scoring - how to make S-BERT keep up with BERT. Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 118–123. https://aclanthology.org/2022.bea-1.16
    • Bexte, M., Laarmann-Quante, R., Horbach, A., & Zesch, T. (2022). LeSpell - a multi-lingual benchmark corpus of spelling errors to develop spellchecking methods for learner language. Proceedings of the Language Resources and Evaluation Conference, 697–706. https://aclanthology.org/2022.lrec-1.73
    • Ding, Y., Bexte, M., & Horbach, A. (2022). Don’t drop the topic - the role of the prompt in argument identification in student writing. Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 124–133. https://aclanthology.org/2022.bea-1.17
    • Horbach, A., Laarmann-Quante, R., Liebenow, L., Jansen, T., Keller, S., Meyer, J., Zesch, T., & Fleckenstein, J. (2022). Bringing automatic scoring into the classroom–measuring the impact of automated analytic feedback on student writing performance. Swedish Language Technology Conference and NLP4CALL, 72–83. https://ecp.ep.liu.se/index.php/sltc/article/view/580/550
    • Laarmann-Quante, R., Schwarz, L., Horbach, A., & Zesch, T. (2022). ‘Meet me at the ribary’ – acceptability of spelling variants in free-text answers to listening comprehension prompts. Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 173–182. https://aclanthology.org/2022.bea-1.22

    Proceedings

    • Kochmar, E., Burstein, J., Horbach, A., Laarmann-Quante, R., Madnani, N., Tack, A., Yaneva, V., Yuan, Z., & Zesch, T. (Eds.). (2022). Proceedings of the 17th workshop on innovative use of NLP for building educational applications (BEA 2022). Association for Computational Linguistics. https://aclanthology.org/2022.bea-1.0

    Chapters in Edited Books

    • Horbach, A. (2022). Werkzeuge für die automatische Sprachanalyse. In M. Beißwenger, L. Lemnitzer, & C. Müller-Spitzer (Eds.), Forschen in der Linguistik. Eine Methodeneinführung für das Germanistik-Studium. Wilhelm Fink (UTB).

    2021

    Journals

    Conferences

    • Bexte, M., Horbach, A., & Zesch, T. (2021). Implicit Phenomena in Short-answer Scoring Data. Proceedings of the First Workshop on Understanding Implicitand Underspecified Language.
    • Haring, C., Lehmann, R., Horbach, A., & Zesch, T. (2021). C-Test Collector: AProficiency Testing Application to Collect Training Data for C-Tests. Proceedings of the 16th Workshop on Innovative Use of NLP for BuildingEducational Applications, 180–184. https://www.aclweb.org/anthology/2021.bea-1.19

    Proceedings

    • Burstein, J., Horbach, A., Kochmar, E., Laarmann-Quante, R., Leacock, C., Madnani, N., Pilán, I., Yannakoudakis, H., & Zesch, T. (Eds.). (2021). Proceedings of the 16th Workshop on Innovative Use of NLP for BuildingEducational Applications. Association for Computational Linguistics. https://www.aclweb.org/anthology/2021.bea-1.0

    2020

    Journals

    Conferences

    • Ding, Y., Horbach, A., Wang, H., Song, X., & Zesch, T. (2020). Chinese ContentScoring: Open-Access Datasets and Features on Different SegmentationLevels. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International JointConference on Natural Language Processing(AACL-IJCNLP 2020). https://www.aclweb.org/anthology/2020.aacl-main.37.pdf
    • Ding, Y., Riordan, B., Horbach, A., Cahill, A., & Zesch, T. (2020). Don’t take "nswvtnvakgxpm" for an answer - The surprising vulnerability of automatic content scoring systems to adversarial input. Proceedings of the 28th International Conference on Computational Linguistics(COLING 2020). https://www.aclweb.org/anthology/2020.coling-main.76.pdf
    • Horbach, A., Aldabe, I., Bexte, M., Lacalle, O. de, & Maritxalar, M. (2020). Appropriateness and Pedagogic Usefulness of Reading ComprehensionQuestions. Proceedings of the 12th International Conference on LanguageResources and Evaluation (LREC-2020). https://www.aclweb.org/anthology/2020.lrec-1.217.pdf

    2019

    Journals

    2018

    Conferences

    • Horbach, A., & Pinkal, M. (2018). Semi-Supervised Clustering for Short AnswerScoring. LREC. http://www.lrec-conf.org/proceedings/lrec2018/pdf/427.pdf
    • Horbach, A., Stennmanns, S., & Zesch, T. (2018). Cross-lingual Content Scoring. Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, 410–419. http://www.aclweb.org/anthology/W18-0550
    • Zesch, T., & Horbach, A. (2018). ESCRITO - An NLP-Enhanced Educational Scoring Toolkit. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). http://www.lrec-conf.org/proceedings/lrec2018/pdf/590.pdf
    • Zesch, T., Horbach, A., Goggin, M., & Wrede-Jackes, J. (2018). A flexible online system for curating reduced redundancy language exercises and tests. In P. Taalas, J. Jalkanen, L. Bradley, & S. Thouësny (Eds.), Future-proof CALL: Language learning as exploration and encounters - short papers from EUROCALL 2018 (pp. 319–324). https://doi.org/10.14705/rpnet.2018.26.857

    2017

    Conferences

    • Horbach, A., Ding, Y., & Zesch, T. (2017). The Influence of Spelling Error onContent Scoring Performance. Proceedings of the 4th Workshop on NaturalLanguage Processing Techniques for Educational Applications, 45–53. http://www.aclweb.org/anthology/W17-5908
    • Horbach, A., Scholten-Akoun, D., Ding, Y., & Zesch, T. (2017). Fine-grained essay scoring of a complex writing task for native speakers. Proceedings of the Building Educational Applications Workshop at EMNLP, 357–366. http://aclweb.org/anthology/W17-5040
    • Riordan, B., Horbach, A., Cahill, A., Zesch, T., & Lee, C. M. (2017). Investigating neural architectures for short answer scoring. Proceedings of the BuildingEducational Applications Workshop at EMNLP, 159–168. http://www.aclweb.org/anthology/W17-5017
Research Center CATALPA | 19.04.2024