Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign
In this paper we report on the complexity of persuasion technique annotation in the context of a large annotation campaign involving 9 languages and approximately 40 annotators. We highlight the techniques that appear to be difficult for humans to annotate.
We introduce HolisticIAA, a new word embedding-based annotator agreement metric and we report on various experiments using this metric and its correlation with the traditional Inter Annotator Agreement (IAA) metrics. However, given somewhat limited and loose interaction between annotators, i.e., only a few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for IAA between annotators tasked to annotate different documents and in different languages. We present our preliminary results on this research problem.
STEFANOVITCH Nicolas;
PISKORSKI Jakub;
2024-10-30
Association for Computational Linguistics
JRC134232
https://aclanthology.org/2023.emnlp-main.6.pdf,
https://publications.jrc.ec.europa.eu/repository/handle/JRC134232,
Additional supporting files
| File name | Description | File type | |
|---|
