In this paper we report on the complexity of persuasion technique annotation in the context of a large annotation campaign involving 9 languages and approximately 40 annotators. We highlight the techniques that appear to be difficult for humans to annotate.
We introduce HolisticIAA, a new word embedding-based annotator agreement metric and we report on various experiments using this metric and its correlation with the traditional Inter Annotator Agreement (IAA) metrics. However, given somewhat limited and loose interaction between annotators, i.e., only a few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for IAA between annotators tasked to annotate different documents and in different languages. We present our preliminary results on this research problem.
STEFANOVITCH Nicolas;
PISKORSKI Jakub;
2024-10-30
Association for Computational Linguistics
JRC134232
https://aclanthology.org/2023.emnlp-main.6.pdf,
https://publications.jrc.ec.europa.eu/repository/handle/JRC134232,
| Name | Country | City | Type |
|---|
This document is only visible at the Commission level.
You are not authorized to publish or distribute it outside the European Commission.
This is a public document. You can share this publication.
Datasets
| ID | Title | Public URL |
|---|
Dataset collections
| ID | Acronym | Title | Public URL |
|---|
Scripts / source codes
| Description | Public URL |
|---|
Additional supporting files
| File name | Description | File type |
|---|