Assessing inter-annotator agreement from collaborative annotation campaign in marine bioacoustics - Equipe Observations Signal & Environnement Accéder directement au contenu
Article Dans Une Revue Ecological Informatics Année : 2021

Assessing inter-annotator agreement from collaborative annotation campaign in marine bioacoustics

Résumé

It is currently widely recognized that automated methods are crucial to help processing long-term recordings of marine bioacoustics. To evaluate the efficiency of such methods, it is essential to develop large-scale annotated datasets. However, besides being laborious and resource intensive, recent studies have suggested that such a task could also be highly subjective with the generation of annotator specific errors. In this work, we investigate the question of inter-annotator agreement from a multi-annotator annotation campaign performed on a marine bioacoustics dataset. After providing quantitative evidence of inter-annotator variability, we investigate potential sources on both the user annotation practice and the annotation data and task to better understand why and how such variability occurs. Our study reveals that the acoustic event type, the Signal-to-Noise Ratio of the acoustic event and the annotator profile are three examples of critical factors impacting the annotation results of a multi-annotator campaign.
Fichier principal
Vignette du fichier
manuscript_wo_journal.pdf (3.47 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03028306 , version 1 (05-02-2021)

Identifiants

Citer

Paul Nguyen Hong Duc, Maëlle Torterotot, Flore Samaran, Paul R. White, Odile Gérard, et al.. Assessing inter-annotator agreement from collaborative annotation campaign in marine bioacoustics. Ecological Informatics, 2021, 61, pp.101185-1 - 101185-12. ⟨10.1016/j.ecoinf.2020.101185⟩. ⟨hal-03028306⟩
193 Consultations
147 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More