Cross-Dataset Analysis of Language Models for Generalised Multi-Label Review Note Distribution in Animated Productions

During the production of an animated film, supervisors and directors hold daily meetings to evaluate in-progress material. Over the course of the several years it takes to complete a film, thousands of text notes outlining required fixes are generated. These notes are manually allocated to various departments for resolution. However, as with any manual process, a significant number of notes are either misassigned or overlooked entirely, which can negatively impact the final quality of the film. This paper investigates the performance of various methods for automating the distribution of review notes across relevant departments using datasets from multiple films produced by an animation studio in Madrid. Since a single note can pertain to multiple departments, the task is framed as a multi-label classification problem. The comparison of the results obtained with datasets from different movies provides critical insights for any Animation Studio evaluating the use of these methods in their process. The methods leverage Large Language Models (LLMs), including encoder-only models such as BERT and decoder-only models like Llama 2. Fine-tuning with QLoRA and in-context learning techniques were applied and evaluated across all datasets, and a cross-dataset analysis is presented. The fine-tuned encoder-only model achieved an F1-score of 0.98 for notes directed to the Animation department. Training was conducted locally on an RTX-3090 GPU, completing in under 30 minutes.

GARCÉS Diego;  SANTOS Matilde;  FERNANDEZ LLORCA David; 

2025-04-30

SPRINGERNATURE

JRC141027

1875-6883 (online),   

https://doi.org/10.1007/s44196-025-00785-9,    https://link.springer.com/article/10.1007/s44196-025-00785-9,    https://publications.jrc.ec.europa.eu/repository/handle/JRC141027,   

10.1007/s44196-025-00785-9 (online),