EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media

Zahra Kolagar, Sebastian Steindl, Alessandra Zarcone


Abstract
This study explores the capacity of large language models (LLMs) to efficiently generate summaries of informal educational content tailored for platforms like TikTok. It also investigates how both humans and LLMs assess the quality of these summaries, based on a series of experiments, exploring the potential replacement of human evaluation with LLMs. Furthermore, the study delves into how experienced content creators perceive the utility of automatic summaries for TikTok videos. We employ strategic prompt selection techniques to guide LLMs in producing engaging summaries based on the characteristics of viral TikTok content, including hashtags, captivating hooks, storytelling, and user engagement. The study leverages OpenAI’s GPT-4 model to generate TikTok content summaries, aiming to align them with the essential features identified. By employing this model and incorporating human evaluation and expert assessment, this research endeavors to shed light on the intricate dynamics of modern content creation, where AI and human ingenuity converge. Ultimately, it seeks to enhance strategies for disseminating and evaluating educational information effectively in the realm of social media.
Anthology ID:
2023.eval4nlp-1.4
Volume:
Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2023
Address:
Bali, Indonesia
Editors:
Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, Andreas Rücklé
Venues:
Eval4NLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
32–48
Language:
URL:
https://aclanthology.org/2023.eval4nlp-1.4
DOI:
10.18653/v1/2023.eval4nlp-1.4
Bibkey:
Cite (ACL):
Zahra Kolagar, Sebastian Steindl, and Alessandra Zarcone. 2023. EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 32–48, Bali, Indonesia. Association for Computational Linguistics.
Cite (Informal):
EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media (Kolagar et al., Eval4NLP-WS 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eval4nlp-1.4.pdf