Leveraging large language models to identify engagement-driving features in vaping-related TikTok videos: a cross-sectional study (Preprint)

• DOI: 10.2196/preprints.76265
• Publication Date: 2025-04-21T04:50:07Z
• Authors: Zidian Xie; Nanda Kishore Kor...; Amisha Dubey; Luchuan Song; Chenliang Xu; Karen M. Wilson; AnaPaula Cupertino; Dongmei Li

BACKGROUND Electronic cigarette (e-cigarette) use is prevalent in youth and young adults. TikTok, a popular social media platform for youth and young adults, has been used to disseminate e-cigarette-related videos, primarily dominated by promotional videos.

OBJECTIVE We aim to identify key e-cigarette-related TikTok video features associated with high user engagement to assist with future video design for vaping prevention campaigns.

METHODS We collected 1,487 e-cigarette-related TikTok videos and related metadata using the TikTok API (Application Programming Interface). We applied large language models GPT-4 and Video-LLaMA to extract video features (e.g., promotion content, background, gender, lifestyle, talking, cartoon, vaping tricks, containing emoji) from e-cigarette-related TikTok videos. We randomly selected and hand-coded 25 videos to check the accuracy of two models in identifying these video features. We utilized generalized linear models with identity link functions to identify significant video features associated with high TikTok user engagement (likes + shares + comments)/views.

RESULTS Compared to the Video-LLaMA model, the GPT-4 model exhibited higher accuracy (83%-100 % vs. 24%-88 %) in video feature identification. Notably, video backgrounds in cars, private spaces, or shops demonstrated significantly higher user engagement than in public spaces. Moreover, videos featuring young adults, smoking or vaping, talking, vape tricks, containing emojis, or funny and silly content exhibited heightened user engagement. Conversely, videos with promotional content or featuring e-cigarettes experienced lower engagement.

CONCLUSIONS TikTok video features like background settings, young adult presence, talking, and containing emojis substantially enhance user engagement. These insights offer valuable guidance for designing compelling videos in vaping prevention campaigns to improve social media user engagement.