Comparing tweet classifications by authors' hashtags, machine learning, and human annotators

Citation

Nishioka C, Scherp A & Dellschaft K (2016) Comparing tweet classifications by authors' hashtags, machine learning, and human annotators. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), volume 1. 2015 International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), 06.12.2015-09.12.2015. Singapore: Institute of Electrical and Electronics Engineers, pp. 67-74. https://doi.org/10.1109/WI-IAT.2015.69

Abstract
Over the last years, many papers have been published about how to use machine learning for classifying postings on microblogging platforms like Twitter, e.g., in order to assist users to reach tweets that interest them. Typically, the automatic classification results are then evaluated against a gold standard classification which consists of either (i) the hashtags of the tweets' authors, or (ii) manual annotations of independent human annotators. In this paper, we show that there are fundamental differences between these two kinds of gold standard classifications, i.e., human annotators are more likely to classify tweets like other human annotators than like the tweets' authors. Furthermore, we discuss how these differences may influence the evaluation of automatic classifications, like they may be achieved by Latent Dirichlet Allocation (LDA). We argue that researchers who conduct machine learning experiments for tweet classification should pay particular attention to the kind of gold standard they use. One may even argue that hashtags are not appropriate as a gold standard for tweet classification.

Keywords
Social media; comparative study; short text clarification; human experimentation;

Journal
Proceedings - 2015 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2015: Volume 1