Hierarchical Deep Multi-Modal Neural Network for Classification of Small Scale Incidents in Microblogs

Abstract

Data imbalance is defined by great differences in the distribution of the classes in the dataset and is predominant and inherent in the real world. Addressing imbalanced data distribution is a difficult task for many classification algorithms as algorithms do not learn properly when a massive difference in size between data classes exist. This classification problem exists in many real world application domains, one of which is emergency situation management where we identify incident-related informative content from the Twitter streams especially for smaller scale incidents where there are only small bits of information. In this paper, we present a two-stage hierarchical multimodal deep learning neural network model (2HDLnet) to identify incident-related informative content from social media. This top-down level-based classification method entails the temporary regrouping and re-labelling of classes of the training data toward a more balanced distribution and performing hierarchical data classification using stacks of deep learning architectures to allow both overall and specialized learning at each level of the data hierarchy. We train separate CNN models for text, image and multimodal approaches at each level and combine the models at both levels to perform hierarchical classification. We consider the use of message contents from Twitter microblogging site consisting of texts and images in which they exist together or separately. The experimental analysis on a home-grown incident dataset demonstrates that the proposed approach can effectively classify crisis images and/or text tweets at each logical layer.

Keywords

References