IIARD International Institute of Academic Research and Development

INTERNATIONAL JOURNAL OF COMPUTER SCIENCE AND MATHEMATICAL THEORY (IJCSMT )

E-ISSN 2545-5699
P-ISSN 2695-1924
VOL. 9 NO. 1 2023

---

Hierarchical Deep Multi-Modal Neural Network for Classification of Small Scale Incidents in Microblogs

Sako, DJS, Bennett, EO and Igiri, CG

---

Abstract

Data imbalance is defined by great differences in the distribution of the classes in the dataset and is predominant and inherent in the real world. Addressing imbalanced data distribution is a difficult task for many classification algorithms as algorithms do not learn properly when a massive difference in size between data classes exist. This classification problem exists in many real world application domains, one of which is emergency situation management where we identify incident-related informative content from the Twitter streams especially for smaller scale incidents where there are only small bits of information. In this paper, we present a two-stage hierarchical multimodal deep learning neural network model (2HDLnet) to identify incident-related informative content from social media. This top-down level-based classification method entails the temporary regrouping and re-labelling of classes of the training data toward a more balanced distribution and performing hierarchical data classification using stacks of deep learning architectures to allow both overall and specialized learning at each level of the data hierarchy. We train separate CNN models for text, image and multimodal approaches at each level and combine the models at both levels to perform hierarchical classification. We consider the use of message contents from Twitter microblogging site consisting of texts and images in which they exist together or separately. The experimental analysis on a home-grown incident dataset demonstrates that the proposed approach can effectively classify crisis images and/or text tweets at each logical layer.

keywords:

Deep learning, Data imbalance, Hierarchical classification; Emergency situation, Multimodal Learning

---

References:

Abel, F., Hauff, C., Houben, G.-J., Tao, K. and Stronkman, R. (2012) Twitcident: Fighting
fire with information from social web streams. Proceedings of the 21st International
Conference on World Wide Web. ACM, Lyon, France, 305–308.

Alam, F., Imran, M. and Ofli, F. (2017). Image4Act: Online Social Media Image Processing
for Disaster Response. IEEE/ACM International Conference on Advances in Social
Networks Analysis and Mining, 601-604. http://dx.doi.org/10.1145/3110025.3110164

DOWNLOAD PDF

---

Back