Multimodal Deep Learning for the Detection of Racist Content Online
DOI:
https://doi.org/10.26438/ijcse/v13i7.19Keywords:
Content Moderation, Deep Learning, Multimodal Racism Detection, Social Media AnalyticsAbstract
This paper responds to the challenges of online racist content, especially insidious, context-dependent forms such as memes and image-text pairs, which tend to elude traditional unimodal content moderation. A multimodal deep learning model that is targeted at detecting this kind of content by jointly considering both textual and visual information. Our suggested approach combines VisualBERT, a vision-language representation-based transformer model, with a Vision Transformer (ViT) for high-level visual feature extraction. This combined model allows the system to capture context-dependent racist cues and successfully discern between offensive and non-offensive ones. The system was thoroughly tested on the Hateful Memes dataset, which contains more than 10,000 meme instances where multimodal understanding is required for proper classification. The model reported a validation accuracy of 0.79, with all recall values higher than 0.79 and an F1-score higher than 0.75 in training. Performance on unseen test data validated the model`s strong generalization ability, with accuracy between 0.76 and 0.79 and high recall at 0.70. These findings underscore its high suitability for improving content moderation as well as furthering safer online communities.
References
[1] A. Al‑Hassan and H. Al‑Dossari, “Detection of Hate Speech in Social Networks: A Survey on Multilingual Corpus”, Proceedings of the 6th International Conference on Computer Science and Information Technology (CS & IT – CSCP), Vol.9, Issue.2, pp.83– 100, 2019
[2] T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated Hate Speech Detection and the Problem of Offensive Language”, Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM), Vol.11, Issue.1, pp.512–515, ?2017.
[3] J.-S. Malik, G.-Pang, and A.-van-den-Hengel, “Deep Learning for Hate Speech Detection: A Comparative Study”, International Journal of Data Science and Analytics, pp.1–17, 2024.
[4] W. Warner and J. Hirschberg, “Detecting Hate Speech on the World Wide Web”, Proceedings of the Second Workshop on Language in Social Media (LSM 2012), pp.19–26, 2012.
[5] A. Mousa, I. Shahin, A. B. Nassif, and A. Elnagar, “Detection of Arabic offensive language in social media using machine learning models”, Intelligent Systems with Applications, Vol.22, Art.?No.200376, 2024.
[6] A. Alabdulwahab, M. A. Haq, and M. Alshehri, “Cyberbullying Detection using Machine Learning and Deep Learning”, International Journal of Advanced Computer Science and Applications, Vol.14, No.10, pp.1–7, 2023.
[7] D. C. Asogwa, C. I. Chukwuneke, C. C. Ngene, and G. N. Anigbogu, “Hate Speech Classification Using SVM and Naive BAYES”, IOSR Journal of Mobile Computing & Application (IOSR-JMCA), Vol.9, Issue.1, pp.27–34, ?2022.
[8] P. Badjatiya, S. Gupta, M. Gupta, and V. Varma, “Deep Learning for Hate Speech Detection in Tweets”, In the Proceedings of the 26th International Conference on World Wide Web Companion (WWW ’17 Companion), pp.759–760, 2017.
[9] S. Malik, H. Chopra, and A. Vashishtha, “Capturing racial & gender inequities on social media platforms using machine learning”, EAI Endorsed Transactions on Creative Technologies, Vol.9, Issue.31, pp.4, 2022.
[10] P. Kapil, G. Kumari, A. Ekbal, S. Pal, and A. Chatterjee, “HHSD: Hindi hate speech detection leveraging multi-task learning”, IEEE Access, Vol.11, pp.101460–101473, 2023.
[11] M. Mozafari, R. Farahbakhsh, and N. Crespi, “Hate speech detection and racial bias mitigation in social media based on BERT model”, PLOS ONE, Vol.15, Issue.8, pp.0237861, 2020.
[12] Marc Djandji, Fady Baly, Wissam Antoun, and Hazem Hajj, “Multi?Task Learning using AraBERT for Offensive Language Detection”, Proceedings of the 4th Workshop on Open?Source Arabic Corpora and Processing Tools (OSACT4), pp.97–101, 2020.
[13] J. A. Benítez-Andrades, Á. González-Jiménez, Á. López-Brea, and J. Aveleira-Mata, “Detecting racism and xenophobia using deep learning models on Twitter data: CNN, LSTM and BERT”, PeerJ Computer Science, Vol.8, pp.906, 2022.
[14] Md?Saroar Jahan and Mourad Oussalah, “A systematic review of hate speech automatic detection using natural language processing”, Neurocomputing, Vol.546, pp.126232, 2023.
[15] Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas, “Exploring Hate Speech Detection in Multimodal Publications”, Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.1459–1467, 2020.
[16] Roy Ka?Wei Lee, Rui Cao, Ziqing Fan, Jing Jiang, and Wen?Haw Chong, “Disentangling Hate in Online Memes”, Proceedings of the 29th ACM International Conference on Multimedia (MM ’21 Companion), pp.5138–5147, 2021.
[17] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine, “The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes”, 2020.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.
