Multimodal Sarcasm Detection via Hybrid Classifier with Optimistic Logic
DOI:
https://doi.org/10.26636/jtit.2022.161622Keywords:
Bi-GRU, improved CCA, LSTM, multimodal sarcasm detectionAbstract
This work aims to provide a novel multimodal sarcasm detection model that includes four stages: pre-processing, feature extraction, feature level fusion, and classification. The pre-processing uses multimodal data that includes text, video, and audio. Here, text is pre-processed using tokenization and stemming, video is pre-processed during the face detection phase, and audio is pre-processed using the filtering technique. During the feature extraction stage, such text features as TF-IDF, improved bag of visual words, n-gram, and emojis as well on the video features using improved SLBT, and constraint local model (CLM) are extraction. Similarly the audio features like MFCC, chroma, spectral features, and jitter are extracted. Then, the extracted features are transferred to the feature level fusion stage, wherein an improved multilevel canonical correlation analysis (CCA) fusion technique is performed. The classification is performed using a hybrid classifier (HC), e.g. bidirectional gated recurrent unit (Bi-GRU) and LSTM. The outcomes of Bi-GRU and LSTM are averaged to obtain an effective output. To make the detection results more accurate, the weight of LSTM will be optimally tuned by the proposed opposition learning-based aquila optimization (OLAO) model. The MUStARD dataset is a multimodal video corpus used for automated sarcasm discovery studies. Finally, the effectiveness of the proposed approach is proved based on various metrics.
Downloads
References
[1] K. Nimala, R. Jebakumar, and M. Saravanan, “Sentiment topic sarcasm mixture model to distinguish sarcasm prevalent topics based on the sentiment bearing words in the tweets”, Journal of Ambient Intelligence and Humanized, vol. 12, pp. 6801–6810, 2021. DOI: https://doi.org/10.1007/s12652-020-02315-1
View in Google Scholar
[2] Y. Kumar and N. Goel, “AI-Based Learning Techniques for Sarcasm Detection of Social Media Tweets: State-of-the-Art Survey”, SN Comput. Sci., vol. 1, no. 6, 2020. DOI: https://doi.org/10.1007/s42979-020-00336-3
View in Google Scholar
[3] A. Banerjee, M. Bhattacharjee, K. Ghosh et al., “Synthetic minority oversampling in addressing imbalanced sarcasm detection in social media”, Multimed. Tools Appl., vol. 79, pp. 35995–36031, 2020. DOI: https://doi.org/10.1007/s11042-020-09138-4
View in Google Scholar
[4] R. Justo, J.M. Alcaide, M.I. Torres et al., “Detection of Sarcasm and Nastiness: New Resources for Spanish Language”, Cogn. Comput., vol. 10, pp. 1135–1151, 2018. DOI: https://doi.org/10.1007/s12559-018-9578-5
View in Google Scholar
[5] R.A. Potamias, G. Siolas, and A. Stafylopatis “A transformerbased approach to irony and sarcasm detection”, Neural Comput. & Applic., vol. 32, pp. 17309–17320, 2020. DOI: https://doi.org/10.1007/s00521-020-05102-3
View in Google Scholar
[6] Y. Du, T. Li, M.S. Pathan et al., “An Effective Sarcasm Detection Approach Based on Sentimental Context and Individual Expression Habits”, Cogn. Comput., vol. 14, pp. 78–90, 2021. DOI: https://doi.org/10.1007/s12559-021-09832-x
View in Google Scholar
[7] L. Ren, B. Xua, H. Lin, X. Liu, and L. Yang, “Sarcasm Detection with Sentiment Semantics Enhanced Multi-level Memory Network”, Neurocomputing, vol. 401, pp. 320–326, 2020. DOI: https://doi.org/10.1016/j.neucom.2020.03.081
View in Google Scholar
[8] M.S. Razali, A.A. Halin, L.S.Y. Doraisamy, and N.M. Norowi, “Sarcasm Detection Using Deep Learning With Contextual Features”, IEEE Access, vol. 9, pp. 68609–68618, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3076789
View in Google Scholar
[9] S. Rathod, “Hybrid Metaheuristic Algorithm for Cluster Head Selection in WSN”, Journal of Networking and Communication Systems, vol. 3, no. 4, 2020. DOI: https://doi.org/10.46253/jnacs.v3i4.a1
View in Google Scholar
[10] N.S. Lakshmiprabha and S. Majumder, “Face recognition system invariant to plastic surgery”, 12th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 258–263, 2012. DOI: https://doi.org/10.1109/ISDA.2012.6416547
View in Google Scholar
[11] A. Onan and M.A. Toco˘glu, “A Term Weighted Neural Language Model and Stacked Bidirectional LSTM Based Framework for Sarcasm Identification”, IEEE Access, vol. 9, pp. 7701–7722, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3049734
View in Google Scholar
[12] Meherkandukuri, “Deep Convolutional Neural Network for Emotion Recognition via EEG Signal”, Journal of Computational Mechanics, Power System and Control, vol. 4, no. 2, 2021. DOI: https://doi.org/10.46253/jcmps.v4i2.a3
View in Google Scholar
[13] S. Rajeyyagari, “Automatic speaker diarization using deep LSTM in audio lecturing of e-Khool platform”, Journal of Networking and Communication Systems, vol. 3, no. 4, 2020. DOI: https://doi.org/10.46253/jnacs.v3i4.a3
View in Google Scholar
[14] J. Russel Fernandis, “ALOA: Ant Lion Optimization Algorithmbased Deep Learning for Breast Cancer Classification”, Multimedia Research, vol. 4, no. 1, 2021. DOI: https://doi.org/10.46253/j.mr.v4i1.a5
View in Google Scholar
[15] C.I. Eke, A.A. Norman, and L. Shuib, “Context-Based Feature Technique for Sarcasm Identification in Benchmark Datasets Using Deep Learning and BERT Model”, IEEE Access, vol. 9, pp. 48501–48518, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3068323
View in Google Scholar
[16] Y. Diao, et al., “A Multi-Dimension Question Answering Network for Sarcasm Detection”, IEEE Access, vol. 8, pp. 135152–135161, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2967095
View in Google Scholar
[17] A. Kumar, V.T. Narapareddy, V. Aditya Srikanth, A. Malapati, and L.B.M. Neti, “Sarcasm Detection Using Multi-Head Attention Based Bidirectional LSTM”, IEEE Access, vol. 8, pp. 6388–6397, 2020. DOI: https://doi.org/10.1109/ACCESS.2019.2963630
View in Google Scholar
[18] Y. Zhang et al., “CFN: A Complex-Valued Fuzzy Network for Sarcasm Detection in Conversations”, IEEE Transactions on Fuzzy Systems, vol. 29, no. 12, pp. 3696–3710, 2021. DOI: https://doi.org/10.1109/TFUZZ.2021.3072492
View in Google Scholar
[19] K. Rothermich, A. Ogunlana, and N. Jaworska, “Change in humor and sarcasm use based on anxiety and depression symptom severity during the COVID-19 pandemic”, Journal of Psychiatric Research, vol. 140, pp. 95–100, 2021. DOI: https://doi.org/10.1016/j.jpsychires.2021.05.027
View in Google Scholar
[20] P. Parameswaran, A. Trotman, and D. Eyers, “Detecting the target of sarcasm is hard: Really?”, Information Processing and Management, vol. 58, no. 4, 2021. DOI: https://doi.org/10.1016/j.ipm.2021.102599
View in Google Scholar
[21] N.Z.Z. Wang, “The paradox of sarcasm: Theory of mind and sarcasm use in adults”, Personality and Individual Differences, vol. 163, 2020. DOI: https://doi.org/10.1016/j.paid.2020.110035
View in Google Scholar
[22] R. Pandey, A. Kumar, J.P. Singh, and S. Tripathi, “Hybrid attention-based Long Short-Term Memory network for sarcasm identification”, Applied Soft Computing, vol. 106, 2021. DOI: https://doi.org/10.1016/j.asoc.2021.107348
View in Google Scholar
[23] N. Basavaraj Hiremath, and M.M. Patil, “Sarcasm Detection using Cognitive Features of Visual Data by Learning Model”, Expert Systems with Applications, vol. 184, 2021. DOI: https://doi.org/10.1016/j.eswa.2021.115476
View in Google Scholar
[24] D. Jain, A. Kumar, and G. Garg, “Sarcasm detection in mash- up language using soft-attention based bi-directional LSTM and feature-rich CNN”, Applied Soft Computing, vol. 91, 2020. DOI: https://doi.org/10.1016/j.asoc.2020.106198
View in Google Scholar
[25] Y. Wu et al., “Modeling Incongruity between Modalities for Multimodal Sarcasm Detection”, IEEE MultiMedia, vol. 28, no. 2, pp. 86–95, 2021. DOI: https://doi.org/10.1109/MMUL.2021.3069097
View in Google Scholar
[26] A. Kamal and M. Abulaish “CAT-BiGRU: Convolution and Attention with Bi-Directional Gated Recurrent Unit for Self-Deprecating Sarcasm Detection”, Cogn. Comput., vol. 14, pp. 91–109, 2022. DOI: https://doi.org/10.1007/s12559-021-09821-0
View in Google Scholar
[27] C.I. Eke, A.A. Norman, S. Liyana, and H.F. Nweke, “Sarcasm identification in textual data: systematic review, research challenges and open directions”, Artif. Intell. Rev., vol. 53, pp. 4215–4258, 2020. DOI: https://doi.org/10.1007/s10462-019-09791-8
View in Google Scholar
[28] A. Kumar and G. Garg, “Empirical study of shallow and deep learning models for sarcasm detection using context in benchmark datasets”, Journal of Ambient Intelligence and Humanized Computing, 2019. DOI: https://doi.org/10.1007/s12652-019-01419-7
View in Google Scholar
[29] L. Ren, H. Lin, B. Xu, et al., “Learning to capture contrast In sarcasm with contextual dual-view attention network”, Int. J. Mach. Learn. and Cyber. vol. 12, pp. 2607–2615, 2021. DOI: https://doi.org/10.1007/s13042-021-01344-2
View in Google Scholar
[30] Z.L. Chia, M. Ptaszyński, and M. Wroczyński, “Machine Learning and feature engineering-based study into sarcasm and irony classification with application to cyberbullying detection”, Information Processing and Management, vol. 58, no. 4, 2021. DOI: https://doi.org/10.1016/j.ipm.2021.102600
View in Google Scholar
[31] A.F. Hidayatullah and M.R. Ma’arif, “Pre-processing Tasks in Indonesian Twitter Messages”, Journal of Physics: Conference Series, vol. 801, 2017. DOI: https://doi.org/10.1088/1742-6596/801/1/012072
View in Google Scholar
[32] N. Hazim Barnouti, et al., “Face Detection and Recognition Using Viola-Jones with PCA-LDA and Square Euclidean Distance”, International Journal of Advanced Computer Science and Applications, vol. 7, no. 5, 2016. DOI: https://doi.org/10.14569/IJACSA.2016.070550
View in Google Scholar
[33] H. Pandey and R. Tiwari, “An Innovative Design Approach of Butterworth Filter for Noise Reduction in ECG Signal Processing based Applications”, Progress In Science in Engineering Research Journal PISER 12, vol. 2, pp. 332–337, 2014.
View in Google Scholar
[34] D. Kim, D. Seo, S. Cho, and P. Kang, “Multi-co-training for dokument classification using various document representations: TF–IDF, LDA, and Doc2Vec”, Information Sciences, vol. 477, pp. 15–29, 2019. DOI: https://doi.org/10.1016/j.ins.2018.10.006
View in Google Scholar
[35] C. Cheng, L. Chunping, H. Yan, and Y. Zhu, “A semi-supervised deep learning image caption model based on Pseudo Label and N-gram”, International Journal of Approximate Reasoning, vol. 131, pp. 93–107, 2021. DOI: https://doi.org/10.1016/j.ijar.2020.12.016
View in Google Scholar
[36] D. Cristinacce and T. Cootes, “Automatic feature localisation with constrained local models”, Pattern Recognition, vol. 41, no. 10, pp. 3054–3067, 2008. DOI: https://doi.org/10.1016/j.patcog.2008.01.024
View in Google Scholar
[37] O.C. Ai, M. Hariharan, S. Yaacob, and L.S. Chee, “Classification of speech dysfluencies with MFCC and LPCC features”, Expert Systems with Applications, vol. 39, no. 2, pp. 2157–2165, 2012. DOI: https://doi.org/10.1016/j.eswa.2011.07.065
View in Google Scholar
[38] T. Kronvall, M. Juhlin, J. Sward, S.I. Adalbjornsson, and A. Jakobsson, “Sparse modeling of chroma features”, Signal Processing, vol. 130, pp. 105–117, 2017. DOI: https://doi.org/10.1016/j.sigpro.2016.06.020
View in Google Scholar
[39] M. Kavitha, R. Gayathri, K. Polat, A. Alhudhaif, and F. Alenezi, “Performance evaluation of deep e-CNN with integrated spatial-spectral features in hyperspectral image classification”, Measurement, vol. 191, 2022. DOI: https://doi.org/10.1016/j.measurement.2022.110760
View in Google Scholar
[40] L. An, et al., “Multi-Level Canonical Correlation Analysis for Standard-Dose PET Image Estimation”, IEEE Transactions on Image Processing, vol. 25, no. 7, pp. 3303–3315, 2016. DOI: https://doi.org/10.1109/TIP.2016.2567072
View in Google Scholar
[41] X. Zhou, J. Lin, Z. Zhang, Z. Shao, and H. Liu, “Improved itracker combined with bidirectional long short-term memory for 3D gaze estimation using appearance cues”, Neurocomputing In Press, vol. 390, pp. 217–25, 2019. DOI: https://doi.org/10.1016/j.neucom.2019.04.099
View in Google Scholar
[42] D. Zhao, J. Wang, and Y. Zhang, “Extracting drug–drug interactions with hybrid bidirectional gated recurrent unit and graph convolutional network”, Journal of Biomedical Informatics, vol. 99, 2019. DOI: https://doi.org/10.1016/j.jbi.2019.103295
View in Google Scholar
[43] L. Abualigah, et al., “Aquila Optimizer: A novel meta-heuristic optimization algorithm”, Computers & Industrial Engineering, vol. 157, 2021. DOI: https://doi.org/10.1016/j.cie.2021.107250
View in Google Scholar
[44] B.R. Rajakumar, “Impact of Static and Adaptive Mutation Techniques on Genetic Algorithm”, International Journal of Hybrid Intelligent Systems, vol. 10, no. 1, pp. 11–22, 2013. DOI: https://doi.org/10.3233/HIS-120161
View in Google Scholar
[45] B.R. Rajakumar, “Static and Adaptive Mutation Techniques for Genetic algorithm: A Systematic Comparative Analysis”, International Journal of Computational Science and Engineering, vol. 8, no. 2, pp. 180–193, 2013. DOI: https://doi.org/10.1504/IJCSE.2013.053087
View in Google Scholar
[46] S.M. Swamy, B.R. Rajakumar, and I.R. Valarmathi, “Design of Hybrid Wind and Photovoltaic Power System using Opposition-based Genetic Algorithm with Cauchy Mutation”, IET Chennai Fourth International Conference on Sustainable Energy and Intelligent Systems (SEISCON 2013), 2013. DOI: https://doi.org/10.1049/ic.2013.0361
View in Google Scholar
[47] A. George and B.R. Rajakumar, “APOGA: An Adaptive Population Pool Size based Genetic Algorithm”, AASRI Procedia, vol. 4, pp. 288–296, 2013. DOI: https://doi.org/10.1016/j.aasri.2013.10.043
View in Google Scholar
[48] B.R. Rajakumar and A. George, “A New Adaptive Mutation Technique for Genetic Algorithm”, In proceedings of IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), pp. 1–7, 2012,. DOI: https://doi.org/10.1109/ICCIC.2012.6510293
View in Google Scholar
[49] F. Chakraborty, P.K. Roy, and D. Nandi, “Oppositional Elephant herding optimization with dynamic Cauchy mutation for multilevel image thresholding”, Evol. Intel. 12, pp. 445–467, 2019. DOI: https://doi.org/10.1007/s12065-019-00238-1
View in Google Scholar
[50] S.H.S. Moosavi and V.K. Bardsiri, “Poor and rich optimization algorithm: A new human-based and multi populations algorithm”, Engineering Applications of Artificial Intelligence, vol. 86, pp. 165–181, 2019. DOI: https://doi.org/10.1016/j.engappai.2019.08.025
View in Google Scholar
[51] F. Ahmed, “Social Spider Optimization Algorithm”, 2015.
View in Google Scholar
[52] M. Dehghani, Š. Hubalovsky, and P, Trojovsky, “Cat and Mouse Based Optimizer: A New Nature-Inspired Optimization Algorithm”, Sensors, vol. 21, no. 15, 2021. DOI: https://doi.org/10.3390/s21155214
View in Google Scholar
[53] M.O. Okwu and L.K. Tartibu, “Ant Lion Optimization Algorithm”, Metaheuristic Optimization: Nature-Inspired Algorithms Swarm and Computational Intelligence, Theory and Applications. Studies in Computational Intelligence, vol. 929, 2020. DOI: https://doi.org/10.1007/978-3-030-61111-8
View in Google Scholar
[54] Y. LeCun, K. Kavukvuoglu, and C. Farabet, “Convolutional networks and applications in vision”, Circuits and Systems, International Symposium on, pp. 253–256, 2010. DOI: https://doi.org/10.1109/ISCAS.2010.5537907
View in Google Scholar
[55] K. Ling-Jing and C.C. Chiu, “Application of integrated recurrent neural network with multivariate adaptive regression splines on SPC-EPC process”, Journal of Manufacturing Systems, vol. 57, pp. 109–118, 2020. DOI: https://doi.org/10.1016/j.jmsy.2020.07.020
View in Google Scholar
[56] Z. Masetic and A. Subasi, “Congestive heart failure detection using random forest classifier”, Computer Methods and Programs in Biomedicine, vol. 130, pp. 54–64, July 2016. DOI: https://doi.org/10.1016/j.cmpb.2016.03.020
View in Google Scholar
[57] P.T. Ilia, “Comparison of a logistic regression and Naive Bayes classifier in landslide susceptibility assessments: The influence of models complexity and training dataset size “, Catena, vol. 145, pp. 164–179, 2016. DOI: https://doi.org/10.1016/j.catena.2016.06.004
View in Google Scholar
[58] –, https://github.com/soujanyaporia/MUStARD.
View in Google Scholar
Downloads
Submitted
Published
Issue
Section
License
Copyright (c) 2022 Journal of Telecommunications and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
