شيماء ابراهيم حسن رزق
مدرس
Performance vs. Complexity Comparative Analysis of Multimodal Bilinear Pooling Fusion Approaches for Deep Learning-Based Visual Arabic-Question Answering Systems
Computer Modeling in Engineering & Sciences • 2025
معلومات البحث
المؤلفون
Sarah Kamel, Mai Fadel, Lamiaa Elrefaei, Shimaa Hassan
الكلمات المفتاحية
Arabic-VQA; deep learning-based VQA; deep multimodal information fusion; multimodal representa
tion learning; VQA of yes/no questions; VQA model complexity; VQA model performance; performance-complexity
trade-off
المجلة العلمية
Computer Modeling in Engineering & Sciences
الناشر
Tech Science Press
المجلد
143
العدد
1
الصفحات
373
publication.type
International
رابط البحث
Open Link
المواد المرفقة
Not Available
الملخص
Visual question answering (VQA) is a multimodal task, involving a deep understanding of the image
scene and the question’s meaning and capturing the relevant correlations between both modalities to infer the
appropriate answer. In this paper, we propose a VQA system intended to answer yes/no questions about real-world
images, in Arabic. To support a robust VQA system, we work in two directions: (Ô) Using deep neural networks to
semantically represent the given imageandquestionina•ne-grainedmanner,namelyResNet-Ô òandGatedRecurrent
Units (GRU).(ò)Studyingtheroleoftheutilizedmultimodalbilinearpoolingfusiontechniqueinthetrade-ošbetween
the model complexity and the overall model performance. Some fusion techniques could signi•cantly increase the
modelcomplexity,whichseriouslylimitstheir applicability for VQAmodels.Sofar,thereis noevidenceofhowe›cient
these multimodal bilinear pooling fusion techniques are for VQA systems dedicated to yes/no questions. Hence, a
comparative analysis is conducted between eight bilinear pooling fusion techniques, in terms of their ability to reduce
the modelcomplexityandimprovethemodelperformanceinthiscaseofVQAsystems.Experimentsindicatethatthese
multimodal bilinear pooling fusion techniques have improved the VQA model’s performance, until reaching the best
performance of —À.ò %. Further, experiments have proven that the number of answers in the developed VQA system
is a critical factor that ašects the ešectiveness of these multimodal bilinear pooling techniques in achieving their main
objective of reducing the model complexity. e Multimodal Local Perception Bilinear Pooling (MLPB) technique
has shown the best balance between the model complexity and its performance, for VQA systems designed to answer
yes/no questions.
scene and the question’s meaning and capturing the relevant correlations between both modalities to infer the
appropriate answer. In this paper, we propose a VQA system intended to answer yes/no questions about real-world
images, in Arabic. To support a robust VQA system, we work in two directions: (Ô) Using deep neural networks to
semantically represent the given imageandquestionina•ne-grainedmanner,namelyResNet-Ô òandGatedRecurrent
Units (GRU).(ò)Studyingtheroleoftheutilizedmultimodalbilinearpoolingfusiontechniqueinthetrade-ošbetween
the model complexity and the overall model performance. Some fusion techniques could signi•cantly increase the
modelcomplexity,whichseriouslylimitstheir applicability for VQAmodels.Sofar,thereis noevidenceofhowe›cient
these multimodal bilinear pooling fusion techniques are for VQA systems dedicated to yes/no questions. Hence, a
comparative analysis is conducted between eight bilinear pooling fusion techniques, in terms of their ability to reduce
the modelcomplexityandimprovethemodelperformanceinthiscaseofVQAsystems.Experimentsindicatethatthese
multimodal bilinear pooling fusion techniques have improved the VQA model’s performance, until reaching the best
performance of —À.ò %. Further, experiments have proven that the number of answers in the developed VQA system
is a critical factor that ašects the ešectiveness of these multimodal bilinear pooling techniques in achieving their main
objective of reducing the model complexity. e Multimodal Local Perception Bilinear Pooling (MLPB) technique
has shown the best balance between the model complexity and its performance, for VQA systems designed to answer
yes/no questions.
