Multimodal learning

Data usually comes with different modalities which carry different information. For example, it is very common to caption an image to convey the information not presented in the image itself. Similarly, sometimes it is more straightforward to use an image to describe information which may not be obvious from text. As a result, if different words appear in similar images, then these words likely describe the same thing. Conversely, if a word is used to describe seemingly dissimilar images, then these images may represent the same object. Thus, in cases dealing with multi-modal data, it is important to use a model which is able to jointly represent the information such that the model can capture the combined information from different modalities.

{{excerpt|Transformer (machine learning model)|Multimodality}}

{{excerpt|Large language model|Multimodality}}

A Boltzmann machine is a type of stochastic neural network invented by Geoffrey Hinton and Terry Sejnowski in 1985. Boltzmann machines can be seen as the stochastic, generative counterpart of Hopfield nets. They are named after the Boltzmann distribution in statistical mechanics. The units in Boltzmann machines are divided into two groups: visible units and hidden units. Each unit is like a neuron with a binary output that represents whether it is activated or not.{{Cite web |last=Dey |first=Victor |date=2021-09-03 |title=Beginners Guide to Boltzmann Machine |url=https://analyticsindiamag.com/beginners-guide-to-boltzmann-machines/ |access-date=2024-03-02 |website=Analytics India Magazine |language=en-US}} General Boltzmann machines allow connection between any units. However, learning is impractical using general Boltzmann Machines because the computational time is exponential to the size of the machine{{Citation needed|date=November 2022}}. A more efficient architecture is called restricted Boltzmann machine where connection is only allowed between hidden unit and visible unit, which is described in the next section.

Multimodal deep Boltzmann machines can process and learn from different types of information, such as images and text, simultaneously. This can notably be done by having a separate deep Boltzmann machine for each modality, for example one for images and one for text, joined at an additional top hidden layer.{{cite web |year=2014 |title=Multimodal Learning with Deep Boltzmann Machine |url=http://www.jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf |url-status=live |archive-url=https://web.archive.org/web/20150621055730/http://jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf |archive-date=2015-06-21 |access-date=2015-06-14}}

Multimodal machine learning has numerous applications across various domains:

• Cross-modal retrieval: cross-modal retrieval allows users to search for data across different modalities (e.g., retrieving images based on text descriptions), improving multimedia search engines and content recommendation systems. Models like CLIP facilitate efficient, accurate retrieval by embedding data in a shared space, demonstrating strong performance even in zero-shot settings.{{Cite arXiv |last1=Hendriksen |first1=Mariya |last2=Vakulenko |first2=Svitlana |last3=Kuiper |first3=Ernst |last4=de Rijke |first4=Maarten |date=2023 |title=Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study |class=cs.CV |eprint=2301.05174}}
• Classification and missing data retrieval: multimodal Deep Boltzmann Machines outperform traditional models like support vector machines and latent Dirichlet allocation in classification tasks and can predict missing data in multimodal datasets, such as images and text.
• Healthcare diagnostics: multimodal models integrate medical imaging, genomic data, and patient records to improve diagnostic accuracy and early disease detection, especially in cancer screening.{{cite news |last1=Quach |first1=Katyanna |title=Harvard boffins build multimodal AI system to predict cancer |url=https://www.theregister.com/2022/08/09/ai_cancer_multimodal/ |access-date=16 September 2022 |work=The Register |language=en |archive-date=20 September 2022 |archive-url=https://web.archive.org/web/20220920163859/https://www.theregister.com/2022/08/09/ai_cancer_multimodal/ |url-status=live }}{{cite journal |last1=Chen |first1=Richard J. |last2=Lu |first2=Ming Y. |last3=Williamson |first3=Drew F. K. |last4=Chen |first4=Tiffany Y. |last5=Lipkova |first5=Jana |last6=Noor |first6=Zahra |last7=Shaban |first7=Muhammad |last8=Shady |first8=Maha |last9=Williams |first9=Mane |last10=Joo |first10=Bumjin |last11=Mahmood |first11=Faisal |title=Pan-cancer integrative histology-genomic analysis via multimodal deep learning |journal=Cancer Cell |date=8 August 2022 |volume=40 |issue=8 |pages=865–878.e6 |doi=10.1016/j.ccell.2022.07.004 |pmid=35944502 |s2cid=251456162 |language=English |issn=1535-6108|doi-access=free |pmc=10397370 }}
• Teaching hospital press release: {{cite news |title=New AI technology integrates multiple data types to predict cancer outcomes |url=https://medicalxpress.com/news/2022-08-ai-technology-multiple-cancer-outcomes.html |access-date=18 September 2022 |work=Brigham and Women's Hospital via medicalxpress.com |language=en |archive-date=20 September 2022 |archive-url=https://web.archive.org/web/20220920172825/https://medicalxpress.com/news/2022-08-ai-technology-multiple-cancer-outcomes.html |url-status=live }}{{Cite arXiv |last1=Shi |first1=Yuge |last2=Siddharth |first2=N. |last3=Paige |first3=Brooks |last4=Torr |first4=Philip HS |year=2019 |title=Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models |eprint=1911.03393 |class=cs.LG}}
• Content generation: models like DALL·E generate images from textual descriptions, benefiting creative industries, while cross-modal retrieval enables dynamic multimedia searches.{{Cite arXiv |last1=Shi |first1=Yuge |last2=Siddharth |first2=N. |last3=Paige |first3=Brooks |last4=Torr |first4=Philip HS |date=2019 |title=Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models |class=cs.LG |eprint=1911.03393}}
• Robotics and human-computer interaction: multimodal learning improves interaction in robotics and AI by integrating sensory inputs like speech, vision, and touch, aiding autonomous systems and human-computer interaction.
• Emotion recognition: combining visual, audio, and text data, multimodal systems enhance sentiment analysis and emotion recognition, applied in customer service, social media, and marketing.

• Hopfield network
• Markov random field
• Markov chain Monte Carlo

{{reflist}}

Category:Artificial neural networks

Category:Multimodal interaction