記述言語：日本語   掲載種別：研究論文（学術雑誌）  

記述言語：日本語   掲載種別：研究論文（学術雑誌）  

記述言語：その他   掲載種別：研究論文（その他学術会議資料等）  

In this paper, we look at the case of a Generic text-to-text NMT model that has to deal with data coming from various modalities, like speech, images, or noisy text extracted from the web. We propose a two-step method, based on composable adapters, to deal with this problem of Multimodal Robustness. In the first step, we separately learn domain adapters and modality specific adapters, to deal with noisy input coming from various sources: ASR, OCR, or noisy text (UGC). In a second step, we combine these components at runtime via dynamic routing or, when the source of noise is unknown, via two new transfer learning mechanisms (Fast Fusion and Multi Fusion). We show that our method provides a flexible, state-of-the-art, architecture able to deal with noisy multimodal inputs.

記述言語：その他   掲載種別：研究論文（学術雑誌）  

We propose a multimodal neural machine translation (MNMT) method with semantic image regions called region-attentive multimodal neural machine translation (RA-NMT). Existing studies on MNMT have mainly focused on employing global visual features or equally sized grid local visual features extracted by convolutional neural networks (CNNs) to improve translation performance. However, they neglect the effect of semantic information captured inside the visual features. This study utilizes semantic image regions extracted by object detection for MNMT and integrates visual and textual features using two modality-dependent attention mechanisms. The proposed method was implemented and verified on two neural architectures of neural machine translation (NMT): recurrent neural network (RNN) and self-attention network (SAN). Experimental results on different language pairs of Multi30k dataset show that our proposed method improves over baselines and outperforms most of the state-of-the-art MNMT methods. Further analysis demonstrates that the proposed method can achieve better translation performance because of its better visual feature use.

DOI： 10.1016/j.neucom.2021.12.076

記述言語：その他   掲載種別：研究論文（学術雑誌）  

We propose word-region alignment-guided multimodal neural machine translation (MNMT), a novel model for MNMT that links the semantic correlation between textual and visual modalities using word-region alignment (WRA). Existing studies on MNMT have mainly focused on the effect of integrating visual and textual modalities. However, they do not leverage the semantic relevance between the two modalities. We advance the semantic correlation between textual and visual modalities in MNMT by incorporating WRA as a bridge. This proposal has been implemented on two mainstream architectures of neural machine translation (NMT): the recurrent neural network (RNN) and the transformer. Experiments on two public benchmarks, English-German and English-French translation tasks using the Multi30k dataset and English-Japanese translation tasks using the Flickr30kEnt-JP dataset prove that our model has a significant improvement with respect to the competitive baselines across different evaluation metrics and outperforms most of the existing MNMT models. For example, 1.0 BLEU scores are improved for the English-German task and 1.1 BLEU scores are improved for the English-French task on the Multi30k test2016 set; and 0.7 BLEU scores are improved for the English-Japanese task on the Flickr30kEnt-JP test set. Further analysis demonstrates that our model can achieve better translation performance by integrating WRA, leading to better visual information use.

DOI： 10.1109/TASLP.2021.3138719

開催年月日： 2023年9月

記述言語：その他   会議種別：口頭発表（一般）  

国名：その他  

開催年月日： 2023年7月

記述言語：その他   会議種別：口頭発表（一般）  

国名：その他  

開催年月日： 2023年6月

記述言語：その他   会議種別：口頭発表（招待・特別）  

国名：その他  

その他リンク： https://aamt.info/event/generalmeeting2023-seminar

開催年月日： 2022年12月

記述言語：英語  

国名：その他  

Multimodal Robustness for Neural Machine Translation

開催年月日： 2021年8月

記述言語：英語  

国名：その他  

TMEKU System for the WAT2021 Multimodal Translation Task System