Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities
As the amount of available data increases, the human ability to locate, process, and analyze it is strained and eventually overwhelmed. To address this challenge for nonproliferation analysts, the authors have been creating a large-scale multimodal retrieval system to help analysts triage and search open source science, technology, and news data for indicators of nuclear proliferation capabilities and activities. The system relies on a set of deep neural networks (DNNs) trained to evaluate conceptual similarities across data modalities, such as text, image, and video. These DNNs can be used to search and prioritize data, according to a nuclear fuel cycle (NFC) process template, that are conceptually closest to the seed query items regardless of data modality. The DNNs have been trained to map conceptually related words, sentences, and images to nearby points in a multimodal feature space, enabling intra- and intermodal retrieval via nearest-neighbor calculations to seed query points. The authors evaluate the system’s ability to retrieve NFC-related data that have been purposely hidden in collections of unrelated background data. They demonstrate quantitative and qualitative results for text-to-image, image-to-image, and image-to-video retrievals. This paper discusses data challenges confronting nonproliferation analysts, describes the DNNs designed to retrieve multimodal data that are proximal in the semantic space, and demonstrates their effectiveness by applying the DNNs to nonproliferation-specific multimodal data sets.