Universal image representation based on a multimodal graph

Abstract

A system for classifying a target image with segments having attributes is provided. The system generates a graph for the target image that includes vertices representing segments of the image and edges representing relationships between the connected vertices. For each vertex, the system generates a subgraph that includes the vertex as a home vertex and neighboring vertices representing segments of the target image within a neighborhood of the segment represented by the home vertex. The system applies an autoencoder to each subgraph to generate latent variables to represent the subgraph. The system applies a machine learning algorithm to a feature vector comprising a universal image representation of the target image that is derived from the generated latent variables of the subgraphs to generate a classification for the target image.