Motif Mining: Finding and Summarizing Remixed Image Content

On the Internet, images are no longer static; they have become dynamic content. Thanks to the availability of smartphones with cameras and easy-to-use editing software, images can be remixed (i.e., redacted, edited, and re-combined with other content) on-the-fly, allowing a world-wide audience to repeat the process many times. From digital art to memes, the evolution of images through time is now an important topic of study for digital humanists, social scientists, and media forensics specialists. However, because typical data sets in computer vision are composed of static content, there has been limited development of automated algorithms for analyzing remixed content. In this paper, we propose the idea of Motif Mining: the process of finding and summarizing remixed image content in large collections of unlabeled and unsorted data. For the first time, this idea is formalized and a reference implementation grounded in that formalism is introduced. We conduct experiments on three meme-style data sets, including a newly collected set associated with the Russo-Ukrainian conflict. The proposed motif mining approach is able to identify related remixed content that, when compared to similar approaches, more closely aligns with the preferences and expectations of human observers.