Characterization and classification of semantic image-text relations

Download statistics - Document (COUNTER):

Otto, C.; Springstein, M.; Anand, A.; Ewerth, R.: Characterization and classification of semantic image-text relations. In: International Journal of Multimedia Information Retrieval 9 (2020), S. 31-45. DOI:

Repository version

To cite the version in the repository, please use this identifier:

Selected time period:


Sum total of downloads: 158

The beneficial, complementary nature of visual and textual information to convey information is widely known, for example, in entertainment, news, advertisements, science, or education. While the complex interplay of image and text to form semantic meaning has been thoroughly studied in linguistics and communication sciences for several decades, computer vision and multimedia research remained on the surface of the problem more or less. An exception is previous work that introduced the two metrics Cross-Modal Mutual Information and Semantic Correlation in order to model complex image-text relations. In this paper, we motivate the necessity of an additional metric called Status in order to cover complex image-text relations more completely. This set of metrics enables us to derive a novel categorization of eight semantic image-text classes based on three dimensions. In addition, we demonstrate how to automatically gather and augment a dataset for these classes from the Web. Further, we present a deep learning system to automatically predict either of the three metrics, as well as a system to directly predict the eight image-text classes. Experimental results show the feasibility of the approach, whereby the predict-all approach outperforms the cascaded approach of the metric classifiers. © 2020, The Author(s).
License of this version: CC BY 4.0 Unported
Document Type: Article
Publishing status: publishedVersion
Issue Date: 2020
Appears in Collections:Forschungszentren

distribution of downloads over the selected time period:

downloads by country:

pos. country downloads
total perc.
1 image of flag of Germany Germany 52 32.91%
2 image of flag of United States United States 42 26.58%
3 image of flag of China China 7 4.43%
4 image of flag of Vietnam Vietnam 6 3.80%
5 image of flag of Europe Europe 6 3.80%
6 image of flag of United Kingdom United Kingdom 5 3.16%
7 image of flag of Serbia Serbia 4 2.53%
8 image of flag of Morocco Morocco 4 2.53%
9 image of flag of Jordan Jordan 4 2.53%
10 image of flag of Pakistan Pakistan 3 1.90%
    other countries 25 15.82%

Further download figures and rankings:


Zur Erhebung der Downloadstatistiken kommen entsprechend dem „COUNTER Code of Practice for e-Resources“ international anerkannte Regeln und Normen zur Anwendung. COUNTER ist eine internationale Non-Profit-Organisation, in der Bibliotheksverbände, Datenbankanbieter und Verlage gemeinsam an Standards zur Erhebung, Speicherung und Verarbeitung von Nutzungsdaten elektronischer Ressourcen arbeiten, welche so Objektivität und Vergleichbarkeit gewährleisten sollen. Es werden hierbei ausschließlich Zugriffe auf die entsprechenden Volltexte ausgewertet, keine Aufrufe der Website an sich.

Search the repository