Panofsky-Style Simple Tag Generation From Pure Visual Analysis
In this study, we explore the capabilities of state-of-the-art vision-language models in generating semantic descriptions for artworks. We employed five leading models - Gemini 2.0, GPT4.1, Pixtral Large, Sonnet 3.7, and Grok 2 - to produce taggings across three interpretative levels derived from Erwin Panofsky's iconological method: pre-iconographic, iconographic, and iconological.
Crucially, no metadata or prior information about the artworks was provided to the models. All taggings were generated through pure visual analysis. This constraint ensured that the evaluation focuses solely on the models' perceptual and interpretive abilities.
To assess the quality and depth of the generated taggings, we used the GPT-03 reasoning model as our evaluator. For each image and category, the evaluator assigned a score from 1 to 10 to each model's output, reflecting its semantic richness, accuracy, and interpretive value.
This page presents the results of this experiment, visualizing both the tags and the evaluation scores for each model, category, and image.
Tag Generation Prompt
You are an art historian and you have a task to create taggings for this painting. Generate as many taggings and keywords as possible that describe this painting. Categorize the tags using Panofsky's framework in three categories: pre-iconographic, iconographic, and iconological.
Model Evaluation Prompt
You are an art historian. Given the ${category} tags from different models, score each from 1-10 on accuracy and descriptiveness.
Judge model: GPT-03
The following chart shows the average evaluation score (1-10) for each model across all images and tagging categories.
Title: Verkündigung Mariens;Die Verkündigung an Maria
Artist: David, Gerard
Object number: 1095
Object type: Gemälde
Date: 1509
Title translation: The Annunciation
Title: Doppelbildnis des Justinian von Holzhausen (1502–1553) und seiner Frau Anna, geb. Fürstenberg (1510–1573);Doppelbildnis Justinians von Holzhausen und seiner Frau Anna, geb. Fürstenberg
Artist: Faber von Kreuznach, Conrad
Object number: 1729
Object type: Gemälde
Date: 1536
Title translation: Double Portrait of Justinian von Holzhausen and His Wife Anna, née Fürstenberg
Title: Löwenrudel in felsiger Landschaft;Landschaft mit Löwenfamilie
Artist: Roos, Johann Melchior
Object number: 356
Object type: Gemälde
Date: 1716
Title translation: Landscape with a Lion Family
Title: Boas übernimmt das Erbe Elimelechs
Artist: Victors, Jan
Object number: 581
Object type: Gemälde
Date: 1651-1653
Title translation: Boaz assumes the Legacy of Elimelech
Title: Blumenstillleben;Stillleben mit Blumenstrauss in einer Terrakottavase;Blumenstrauss in einer steinernen Vase und Früchte
Artist: Walscapelle, Jacob van
Object number: 617
Object type: Gemälde
Date: 1677
Title translation: Still Life with Bouquet of Flowers
Title: Ein Bauernhof
Artist: Morgenstern, Johann Ludwig Ernst
Object number: 655
Object type: Gemälde
Date: 1794
Title translation: A Farm
Title: Landschaft bei aufziehendem Gewitter;Landschaft bei abziehendem Gewitter
Artist: Ermels, Johann Franciscus
Object number: 663
Object type: Gemälde
Date: 1665-1670
Title translation: Landscape before a Thunderstorm;Landscape after a Thunderstorm
Title: Landschaft mit Hirtenfamilie und ihrer Herde;Vieh auf der Weide
Artist: Hirt, Friedrich Wilhelm
Object number: 682
Object type: Gemälde
Date: 1768
Title translation: Landscape with a Herdswoman and Farm Animals
Title: Bildnis eines Gelehrten
Artist: Massys, Quentin
Object number: 766
Object type: Gemälde
Date: 1525-1530
Title translation: Portrait of a Scholar
Title: Madonna mit Kind, Johannes dem Täufer und der heiligen Elisabeth
Artist: Bellini, Giovanni
Object number: 853
Object type: Gemälde
Date: 1490-1500
Title translation: Madonna and Child with Saints John the Baptist and Elizabeth