The increasingly large user base on social media means advertising on these sites is becoming a mainstream way for brand managers to engage and retain consumers (Alalwan et al., 2017). Brands use social media advertising to deliver product information, which can help consumers understand more about the product, improve their perception of the brand, and increase their willingness to choose the products of that brand (Kato, 2021). Luna-Nevarez and Torres (2015) demonstrated that when consumers find brand-related product information in social media advertisements useful, they hold a positive attitude toward the advertisement, which promotes interaction with the brand.

Content of advertising on social media can be presented in a variety of forms, including text, images, and videos (Hardey, 2011). The visual effect of images is used in social media advertising to arouse consumers’ interest by showing external information about the product. Consumers’ engagement behavior with advertisements is significantly influenced by their initial visual perception of product images (Vetter & Newen, 2014). Vision is the individual’s first point of contact with external stimuli and their psychological response depends on the visual experience of the stimulus (Recarte & Nunes, 2003). As an external environmental stimulus, product images can attract a consumer’s subsequent engagement only once the person’s attention has been drawn to them.

The text content of an advertisement is a summary of the intrinsic properties of the product, presented to consumers in a condensed manner to strengthen their knowledge about the essence of the product. Antoniadis et al. (2019) proposed that posts on social media with visual elements such as images and videos get more consumer engagement than those with only plain text content. The combination of images and text makes the advertising effect more vivid, increasing the time consumers spend on the brand page and strengthening their understanding of the product (C. Li et al., 2021). Y. Li and Xie (2019) studied the impact of image attributes in user-generated content on consumer engagement on social media platforms, and showed that professionally shot images and high image quality resulted in better consumer engagement. However, the impact of the matching of images and descriptive product information text in an advertisement on consumer engagement has not been fully explored in the context of social media advertising.

To fill this research gap we explored image–text matching, which is defined as the impact of matching the external display content of advertising images with the internal core expression of advertising text, on consumer engagement. To measure consumer–brand interaction behavior based on the number of likes, comments, and shares posted on a social media site, we used machine-learning algorithms to identify image characteristics (image color, image texture, image shape) and textual theme features of brands’ social media advertisement content. This allowed us to quantify product appearance and intrinsic attribute information and then construct a consumer engagement model. Our aims were to help potential consumers better understand the product information presented, improve their perceived trust in the brand, and stimulate interactive behavior.

Consumer Engagement in Social Media

Brand social media advertising refers to product advertising that brand managers post on social media platforms. The aim of consumer engagement is to identify and analyze the behaviors and emotions that consumers generate when they interact with brands. According to Dessart et al. (2015), consumer engagement can be subdivided into cognitive engagement (consumers receive the stimulus of the advertisement and become interested in the content), emotional engagement (the advertisement stimulates emotional and rational thinking), and behavioral engagement (the advertisement triggers actual consumer behavior, e.g., spreading the brand advertisement). Brands can connect with consumers through social media. Since consumer engagement on social media cannot be observed directly, it needs to be measured indirectly (van Doorn et al., 2010). Khobzi et al. (2018) stated that brand interaction behavior can be used to measure consumer engagement, including sharing, liking, and commenting on posts. There are many related studies in which consumer engagement has been measured using these points of interaction (see, e.g., Bai et al., 2020; Shin & Perdue, 2022).

Construal Level Theory

Construal level theory holds that consumers’ attitude toward products depends on their psychological representations of different construal levels for the products, and that the interaction between construal level and psychological distance will change the corresponding perceived value (Trope & Liberman, 2010). Factors such as the virtual nature of social media and information overload of brand advertising can affect consumers’ perception of value (Khan, 2022). As the primary stage of cognitive activity, the image–text matching of brand product information presentation can attract consumers’ attention. When the initial presentation of the product based on the image matches the text-based internal description of the product, the psychological distance between the advertisement and the consumer is narrowed, thus generating a better value judgment and a stronger likelihood of engaging with the product than when there is a mismatch (Soderberg et al., 2015).

Cue Consistency Theory

Cue consistency theory proposes that when multiple types of information or cues for content are consistent, their effects on consumer attitudes can be linearly integrated, meaning that the value of the cue can influence attitude and behavior (Kang & Jung, 2015). Miyazaki et al. (2005) found that when two cues have the same influence on a consumer’s attitude, perception, and emotions, there is a synergy between them, and the effect of any one cue is enhanced by the presence of the other.

Image–Text Match

In the social media environment, advertising content regarding product information is mainly divided into image information and text information (Escalas, 2004). Product images significantly influence consumers’ engagement as they view these images before accessing additional textual product-related information. Visually more prominent product descriptions of external information reduce consumer browsing time and effort (Geise & Baden, 2014).

Consumers rely on image information to understand the product and increase their knowledge about it through the written description of its functions. Text information then forms the main body of communication, conveying specific internal product attributes to potential consumers, thus having a significant impact on consumers’ brand advertising emotions and attitudes (Mulcahy & Gouldthorp, 2016).

The difference between the external product information conveyed in images and the internal product attributes of text descriptions leads to different degrees of consumer cognition. Mayer and Chandler (2001) suggested there will be a consistency effect, that is, if the content of the message is consistent in the image and the text, this will help consumers to integrate the message and to show a positive intention to buy the brand product. Hence, we proposed that the matching of images and text would have a positive impact on consumers’ engagement with brand/product advertising on social media.

Research Framework

We used machine-learning algorithms to identify image characteristics and text characteristics in advertising content, and quantify product appearance information and intrinsic attribute descriptions. Thus, we constructed a consumer engagement model, as shown in Figure 1.

Method

Sample and Data Collection

Among the many social media sites in China, Sina Weibo has the largest user base. In 2021 it was forecast that Weibo active users would reach 530 million per month (Wang et al., 2020). Brand managers can influence their target consumers through Weibo, where consumers can provide feedback and help spread their influence through posting their likes, shares, and comments. Therefore, we conducted our research on Sina Weibo and selected cosmetic brands as our study products, considering that social media sites have become a major channel for cosmetics consumers to seek information and share their experiences, perceptions, and preferences widely (Gans et al., 2021). Among the cosmetic brands, Estée Lauder has a large consumer base that covers the age range of people who use social media, making it ideal for studying the impact of social media advertisement content on consumer engagement.

We used Python programming language to capture the image and text content of advertisements posted by Estée Lauder on Sina Weibo between January 2020 and January 2021, along with the number of likes, comments, and shares received by each advertisement. The crawled information dimensions are shown in Figure 2. We manually filtered out advertisements that did not contain product information, retaining only those that featured matched image and text content. The final dataset consisted of 322 advertising posts related to brand products, which included 322 blocks of text and 715 images, forming a self-built mini corpus.

Procedure

To measure the effect of matching content, we extracted characteristics for both images and text. Characteristics extraction is a quantification of the description of product information in images and text, effectively transforming visual and textual content into a vector or matrix representation.

Image Characteristics Extraction

We included 715 images in the sample data and used machine-learning algorithms to write programs to extract color, texture, and shape visual characteristics from the images (see Figure 3). Single-image base characteristics sometimes do not reflect the true face of the image and focus on one kind of image information, so that other information may be ignored. Therefore, through the complementary effect of information from the three dimensions of color, texture, and shape, the product information contained in the image can be fully displayed and attract the attention of target consumers to the greatest extent.

Pretreatment was first performed on the image with the aim of eliminating irrelevant information. The process included size standardization, normalization, and enhancement. Due to inconsistencies in the size of the acquired sample image data, it was necessary to standardize the image size using a manual segmentation method for better extraction of characteristics. Normalization is a standard form of image processing that makes certain characteristics of an image invariant under a given transformation, thus ensuring controlled illumination. Enhancement is the selective enhancement and suppression of information in an image, transforming the image into a form more suitable for machine processing.

Color characteristics represent the visual color variations in an image. The color histogram depicts the proportion of different colors in the image, independent of image rotation and translation changes, and is suitable for processing images that are difficult to segment automatically and images for which it is not necessary to consider the spatial location of objects. The steps to extract the color histogram were as follows: First, the appropriate color space was selected and the image was converted to that color space. Second, the components of the color space were quantified. Then, the number of occurrences of different pixels in each color component was counted and the proportion of different pixels in the whole image was calculated. Finally, the color components were synthesized into a whole vector, which constitutes the color characteristics vector of an image.

Texture characteristics show the structure of the internal texture of the image. Local binary pattern (LBP) is an algorithm used to describe image texture characteristics, and it has been applied by more researchers in the field of image research in recent years because of its advantages, such as rotation invariance and grayscale invariance (Doshi et al., 2016). LBP is used to encode a single pixel as the center, after which the radius (r) and eight pixel points are selected on the circumference, then the pixel value between the selected point and the center point is compared. If the value at the selected point is greater than that at the center point, it is recorded as 1; if it is smaller, it is recorded as 0. LBP encoding is obtained by stringing numbers into a binary chain in a clockwise direction.

Shape characteristics identify image edge or contour information and separate the image subject from the image background. The histogram of oriented gradients (HOG) algorithm represents the structural characteristics of the edges and therefore describes the local shape information. The scientific modeling of the HOG algorithm is supported by the relevant mathematical foundations and optimization theory (Zhong et al., 2021). The HOG characteristics extraction steps were as follows: First, the gradient was calculated. Second, the HOG direction of individual cells was completed, and the characteristics extraction of the preprocessed commodity images was performed with different sizes of cells. Finally, the histograms of each region were combined into one large histogram to represent the characteristics of the whole image.

The product information contained in the image is rich and diverse, and the external content is often not adequately described by only a single characteristic. Therefore, the characteristics of the three dimensions of color, texture, and shape were fused using the early fusion method, which is a forward fusion approach to image description that involves cascading multiple characteristic representations. Different characteristics are utilized to achieve complementarity and fully visualize the product information contained in the images (Liu et al., 2014).

The final extracted image characteristics in this study included color histogram 1 × 256 dimensions, HOG 1 × 144 dimensions, LBP 1 × 256 dimensions, and fusion to obtain a 656-dimensional characteristic matrix. In this way, the content of visual elements was used to show consumers the products’ external information, so that they had a preliminary perception of the product.

Text Characteristics Extraction

Brand advertising on social media is often presented in a combination of one block of text and multiple images. Thus, the number of blocks of text corresponding to the 715 images in the sample data of this study was 322. Consumers rely on image information to understand and increase their knowledge of the product through the description of its functions in the text of the advertisement. Social media advertising text is the main means for conveying to potential consumers the specific product details (Bell et al., 2017), which have a significant effect on consumers’ brand advertising emotions and attitudes, thus influencing their behaviors in regard to interacting with the advertising. In this study we wrote a program to quantify the intrinsic information contained in the textual descriptions of products in brand advertisements through machine-learning algorithms (see Figure 4).

Latent dirichlet allocation (LDA) was proposed by W. Li et al. (2012) to infer the topic distribution of text. This allocation provides the topic of each block of text in the form of the probability distribution so that after extracting the topic distribution by the algorithm, topic clustering or classification can be performed. LDA reflects the topic by the probability distribution of the word. For the corpus of brand-product-related text, the textual topic characteristics are extracted according to the principle of the LDA model.

During LDA, the text was first preprocessed through data cleansing, which entails normalization transformations of nonnormal objects, such as removing spaces from the text and removing punctuation marks. The Jieba system was used as a word segmentation tool to process the cleaned data. Stop words are function words that do not reflect the theme in the text. This paper selected a stop word list in the field of social media brands and performed stop word processing on the word segmentation results to remove irrelevant words from the text. Next, we used the function to filter and process the topic-independent noise data in the text, and filter data such as English words.

Measures

On the basis of the characteristic matrices of product descriptions in the images and text, we calculated the Euclidean distance between the two matrices to quantify how well the sample data in the different modalities of images and text matched in conveying to consumers the interaction of product appearance and inner information.

As the dimensionality of the characteristic matrices obtained from image extraction and from text extraction is different, to be able to perform similarity calculations, a matrix needs to be trained by writing a Python program so that the image and text characteristics mapped into this matrix achieve the same dimensionality. The Euclidean metric is the most common distance metric used to measure the absolute distance between two points in a multidimensional space (Cardarilli et al., 2020). Euclidean distances represent the true distance between two points in n-dimensional space (see Equation 1).

The matching degree value of a block of text and an image is obtained through the above calculation. Because the degree of match between several images and the block of text in each advertisement varies, and considering that only a set of likes, comments, and shares exist in an advertisement to measure consumer engagement, the numbers of inputs and outputs need to be consistent when constructing the model. We used the mean value method to calculate degree of match, so that this could be unified. We used SPSS 23.0 to perform a regression analysis of each variable to explore whether there was a causal relationship between image–text matching of brand advertising on social media and consumer engagement in regard to brand interactions.

Results

Following the model described above, we used the advertisement image–text match value as the independent variable and consumer engagement (i.e., number of likes, comments, and shares) as the dependent variable, and performed a linear regression analysis of data from 322 samples. The results are shown in Table 1.

Discussion

Brands use social media to expand the activity and influence of their products and enhance the appeal of their advertisements to potential consumers by adapting the content. It is necessary to understand how the effect of matching images and text in an advertisement containing a product description affects social media users’ interaction with the brand. This study explored the impact of integrated brand advertising content on consumer engagement from the perspective of image–text matching. The results show that consumers engaged more actively when there was a match between images and text product descriptions in a brand’s social media advertising.

When the visual content presented in an advertisement image at a low construal level matches the detailed information in the text at a high construal level, although the level of construal of images and text is different, matching shortens the psychological distance, thereby promoting consumer engagement (Kim & Slocum, 2008). This creates cognitive coordination among consumers, which, in turn, promotes their engagement. This is in line with cue consistency theory and with the finding by Miyazaki et al. (2005) that there is synergy between high prices and other positive cues when they are congruent, with the effect of either cue being enhanced by the presence of the other. Our findings also align with those of Van Rompay et al. (2010), who found that the consistency of the image atmosphere (comfortable, neutral, modern) presented on an online booking site with that of the textual description (comfortable, modern) had a positive impact on users’ processing fluency, which, in turn, made their brand attitudes more favorable.

This study provides meaningful theoretical references for academic researchers. First, this research is an important attempt to extend the effect of matching images and text in an advertisement to the field of social media advertising. Most of the existing studies on the impact of social media advertising content on consumer engagement have been focused on the role of text or of images, which has limitations. By using construal level theory and cue consistency theory to explore the theoretical mechanism of the impact of image and text matching in social media advertisements on consumer engagement, we have expanded the application scope of the theories. Second, scholars have analyzed the effect of text or of images in attracting consumers to interact with brands, a research angle that is rich but single-stranded. Therefore, the correlation between textual content and visual elements has been ignored and research on the factors influencing consumer engagement in the context of social media is not well developed. This study extends the literature on the drivers of consumer brand engagement.

Our findings also provide practical guidance for social media managers. First, the study highlights the importance of brand advertising that combines images and text. We suggest that brand managers pay attention to matching the appearance of the product in the image and the specific description of the product in the text when publishing advertisements. Second, we diverged from the tools previously used for measuring image–text matching and proposed the use of machine-learning algorithms to efficiently and scientifically extract the content characteristics of images and text. The degree of matching between images and text was measured by calculating the distance metric.

This study has the following limitations: First, we analyzed advertisements in the form of still images and text posted on social media. Future research could be extended to include videos and other forms of advertisements. Second, the study’s conclusion that image–text matching is effective in promoting consumer engagement has been verified only for one cosmetics brand on the Sina Weibo platform. As there are differences in the content characteristics of brands in different industries, subsequent studies could explore whether the same conclusion can be reached for brands in categories other than cosmetics. Third, this study measured image–text matching based on the degree of similarity (Euclidean distance) between images and text descriptions of product information. Future researchers could apply distance formulations other than Euclidean distance to assess the characteristics matrix. Finally, the sample data collected were confined to advertisements for the Estée Lauder brand posted within 1 year, which will have had an impact on consumer engagement. Subsequent studies could expand the range of years for brand advertising.

This study was supported by the Research and Planning Project of Philosophy and Social Sciences in Heilongjiang Province (20GLB115).

Hongqi Xia, School of Computer and Information Engineering, Harbin University of Commerce, No. 1 Xue Hai Street, Song Bei District, Harbin, People’s Republic of China 150000. Email: [email protected]