Review and Prospect of Multimodal Metaphor Research/YANG Youwen

 Studies in Literature and Language

Vol. 23, No. 2, 2021, pp. 1-6

DOI:10.3968/12202

 

Review and Prospect of Multimodal Metaphor Research

                                                                                  YANG Youwen

(Vui lòng nếu có trích dẫn xin tôn trọng tinh thần học thuật; tác giả cùng cơ sở xuất bản bằng cách ghi rõ tên tác giả, năm xuất bản, cơ sở,/ NXB/ Tạp chí)

(Kim Rin sưu tầm và giới thiệu bằng file word. Nếu bạn cần bản tiếng Việt hãy search từ khóa bằng tiếng Việt trên trang blog của mình, mục ngôn ngữ học hoặc bạn cũng có thể email hay để lại bình luận cho mình)

Abstract: Multimodal metaphor, pushing the study of metaphors in the language level into other fields and pouring more cultural implications for it, has become one of the hot spots in current linguistic research. This study reviews the origin, current situation, the approach of multimodal metaphor and talks about its research prospects, which is conducive to providing thinking inspiration and paradigm reference for following researches.

Key words: Multimodal metaphor; Research review; Research prospect

1. THE ORIGIN OF MULTIMODAL METAPHOR RESEARCH

The concept metaphor proposed by Lakoff & Johnson (1980) leads to the cognitive turn of metaphor research. They think that metaphor is a kind of rhetoric or speech expression means on the surface. But in essence, metaphor is “understanding and experiencing another different kind of things through something”, which is an external embodiment of the basic thinking mode of human beings, and the most basic and reliable means for human beings to leap from the nature to the necessity. At the same time, it is also the inevitable stage for human beings to go from known to unknown, from concrete to abstract, from apperance to regulation. It can be concluded that conceptual metaphor breaks through the barriers of rhetoric, which is widely existed in people’s language and behavior. It is the most important way of thinking of human beings in essence, which firmly maintains the almost all connections between human and the external world, and becomes a powerful tool for our thinking extension, and even becomes the heavy foundation and unremitting power of the development of human social civilization. After continuous exploration and discussion by many scholars for more than 40 years, the research on conceptual metaphor has made great progress in all aspects. Among them, the “multimodal metaphor” advocated by Forceville (2009) has pushed the conceptual metaphor research confined to the language level to a broader multilingual category and integrated into amore cultural connotations. In the book Pictorial Metaphor in Advertising, Forceville (1996) combines cognitive metaphor theory with advertising design, starts with interaction theory and combines the main viewpoints of relevance theory, analyzes the pictorial metaphor phenomenon in graphic advertising in detail, and puts forward four basic types of pictorial metaphor from macro perspective. Since then, Forceville (2009) made a great contribution to multimodal metaphor. Multimodal Metaphor claims that multimodal metaphor refers to the metaphors which are represented by different modes (pictorial symbols, written symbols, verbal symbols, gestures, sounds, music, smells, tastes and contacts, etc), which are widely used in advertisements, comics, posters, gestures, music, sculpture, architecture, dance, film, television and other fields covering almost all the areas of human social and cultural life. Although multimodal metaphor originated from the conceptual metaphor mentioned above, there are many differences in their characteristics. Multimodal metaphor has the characteristics of dynamic, narrative, energetic, concrete, metonymy as the premise and cultural experience driven understanding, etc.

 

 

2. CURRENT SITUATION OF MULTIMODAL METAPHOR RESEARCH

The essence of metaphor is to understand another concept domain (target domain) by means of one concept domain (source domain). With the deepening of the research, the scholars have reached a consensus on this argument. The thinking and conceptual nature of metaphor determine that the study of metaphor should be extended to a wider field. The multimodal metaphor advocated by Forceville (2009) largely makes up for the defect of metaphor research. It can be concluded that multimodal metaphor is the necessary stage for the development of conceptual metaphor research and the inevitable trend of metaphor research. It is necessary to review and sort out the current situation of multimodal research abroad and domestic systematically.

2.1 The Current Situation of Multimodal Metaphor Research Abroad

The research of multimodal metaphor originated from the scholars’ discussion of visual metaphor and pictorial metaphor, which laid a solid foundation for the concept and feature description of multimodal metaphor. Advertising, cartoon and body language are three categories of multimodal metaphor studies abroad, which have been focused earlier and more widely. Many scholars are aiming at advertising (Forceville, 2007; Koller， 2009； Caballero，2009； Yu，2009； Urios-Aparisi， 2009； Lantown & bobrova, 2012), comics (EL refaie, 2003; Yus，2009； Mar í a Enriqueta Cort é s de los R í os, 2010；Boungeru, L. & C. Forceville, 2011) and body language (Langacker, 2008; Cornelissen & Clarke,2012； in cienki, 2016). The multimodal metaphor has carried out many related studies, such as operation mechanism, construal path, metaphor and metonymy interaction, distinctive characteristics, meaning communication mode and mapping expression from source to target domain. Their research mainly attempts to construct a theoretical system of multimodal metaphor by describing and interpreting a small amount of corpus in different categories of languages, and at the same time, it compares and verifies the relevant theoretical views of conceptual metaphor drawn from text level. In addition to the above-mentioned static multimodal metaphor, foreign scholars have paid attention to dynamic multimodal metaphor and made some progress. Fahlenbrach (2016) firstly unified the categories of movies, TV and video games into dynamic video categories in the book Embedded Metaphones in Film, Television, and Video Games. The division is based on the distinctive dynamic characteristics shared by the three items, namely, they are all through the continuous image, sound and text modes to complete the metaphor construction and information transmission. Besides the classification of dynamic and static metaphors, this book also analyzes and explains the narrative structure, sense perception, emotional expression, mental representation and metaphor-mytonymy interaction in dynamic metaphors. In addition, the author argues that video games have more active sensory movement than others in dynamic images, and provide a larger display space for experiencing image schema, which should be paid more attention by scholars. Overall, the book has divided the categories involved in multimodal metaphor research and expanded its research scope to a certain extent, which is very helpful to clarify the research orientation of multimodal metaphor at macro level and clarify the research route for the subsequent research.

2.2 The Current Situation of Multimodal Metaphor Research in China

In recent years, the research of multimodal metaphor has also flourished in China. The related research are mainly focused on the following three aspects: introduction and review of multimodal metaphor (Zhaoxiufeng, 2011), theoretical basis study of multimodal metaphor (Fengdezheng, 2011), a multimodal metaphor representation study on a small-scale corpus (Yang Youwen, 2015), Case study of multimodal metaphors in different fields( Lanchun and Cai Ying, 2013).It should be noted that Yu Yanming (2013) firstly put forward with the mode “implication” into the representation system of multimodal metaphor, enriching the representation types of multimodal metaphor, and its division criteria are more rigorous and convincing, thus greatly expanding the research dimension of multimodal metaphor. It is a basic preparation for the following series of research on characterization methods. In addition, domestic scholars also try to introduce multimodal metaphor theory and methods into foreign language teaching practice. These studies are both pioneering and innovative, and are becoming an indispensable part of multimodal metaphor research. Li Yi and Shi Lei (2010) argues that multimodal metaphor teaching research can further develop cognitive linguistics theory, and help people better study various factors in the teaching process, and then feed back the results of the study to the new teaching practice, and further improve the teaching effect. Liang Xiaohui (2013) believes that multimodal metaphor will become the latest guiding method in English writing teaching. Taking comparative articles as an example, he elaborates the enlightenment and promotion of multimodal metaphor on English writing teaching from three aspects: the connotation difference, the usage criteria and the application scope of the two kinds of articless. Through a teaching experiment, Wu Jingjing (2017) applied multimodal metaphor to the teaching of College English writing. The research finds show that the introduction of multimodal metaphor can promote subjects’ of metaphor ability and further improve the writing ability. This study can provide a new method and path for the teaching and research of College English writing. Compared with conceptual metaphor, the application of multimodal metaphor in foreign language teaching is just beginning. At present, it mainly discusses the approaches and methods of multimodal metaphor in foreign language teaching. The main content is focused on the teaching of foreign language writing. The subsequent large-scale empirical research needs to be carried out in oral, listening and reading and other teaching practices. Of course, the mature multimodal metaphor theory framework is the prerequisite for its successful development of application research. In turn, relevant application research can test the distinctive features of multimodal metaphor in a broader space and realize its multi-function.

3. APPROACH TO MULTIMODAL METAPHOR RESEARCH

3.1 Scope of Study

Metaphor is not only the basic attribute and the form of language, but also the embodiment of the essential characteristics of human population, and is an important tool and solid foundation of human culture. The multimodal metaphor derived from metaphor is permeated in various cultural forms of human society, such as language, art, religion, myth, science, etc., and is an important tool to reveal the occurrence, development, nature and function of different cultural behaviors and cultural forms. Based on the thinking attribute and concept essence of conceptual metaphor, we can think that multimodal metaphor exists almost in every corner of human social and cultural life, such as advertisement, painting, music, dance, architecture, cartoon, poster gesture, sculpture, film, television, etc., so its research scope is also very broad. Forceville (2009) proposed the concept, distinguishing features, research methods and reference paradigm of multimodal metaphor, which greatly expanded the depth and scope of its research from the technical perspective, and provided useful thinking reference and reference for its practical application in multilingual category

3.2 Research Methods

The extensive research category, rich representation mode and representation mode of multimodal metaphor determines that the research methods are also becoming more diversified. First, the method of combining description with interpretation. Shu dingfang (2003) argues that “Modern metaphor studies should have two goals: one is the description of metaphor phenomenon, the other is the explanation of this phenomenon”. Description is the precondition of interpretation, and interpretation is the ultimate purpose of metaphor research. The research of multimodal metaphor should describe its conceptual model and representation types comprehensively and systematically, and analyze the multi-dimensional and wide category from semantic and pragmatic aspects, and then make a deep theoretical explanation of the metaphors based on description. Second, Deignan (2004) believes that the quantitative analysis method using corpus in metaphor research can reduce the subjectivity and onesideness of the study, and enhance the objectivity and comprehensiveness. Besides qualitative analysis of multimodal metaphors, relevant researches can also be empirically guided by corpus software and combing statistical methods to search and identify specific corpus exhaustively or pertinently, and summarize the corresponding conceptual models and multimodal metaphor presentation types from the bottom up. Third, the comparative analysis method. Language is the external expression of value system, thinking mode, concept and meaning system of different nationalities, and the cognitive metaphor at the conceptual level is the same. Shu dingfang(2000) claims that The real contribution of Chinese scholars to metaphor research should provide a more powerful and unique basis for the establishment of metaphor theory through comparative study of Chinese metaphor and other language metaphors. The construction and recognition of multimodal metaphors in different cultural backgrounds should be different from each other. We can make a comparative analysis of the results from the cognitive perspective, and then make a deep explanation by comparing the different language culture, thinking framework and philosophy basis.

 

3.3 Research Tasks

Multimodal metaphor is not only an inevitable product of the development of conceptual metaphor, but also a new research paradigm which emerged under the background of the trend of thought of multimodal in linguistic academic field. It has changed from the single linguistic and literal representation of conceptual metaphor to multimodal representation in many linguistic fields. It turns From the static cognitive function of conceptual metaphor to the construction of dynamic narration and meaning structure. This new paradigm, based on the integration of metaphor concept and multimodal discourse, marks a new stage in cognitive metaphor research, and opens up a new space for its further development. From the macro perspective, there are two main tasks in the research of multimodal metaphor. Firstly, the theoretical achievements of conceptual metaphor based on language field are extended to other fields in order to be tested, supplemented and perfected, so as to provide unremitting force for the sustainable development of cognitive metaphor research. Secondly, it provides theoretical guidance and creative reference for the practical innovation of multilingual category (advertising, cartoon, poster, photography, film, TV, music, dance, architecture, sculpture and gesture, etc.), thus building a bridge for the theoretical achievement of multimodal metaphor to practical application. From the micro perspective, there are two main tasks in multimodal metaphor research. The first one is to construct a theoretical system which conforms to the characteristics of multimodal metaphor from multiple perspectives, and lay a solid foundation and broad way for its further research. The second one is to divide the external representation of multimodal metaphor, thus increasing its research dimension, highlighting its powerful cognitive innovation function and dynamic narrative function.

 

4. THE RESEARCH T REND OF MULTIMODAL METAPHOR

Multimodal metaphor research is an inevitable extension of the development of conceptual metaphor research. Its broad research space and diverse representation types have attracted the attention of many scholars. On the whole, the study of multimodal metaphor is still based on the conceptual metaphor theory of cognitive linguistics. Introspection and speculation, description and explanation are its main research methods, and multi-dimensional analysis of cases in different categories is its main research focus. Looking forward to its research trend on the basis of previous research will help to grasp its latest research trends, accelerate its interdisciplinary integration, improve its theoretical system construction, and finally create a broader development space in various disciplines involved, and get more effective practical application at the same time. Looking back on the research achievements of scholars at home and abroad, there are four research trends of multimodal metaphor: theoretical innovation, interdisciplinary development, conjoint analysis and empirical research. 4.1 Trend of Theoretical Innovation Forceville (2009) believes that multimodal metaphor is developed from the study of conceptual metaphor and they should be closely related to each other. However, the theoretical system of conceptual metaphor can not fully fit the distinctive characteristics and interdisciplinary attributes of multimodal metaphor, so it can not fully meet the in-depth development of its research. As a matter of fact, from the register of the two studies, although it is claimed that the study of conceptual metaphor should not be limited to the language level, in practice, the vast majority of the study of conceptual metaphor still takes language (written or oral) as its main corpus selection scope. The single source of corpus leads to its research process falling into a cycle of self-proof, and whether its research results are objective and universal has also been widely questioned. On the contrary, as mentioned above, multimodal metaphor extends the selection of its research corpus to more other categories, and really breaks away from the confinement of language itself. A large number of multimodal related researches have been carried out. The research process can support, confirm and supplement each other, and the research results are attached with higher acceptability. Therefore, the close relationship between conceptual metaphor and multimodal metaphor does not mean that multimodal metaphor is always in a passive position, nor does it mean that multimodal metaphor can only be a simple connection and extension of conceptual metaphor. On the contrary, it should be its superior concept, that is, there may be a relationship of covering and being covered between multimodal metaphor and conceptual metaphor. Since the theoretical system of conceptual metaphor can not meet and support the research of multimodal metaphor, it is urgent to carry out the theoretical innovation research that conforms to the characteristics of multimodal metaphor. The extension of multimodal metaphor theory should not only analyze and refine the theoretical system of conceptual metaphor, but also take into account its distinctive features. It is also necessary to learn from the research theories and practices of different disciplines, rather than being confined to linguistic discipline. In a word, the absence of the theoretical system greatly restricts the research approaches and patterns of multimodal metaphor, and also makes the related research lack of theoretical support and method guidance, and furthermore, the final research achievements are not so persuasive. Therefore, it is imperative to construct a theoretical system suitable for the characteristics of multimodal metaphor at a higher conceptual level, and it is also an inevitable requirement for its related research to be carried out in depth.

4.2 Trend of Interdisciplinary Development

According to Feng Dezheng (2011), “multimodal m e t a p h o r d i s c o u r s e a n a l y s i s i s a n o p e n a n d interdisciplinary research field, which allows research from different levels and perspectives.”. This shows that multimodal metaphor research should not only be carried out in a variety of registers, but also absorb nutrition from the theoretical system and research paradigm of a variety of disciplines, so as to make its research meet the specific needs of different disciplines, and gain a broader research space and more long-term vitality. At present, the interdisciplinary study of multimodal metaphor is attracting more and more attention from scholars. The quantity and quality of related research are gradually increasing, which will be an important trend of multimodal metaphor research for a long time in the future. In fact, the basic characteristics of multimodal metaphor ’s representation types diversity also determine the interdisciplinary nature and trend of its related research. Image is not only the most important representation mode of multimodal metaphor, but also the earliest means of meaning communication in human society. It is widely used in various visual media such as static images, such as painting art, line art, color art, advertising picture, poster picture, illustration art, photography art, cartoon art and ynamic images such as movie images, game images, TV images and network video images, etc. The above image types are widely distributed in journalism, art, advertising, photography, archaeology, history, dance, architecture, even economics, politics, ethics, religion, military science, physics, biology and literature, chemistry, mathematics and almost all other disciplines. It can be concluded that with the increasingly extensive communication among researchers of various disciplines and the increasing integration of knowledge systems of various disciplines, the interdisciplinary study of multimodal metaphor should be one of the focuses in metaphor research in the future.

4.3 Trend of Conjoint Analysis

Forceville (2016) believes that in the visual metaphor of film, we should clearly distinguish between monomodal metaphor and multi-modal metaphor, but at the same time, we also need the joint analysis and common use of the two. The theoretical system of multimodal metaphor originates from conceptual metaphor, and its external representation is evolved from image metaphor. Image metaphor is a collection of single and multimodal metaphors, which are closely related, so it is an inevitable trend to make a joint analysis of them. Scholars have made some progress in this field. For example, Almeida & Bibiana (2015) analyzed and compared the multimodal metaphors contained in the sports newspaper cover describing the famous football star Cristino Ronaldo. Their research has drawn two conclusions: first, the source domains of metaphor involved in mono-modal metaphor and multi-modal metaphor are different from each other; Secondly, compared with the mono-modal metaphor, the number of source domains of multi-modal metaphor is more limited. It is helpful to enrich and perfect the theory of conceptual metaphor and multimodal metaphor. In addition to the mono-modal and multimodal analysis of metaphor, multimodal analysis of metaphor in many other registers is bound to increase. The existing researches on multimodal metaphor mostly focus on the corpus in a specific register, but rarely analyze multiple categories within the same framework, which restricts the universality of their research results. As mentioned above, the problems of data selection and circular verification in conceptual metaphor research have been criticized widely. Similarly, the single register research of multimodal metaphor will lead to the same research defects. As a matter of fact, multi register is the internal requirement and characteristic of multimodal metaphor research, and it is also the most important difference from conceptual metaphor research. It can be predicted that the multi register analysis of multimodal metaphor will become an important growth point, such as the dynamic multimodal metaphor and static multimodal metaphor in level of macro level, and the conjoining analysis of cartoon, advertising, photography, posters and illustrations in the micro level. This kind of conjoint analysis can break through the barriers of single register, enhance the systematicness and objectivity of the research process, enhance the accuracy and comprehensiveness of the research results, and is especially conducive to the systematic construction of multimodal metaphor theory. 4.4 Trend of Empirical Research The research approach of multimodal metaphor is similar to that of conceptual metaphor. In the initial stage of research, introspective and speculative methods are mainly used to analyze and infer the introspective or induced data. This research method relies more on the language intuition and subjective speculation of researchers, and the research conclusions are quite questioned and challenged. With the gradual maturity of the theoretical system and the in-depth development of research practice, multimodal metaphor research is bound to adopt more empirical methods to enhance the objectivity and acceptability of its research conclusions. It can be concluded that multimodal metaphor still inherits the existing approach of conceptual metaphor, and its related empirical research can be carried out from two aspects. On the one hand, from top to bottom, the existing theories are used to guide the relevant applied research, such as teaching experiments to explore whether multimodal metaphor has a positive effect on foreign language teaching. On the other hand, quantitative analysis of a large number of natural corpora is carried out from bottom to top through statistical software and mathematical analysis methods, so as to reveal the conceptual system and cognitive rules existing in human thinking level, such as corpus research of multimodal metaphor and ERP research. As a matter of fact, the empirical research on multimodal metaphor, including the application of small[1]scale corpus and mathematical statistics, is gradually unfolding, and has achieved remarkable results. Indurkhya & Amitash (2013) investigated the cognitive similarity in the process of visual metaphor understanding through an eye movement study. The experimental results show that both the generation and understanding of multimodal metaphor and the cognitive similarity of the two conceptual domains occur at the subconscious level, which helps to quickly search for the significant correlation between the two concepts, so as to identify the similarities between them. Littlemore & PéRez-Sobrino (2017) mainly adopts the quantitative research method to measure the speed and depth of multimodal metaphor understanding, the attractiveness to the subjects and the physiological influence on the subjects of different cultural backgrounds (Britain, Spain and China). The results show that there are great differences in the understanding of multimodal metaphor in advertisements among the subjects of different cultural backgrounds. At present, how to use corpus technology to effectively retrieve and identify multimodal metaphor is the biggest blocking point of empirical research, which will also be an important focus of multimodal metaphor research in the future. In any case, more and more empirical methods will appear in the study of multimodal metaphor, which is also the objective need of the construction and research of multimodal metaphor theory

REFERENCES

1. Almeida, C. M., & Bibiana F. (2015). Monomodal to multimodal metaphors in the Portuguese sports newspaper A Bola. RBLA, Belo Horizonte, (2), 403-420.

2. Forceville, C. (1996). Pictorial Metaphor in Advertising. London/New York: Routledge.

3. Forceville, C. (2007). Multimodal metaphor in ten Dutch TV commercials. The Public Journal of Semiotics, I(1), 15-34.

4. Forceville, C., & E. Urios-Ararisi. Multimodal Metaphor[M]. Berlin/New York: Mouton de Gruyter, 2009.

5. María Enriqueta Cortés de los Ríos (2010). Cognitive devices to communicate the economic crisis: An analysis through covers in The Economist. Ibérica, 20, 81-106.

6. Urios-Aparisi, E. (2009). Interaction of multimodal metaphor metonymy in TV commercials: four case studies. In C. Forceville & E. Urios-Aparisi (Eds), Multimodal metaphor (pp.95-117). Berlin/New York:

7. Mouton de Gruyter. Wang, Y. (2007). Cognitive linguistics. Shanghai: Shanghai Foreign Language Education Press.

8. Xu, Y. L. (2002). Contrastive linguistics. Shanghai: Shanghai Foreign Language Education Press.

9. Yang, Y. W. (2015). A study of multimodal metaphorical representation types in poster discourse. Foreign Languages Research, (3), 30-35.

10. Yu, N. (2009). Nov-verbal and multimodal Manifestation of metaphor and metonymies: A Case Study. In C. Forceville & E. Urios-Aparisi (Eds), Multimodal metaphor (pp.119-143). Berlin/York New York: Mouton de Gruyter.

11. Yu, Y. M. (2013). A study of multimodal metaphorical representation of news comics. Foreign Languages Research, (1), 1-9

(Vui lòng nếu có trích dẫn xin tôn trọng tinh thần học thuật; tác giả cùng cơ sở xuất bản bằng cách ghi rõ tên tác giả, năm xuất bản, cơ sở,/ NXB/ Tạp chí)