First Year Review

The Handbook of Emotional Cognizance Design

Table of Contents:

Introduction

Topical Section 1: Analysis and Application of Emotional Intelligence

Topical Section 2: Visual Media Interpretation and Influence

Topical Section 3: Cultural Influence on Emotions

Introduction

At the heart of all storytelling is connection to the audience. It may be a large generalization but if there isn’t any sort of connecting point or link between the listener and the content, then they aren’t going to care about the story they are being told. In modern days, there are endless bounds of content being uploaded to the internet everyday. In fact, on YouTube, according to their own website, it is said that more than five hundred hours of content is uploaded every minute (YouTube for Press 2024). That means that there are over twelve thousand days worth of video content uploaded to a single website every single day. And that is just one website – one platform. There are hundreds of other platforms for video content without even talking about major production studios and other methods of distribution. It doesn’t include movies released to box offices, indie films making their debut to the world, or film festivals showing specialized content. And it definitely doesn’t include a majority of the films made before the internet was popularized. That being said, it is virtually impossible for someone to be able to see every single film or video that is produced in the world.

Apart from that, modern society is constantly moving into a digital space. More and more traditional mediums and experiences are moving into an online format. Keeping up with friends, talking to family, interacting with strangers, finding information, navigating to places, ordering food, scheduling appointments, and many more activities now all take place online. Common tasks have turned into a digital experience making it extremely hard to do anything other than exist digitally. Companies are constantly striving to find the most efficient ways to grab viewers attention and absorb their media as extensively as possible. Looking at all of this, I’ve come to the conclusion that this makes information and stimulation incredibly more accessible and almost a constant in the lives of those that use a computer.

So, in a world saturated with a practically infinite amount of media to choose from and interact with, coming from a variety of different outlets, how is anyone meant to retain any of what they see? When watching endless bounds of digital content, what sticks with you and what gets cast aside? What do you remember and what impacts you the most?

I believe the answer to these questions could lie in the content that is being created and how viewers are able to interpret and internalize that information. Specifically, I think that the thing that makes all of this content worth watching and remembering is the emotional connection it is able to make between what is on screen and its viewers.

The stories that resonate the most with me and the ones that have stuck with me throughout the decades are the ones that touched me emotionally. The films where I felt myself going along on the ride with the characters, where I laughed and cried and poured my heart out alongside them on screen. The films where I couldn’t stop thinking about them for weeks afterwards, always in the corner of my mind. The ones where I felt so connected to the characters I watched that I saw myself as them. I embodied them. I imagined myself becoming my favorite fictional characters and in return had a piece of their story become a part of me, to carry forever.

That is not to say that only content with heavy emotional themes can be memorable, but more so relates to the ability to connect with the content and what is being shown on screen emotionally. I believe every action, even something mundane, can be linked to an emotion and some sort of emotional response. Content, boredom, and even neutrality are all very real emotions to explore and potential connection points between the viewer and the film. For example, a science fiction movie where the characters are fighting aliens and jumping through portals can be grounded by cooking or eating dinner. A show about magical creatures in a forest can be counterbalanced by characters working their day jobs or chatting idly with another character about something other than the direct plot. These emotions and events are not over the top or for example, contain heart wrenching depictions of lost love or death or being reunited with something forgotten. Everyday actions and everyday emotions have the potential to carry the same weight when it comes to connection as something grander. It could be said that this is in part due to the commonality and the similarities between the real world and the fictional world, and the direct relationship between what is being shown on screen and what the viewer experiences naturally. In fact, I would argue that without these ‘plain’ emotions being incorporated into films, then there wouldn’t be as strong of a connection to the more intense things.

There have been some studies done on the mundane in films, but one I would like to call attention to is the research done by Zoe Crombie published in the Journal of Anime and Manga Studies. Her works look at the mundane and everyday scenes and plots that are highlighted and explored in worldly recognized films like those from Studio Ghibli. She talks about how normal things like food, cleaning the house, and going shopping are sensationalized and the impact that has. She says “films have a phenomenological impact, leading people to want to experience the sensations they achieve through watching the films more directly and physically” (Crombie, 2021). She talks about how people are connecting greatly with the things they are seeing on screen and trying to recreate them at home. In one example, people are going out of their way to recreate the meals they see characters eating to connect the fictional content to their lives in the real world. . After going through some more examples and points, she goes on to conclude “By romanticizing domestic and everyday tasks in the majority of their works, Studio Ghibli provides an unusual reverence for the mundane … These scenes of the spectacular mundane exemplify the compassion that can come from domestic life” (Crombie, 2021). Emotions come in all forms, happy, sad, big, and small, and there is no limit on which ones are able to have an impact on a viewer. In this regard, are all emotions made equal?

All of that is to say, the main issue being covered in this handbook is the theme of emotional cognizance. Emotional cognizance can be defined as the awareness, knowledge, and/or perception of emotions in oneself and how these emotions relate to others. More commonly, it is often referred to as emotional intelligence which is defined by the Oxford dictionary as the capability to be aware of, control, and express one’s emotions. To put it extremely generally, emotions are something that everyone feels or has the potential to feel. They come in all sorts of shapes and sizes and impact people in a variety of different ways. Relating back to the previously discussed digital content consumption and animated films, I believe emotions play a key role in storytelling and having media resonate and impact viewers. It is my belief that emotional connection can come in a variety of different forms and that being able to recognize those emotions, the connections, and how those connections are being made is taking a step toward controlling them and better understanding the media we are consuming and creating. 

This handbook's goal is to explore emotional cognizance and provide an overview of the problem space in which this topic resides. It goes over topics such as the analysis and application of emotional intelligence, visual media interpretation and influence, and the potential cultural influence on emotions. It explores a variety of different perspectives and backgrounds that contribute to arguments in the field and overall aims to give a basic understanding of what incorporating emotional cognizance into artistic creation and design really entails. What are the different things to consider and how do they connect with one another?

The first topical section of the handbook looks into how emotional intelligence is being utilized in a variety of different fields currently in the hopes that this understanding can inform its potential uses for design exploration and what areas are in need of improving on.

There are many different ways emotional cognizance and emotional intelligence learning are being implemented in the world and even more gaps in this that can be explored. In this handbook, there will be several different ones discussed, but there are few major themes and places where they can be seen most commonly. Moving through time, at an early age of participants, emotional intelligence learning programs and lessons are implemented in schools to teach young children how to handle their emotions. While everyone feels emotions, it can be incredibly difficult – especially in the beginning – to identify those emotions and what can and/or should be done about them. In the article Teaching emotional labor through television and film, the researcher James talks about how emotional intelligence is currently being taught and then focuses on the film application of emotional labor for children. Children know they are upset, but they often aren’t able to understand why they are upset, what caused it, and the specific emotion that is connected to that upset. Are they angry, sad, frustrated, a combination of these, or none of the above? And then what? The child is upset, but how are they supposed to remedy that? What are the strategies to regulate emotions and get back to an emotional neutral or positive state? (James, 2015). Teaching emotional intelligence in these spaces consists of a lot of picture diagrams with scales to signify how happy or sad someone is, a lot of picture book readings, and talking through issues to help self regulation.

Looking at traditional (U.S. centered) children's media intended for education that can be seen on networks like Nickelodeon Junior and PBS (Public Broadcasting Service) Kids, emotional learning takes on a completely different form. These television shows focus on simple language and in a lot of cases, talking directly to the camera – or to the child on the other end of the screen. After watching a variety of these television shows, I’ve noticed that the episodes that focus on teaching some sort of emotion as their main goal utilize direct reinforcement techniques to emphasize what characters are feeling. This means saying phrases like “(this character) is sad because this thing happened”, and usually follows up with some sort of explanation of what should be done about the emotional situation like hugging or apologizing. Shows like this and a good amount of classroom content takes this very direct approach to teaching and learning.

Another space where emotional intelligence learning research was found to be popular was in the work environment. Companies all over the world are implementing different policies and programs intended to increase their workers' empathy to promote a better and more productive working environment. As discussed later on in the handbook, the case study Emotional Intelligence and Soft Skills: A Case Study on Information Technology Employee Performance looks at the application of emotional cognizance and learning in the workplace – an office environment – with a focus of the soft skills that are both informed by and through emotional intelligence (Tyler, 2022). 

A big thing that this section of the paper addressed however is the different aspects of emotional cognizance research that are currently happening and how it is being utilized. For example, a big focus in a few different articles in emotional facial recognition. Both Incorporating structured emotion commonsense knowledge (Chen et.al, 2022) and Real-Time Video Emotion Recognition (Zhang et. al, 2022) both talk about how they are using different technologies – including video analysis and biometrics – to analyze how people are feeling, their emotional perceptions, and recognizing the similarities and differences between minute facial features and responses. This can be utilized in standardizing media content in understanding how people react to specific stimuli and understanding how context impacts each individual's interpretation of what is happening (Chen et.al, 2022). Understanding the context and the individual factors that go into a person's response to what they are seeing could be an important part of being able to understand their triggers and the emotions they are displaying.

The third topical section focuses on the interpretation of emotions and how that differs across different cultures and demographics. By looking into potential differences amongst groups of people and/or individuals, the hope is that a more universal understanding of how emotions can be applied to design solutions can be made. For starters, once again speaking broadly, there are differences between how different countries view the expression of emotions. Not every country has these differences and if they do they are not always significantly large, but it is important to acknowledge the potential for these differences.

For example, in the research study The Sun is no fun without rain, Jobson looks at the color–emotional associations across different cultures. Specifically, this study looks at the connection between the color yellow and the emotion joy. The study suggests that the association between yellow and the positive emotion of joy is stronger in regions further from the equator and in countries with more rainfall. This could lead to suggest that the physical environment influences the emotional meaning of colors (Jobson et.al, 2019). Looking at the association between emotion and color and how it varies between different regions of the world could be an important factor in determining emotional impact and ways of establishing connections. If people connect and associate different emotions to different things, what is the best way to make an effective design or artistic piece that will get the message across? Can design be made universal to account for these interpretational differences or are there limitations?

When attempting to design solutions, the audience that the solution it intended for needs to be taken into consideration. Is there a way to cover everyone under one design solution? Is there a way of standardizing what this looks like through multiple cultures? Probably not. However, when picking a specific audience to tailor your design to and then deciding on the audience you want to reach, the research in this section has led me to believe that cultural differences is most likely an important factor in making an effective design. How important do different groups view emotional intelligence? All of these things can be taken into account when designing for and with emotions.

Curating characters, plot, and visual elements inside of digital media – especially films, animations, and games – based on emotional feedback is nothing new. For games, there are play tests – what do the experienced users do, where do they go, what did they enjoy the most, what was confusing, etc? For film content, where did the viewer look away, where did they get bored, what made them so invested that they couldn’t look away from the screen? That being said, creators have recently started taking on the task of getting to the bottom of what makes users or viewers more engaged with technology and working to optimize their emotional impact this way. 

Currently, there are several different innovations happening in the digital design space when it comes to emotional cognizance. For example, there has been experimentation with using different personal tracking technology to understand what is most appealing to audiences and what draws their attention. For example, the book Affective Information Processing looks at computer-human interaction and analyzes facial expression recognition and emotional speech synthesis (Tao, 2009). Here, this mainly includes eye tracking software to see what viewers are most drawn too. There are also cases of heart rate monitors being used to understand how different visual images are being interpreted by the audience.

Facial tracking softwares is being used to teach computers what certain emotions look like on a human face and using that data to transfer these emotions onto different characters and objects. There is research being put in to best understand how to convey each motion the most effectively and efficiently by looking at how it is described on real people like in the study Effect of Adding Emotion Recognition to Film Teaching (Shang-Chin, 2021).

All of this and more is discussed in topical section two, which focuses on visual media interpretation and influence. Another topic that is highlighted here is the differences in video games as visual media and the teaching and learning process. Both researchers Newman and Sedeño in separate research studies described utilizing the video game system to customize the experience for different demographics and age groups and looking at the impact that that customization has on the impact and messaging of the content (Sedeño, 2010) (Newman, 2024).

Most interestingly to me, when researching emotional cognizance, I came across a case study that introduced to me the idea of personalized animation. The case study A Trip to the Moon:Personalized Animated Movies for Self-reflection by Fengjiao Peng et al. looks at creating customized animations based on user feedback and self-tracking data (Peng et al., 2018). They took input data from surveys and questionnaires, and they used that data and other design and film principles to try and customize an animated film that would have the most emotional impact to the viewers who filled out those surveys. They looked at things like personal history and other demographics to try and determine what changes they could make to their base – control – story in order for it to have the most impact. They changed things like color schemes, music playing, character movement and actions, but otherwise kept the plot of the story mainly the same.

This idea of personalized animation is highly intriguing and makes me wonder what areas of it could be improved on and what gaps in research there are for me to explore? Upon first searches, there didn’t appear to be many other instances that matched this study exactly. There is of course interactive storytelling media out there in the form of branching narratives and video games with multiple endings, but there didn’t seem to be a lot in the way of films specifically. Initial conjectures lead me to believe that there would be space to make this have real-time impact and combine previously mentioned biometric or facial tracking data in order to best create something of major influence and impact. There is also cause to say that recreating the study in order to attempt to validate it is also an option. 

All of this does bring up some other questions and concerns about what design solutions look like in a creative and digital field. What ways can emotional intelligence be added to the media industry to make better products and solutions? What areas need this involvement or could benefit from it the most? What would be hurt by it? Does increasing efficiency in being able to connect with others and an audience in digital media have any negative impacts? When things become too personalized, does it too narrowly scope the kind of content people are viewing? While giving the illusion of expanding art, does it really limit it? Should creative works be left up to the sole universal interpretation of the viewer?

This handbook delves into these complexities, offering insights into the intersection of emotional intelligence, design, and cultural diversity. From classroom curriculums to corporate environments, from film studios to interactive media, the threads of emotional cognizance weave a tapestry that shapes our understanding of storytelling in the digital age.

As we navigate the intricate web of emotional cognizance within storytelling and media consumption, I’m compelled to wonder: Amidst the vast ocean of digital content, what truly resonates with audiences? Could it be the emotional bonds woven between content and viewer that leave an enduring imprint? Through exploration and introspection, we're challenged to navigate the evolving terrain of storytelling in the digital age, armed with the vital tool of emotional intelligence. So, as we venture forward, could it be that emotional cognizance holds the key to unlocking a more empathetic and inclusive media landscape.

Works Cited

Chen, J., Yang, T., Huang, Z., Wang, K., Liu, M., & Lyu, C. (2022). Incorporating structured emotion commonsense knowledge and interpersonal relation into context-aware emotion recognition. Applied Intelligence : The International Journal of Research on Intelligent Systems for Real Life Complex Problems, 53(4), 4201–4217. https://doi.org/10.1007/s10489-022-03729-4

Crombie, Z. (2021). Spectacular mundane in the films of Studio Ghibli. The Journal of Anime and Manga Studies, 2, 1–26. https://doi.org/10.21900/j.jams.v2.507

James, K., & Paino, M. (2015). Teaching emotional labor through television and film. Feminist Teacher: A Journal of the Practices, Theories, and Scholarship of Feminist Teaching, 25(2–3), 139–152. https://web.p.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=0&sid=b1e53dd0-d9c9-4b14-b89e-91959b595a5d%40redis

Jobson, L., Mirabolfathi, V., Moshirpanahi, S., Parhoon, H., Gillard, J., Mukhtar, F., Moradi, A. R., & Mohan, S. N. (2019). Investigating emotion in Malay, Australian and Iranian individuals with and without depression. Scientific Reports, 9(1). https://doi.org/10.1038/s41598-019-54775-x

Newman, E. J., & Schwarz, N. (2024). Misinformed by images: How images influence perceptions of truth and what can be done about it. Current Opinion in Psychology, 56, 101778. https://doi.org/10.1016/j.copsyc.2023.101778

Sedeño, A. (2010). Videogames as cultural devices: Development of spatial skills and application in learning. Comunicar, 17(34), 183–189. https://doi.org/10.3916/c34-2010-03-18

Shang-Chin, T., & Hao-Chiang, K. L. (2021). Effect of Adding Emotion Recognition to Film Teaching—Impact of Emotion Feedback on Learning through Puzzle Films, 13(11107), 11107–11107. https://doi.org/10.3390/su131911107

Tao, J., & Tan, T. (2009). Affective Information Processing. https://doi.org/10.1007/978-1-84800-306-4

Tyler, J. (2022). Emotional Intelligence and Soft Skills: A Case Study on Information Technology Employee Performance. https://scholarworks.waldenu.edu/dissertations/13856/

YouTube for Press. blog.youtube. (n.d.). https://blog.youtube/press/

Zhang, K., Li, Y., Wang, J., Cambria, E., & Li, X. (2022). Real-time video emotion recognition based on reinforcement learning and domain knowledge. Ieee Transactions on Circuits and Systems for Video Technology, 32(3). https://doi.org/10.1109/TCSVT.2021.3072412