Simulating Social-Emotional Interactions to Improve Communication and Empathy

Clint Carlson, Director of Education Technology Innovations - Department of Psychiatry, University of Colorado

As we navigate our changing world, using technology to adapt the tools we use to communicate allows us to collaborate and build using new platforms and strategies to connect. In this process, it is critical that we also emphasize the role of human connection in the establishment of social-emotional soft skills, and consider how they may be aided by new technologies.

The pandemic has poked holes in the dream of reaching all learners via technology— unsurprising to those of us embedded in the intersection of academia and technology, but a shock to many others. While we’ve scrambled to get remote classrooms and asynchronous teaching and learning in place, it has been difficult to take a long view. However, even as we have struggled to maintain learning communities, recent technological strides are making remarkable progress which enables us to better interact with each other.

We have simultaneously entered a new era, wherein what and how we say things are broadcast globally. While the benefits of addressing inequity in our society have been profound, honest missteps in speaking to others can now be live-streamed and preserved for all time, as if we are static beings. The need to build social-emotional communication skills, and to provide space to make mistakes and learn, is imperative. These skills are a crucial part of individual growth and are more critical than ever. Virtual and augmented reality provide one opportunity to construct such spaces, and artificial intelligence and machine learning can allow us to avoid harm and dynamically improve the learning process.

Among all the buzz around the Metaverse and Web3.0, there are a variety of tools and workflows available right now advocating the landscapes of VR/AR and artificial intelligence (AI) to provide training and education opportunities for the improvisational side of working with others. The intersection of these bleeding-edge technologies results in realistic simulations of human conversation and allows us to re-create a range of detailed experiences in the human condition; providing natural verbal (rate of speech, tone) and non-verbal (body posture, eye contact) communication with a range of life-like avatars. Harnessing dynamic data into machine learning profiles and allowing these avatars to understand your specific communication style and history of your interactions provides life-like simulations around a variety of difficult topics (personal, medical, et al.). This is a huge leap from what most of us have experienced from the basic applications of this technology currently seen in online chatbots and A.I.-driven “smart-home” apps.

“While the benefits of addressing inequity in our society have been profound, honest missteps in speaking to others can now be live-streamed and preserved for all time, as if we are static beings”

These simulations provide a safe place to explore communication techniques, both new and old, allowing us to skill up before entering these “one-shot” conversations with students, patients, peers, and strangers. Thus, we are able to hone important soft skills in advance of critical first impressions and the delicate handling of difficult topics.

Furthermore, these advanced technologies allow us to simulate not just the now, but also what a follow-up conversation may look like in a few days or a few years down the road as we simulate key moments of an individual’s lifespan. This further allows us to emulate how understandings, personalities, and the techniques needed to interact appropriately evolve in individuals over time providing a bigger picture of holistic care.

Like most disruptive technology, the vulnerability lies in anticipating where these technologies will lead. It is reasonable to expect many deeper applications of this work as AI-enabled AR avatars become better at learning our communication styles. This has the potential to dynamically challenge and continue to educate upwards, and we are left with endless potential for branching applications.

For example, we can connect a doctor with a very detailed communication style (rate of speech, eye contact, words used, pauses, etc.) with a patient determined to best receive that nuanced communication style. For another example, imagine parents interacting with an avatar to build skills in discussing depression with a teenager, who can pivot and experience the simulation from the point of view of the teenager. Allowing a parent to experience being talked to by adults using a variety of language and being limited by responses from that child’s POV could quickly build empathy and more authentic understanding— all from the repeatable, safe comfort of your device at home.

While these examples are on the horizon, even closer are augmented technologies that will allow a doctor to review and collect patient medical information while maintaining eye contact and fostering a more human connection in real-time.

As is often the case with emerging technology, it can be difficult to see the long-term benefits of new tools when blended with other methods. These tools, like most, belong in a metaphorical toolbox with other existing and emerging pathways to best address unique individual needs. As technology advances and aligns more closely with the realities of our world, taking a moment to pause and look beyond the week reveals an exciting landscape that consistently unveils innovative ways to enhance our interactions.

Artificial Intelligence

Machine Learning

Augmented Reality

review