How digital people could improve access to healthcare

Waipapa Taumata Rau, University of Auckland’s ABI researchers are recipients of MBIE’s new Catalyst Strategic Investment Fund, in partnership with Soul Machines. This was set up in 2021 for internationally connected research projects to explore the potential of AI in healthcare.

Soul Machines is a New Zealand AI company that pioneered the creation of animated digital people that interact with humans in real time and in a lifelike manner. Recipients of the fund have access to Soul Machines’ digital people for the duration of their projects.

Professor Merryn Tawhai, deputy director of the ABI, will use the funding to lead an international consortium to explore how Soul Machines’ digital people could be used to support the management of Type 2 diabetes and cardiovascular disease (CVD).

Both conditions are intrinsically linked, treated in tandem and managed primarily at home by the patient with clinician advice and prescription medications.

“But many patients struggle to understand their condition and how they should manage it, and very often don’t comply with treatment strategies recommended by their doctor,” says Professor Tawhai.

“This is especially true of lifestyle changes needed to get the greatest improvements in health and quality of life. While we know intensive interventions with clinical and personal coaching are effective, their cost puts them out of the reach of most people.”

Her team aims to create a platform of interconnected digital tools, starting with a culturally appropriate digital health navigator to coach patients in understanding and self-managing their condition.

The project will draw on the research being done as part of the ambitious MBIE-funded ‘12 Labours’ project at the ABI, which involves the development of ‘digital twins’ (representations of human physiology built on mathematical modelling) using clinical images, diagnostics data, implantable and wearable devices.

The project will involve linking in-home and body-worn sensor data with a digital health navigator that can ‘read’ patient-specific information and ‘talk’ to a patient in real time to help them better understand and manage their condition.

“Rather than the episodic and reactive doctor-patient relationship typical in the real world, this would allow for continuous interaction that encourages and supports the management of chronic disease, reducing barriers in access to care and reducing health care inequity,” says Professor Tawhai.

Professor Mark Billinghurst, head of the Empathic Computing Lab at the ABI will use the funding to lead the ‘Tōku Hoa: A Personalised Agent for Mental Health’ project. This international collaboration will explore how Soul Machine’s digital people could provide long-term monitoring of mental wellbeing, and when appropriate, remote access to professional treatment.

The project will involve the development of software that uses smart phones, watches, and other wearable sensors to monitor a person’s emotional wellbeing through physiological cues such as heart rate, speech, and activity.

A digital person will appear on the user’s phone or computer in response to changes in the user’s mood or when requested and use a conversation-based approach to provide personalised feedback to help people manage their own depression. “The goal is for the digital person to become a trusted virtual companion,” says Professor Billinghurst.

The aim is also to create a digital character that can be customised to match individual preferences, such as their ethnicity, age and gender. The initial target audience is young Māori men, who are overrepresented in mental health statistics; the researchers will be working closely with Māori to incorporate a Māori approach to mental health treatment.

The project also involves developing an interface that would enable users to connect to a remote professional therapist and have a live therapy session, mediated and supported by the user’s digital person which will have detailed knowledge of the user’s mental health history. The sessions could be conducted online or even using Virtual Reality where both the person and the therapist could be immersed in the same space, regardless of where people were in the country.

“The main idea behind the research is to create a platform that can provide an innovative combination of targeted mental health monitoring, support from a personalised digital person and an immersive therapeutic environment that increases people’s ability to engage with real mental health professionals,” says Professor Billinghurst. “This will help address the long wait time and lack of access to mental health support services in Aotearoa New Zealand, particularly for rural Māori.”