Architects and bioengineers create shape-shifting structure that talks in poetry

The installation, LightSense, is one of 20 installations to feature in Garden Aotearoa co-hosted by the University of Auckland and Te Herenga Waka-Victoria University of Wellington, which can be visited online as part of the 2021 Ars Electronica festival.

Ars Electronica is an international festival that explores the intersection of art, design and technology. The festival originated in Linz, Austria, and in pre-pandemic circumstances took place there.

In 2020 the festival moved online due to travel restrictions, when the New Zealand gallery was co-hosted by the University of Auckland and Te Herenga Waka-Victoria University of Wellington, as it has been again this year, at Garden Aotearoa.

LightSense was created by Uwe Rieger and Yinan Liu of the arc/sec Lab at the School of Architecture and Planning, in collaboration with the Empathic Computing Lab and Augmented Human Lab at the Auckland Bioengineering Institute (ABI).

LightSense is a structure that invites visitors to talk to it. It responds to the tenor of the conversation and its mood changes as a result. The shifting mood of the structure is demonstrated by changing shape and with lights and graphics, creating pavilions of caution, love, hostility and joy. “It’s about exploring new ways of that architecture can engage with people,” says Associate Professor Uwe Rieger.
 

The structure was taught to to listen and converse with visitors by Tharindu Kaluarachchi, ABI, who used Artificial Intelligence (otherwise known as machine-learning) and 60,000 poems.

Poetry is a form of literature that distils emotions, such as love or happiness through language, so an ideal resource for this project. “Using AI, we were able to train it with poetry, to give the structure a personality,” he says.

An example of a real conversation, recorded in his trials:

Person: Happiness is when you are enjoying the moment, feel happy, glad and comfortable. When you spend time with family you love, or chat with your best friends, travel around with friends you like, that’s happiness.

Pavillion: I am happy to have you by my side, and take you along to our new adventures. My energy is endless. Together, we will go to the end of the rainbow … I will take you in my arms, and shelter you with silk roses.

Enabling the structure to talk involved using WaveNet voices using a combination of male, female and gender-neutral voices, and modifying pitch, and adding punctuations and pauses in the right places. “It doesn’t exactly talk like a human being might,” he says. “It does have a synthetic quality about it, but AI is getting there”.

Mr Kaluarachchi is studying his PhD at the ABI supervised by Associate Professor Suranga Nanayakkara, head of the Augmented Human Lab, whose team is focussed on creating ‘enabling’ human-computer interfaces as natural extensions of our body, mind and behaviour.

His research focuses on Human-Centred Machine Learning (HCML, a relatively new area of research around the world), in which he aims to bridge the gap between AI research and human behaviour and needs.

“Through LightSense, I could explore HCML approaches alongside architects and designers to better understand the intersection of AI and architecture,” he says. “Apart from this, it helps me better understand the general population’s perception of AI, how AI can enhance entertainment, and the gaps between the direction of AI research on entertainment and human expectations.”

Ars Electronica can be visited online 8-12 September, and all events are free, though some aspects of the Ars Electronica festival originating in other countries carry modest charges. A full programme of New Zealand events is available at ars.nz.