If you could step into a parallel universe, would you?
How can we meet in an alternate reality?
Like so many people living in a different country to the one where they grew up, Associate Professor Taehyun ‘TJ’ Rhee just wants to hug his mum.
“I haven’t seen my family and friends in South Korea for nearly three years, since the pandemic began. I want to feel as though I’m in my mum’s living room, communicating with her in a face-to-face, physical way. She is the same—she wants to see how we live in New Zealand, and see how her grandchildren have grown.”
Luckily, Associate Professor Rhee has the experience and technical expertise to solve this issue. He and his team at Te Herenga Waka’s Computational Media Innovation Centre (CMIC) are developing an exciting and potentially game-changing concept that will allow people to communicate and collaborate—or just hug their mums—across any distance while feeling as though they’re in the same place.
Step into the Televerse
“Connection is the thing humans need and want. We want to harness that fundamental human desire while trying to fill that gap between imagination and reality,” explains Associate Professor Rhee. “So we developed the Televerse—it’s an augmented version of the real world that we can teleport to virtually, from any place in the world. You might have heard of the Metaverse, which in its current form is a completely virtual environment—this is different. We want to build a more unified, real-time framework to help people connect in a richer way.”
He says people want to connect in a real environment, but there are limitations due to people’s physical locations “This has been amplified by the COVID-19 pandemic and travel restrictions.”
“So the technology we are providing with the Televerse is allowing people to connect in a real environment without the limitation of their ability to physically travel.”
The Televerse is a real-time digital twin of the real world, says Associate Professor Rhee. “It allows you to feel as though you’re in the other place while you’re at a different physical location. This is a big step up from video calls where users are limited by their own situational awareness and what they can see within the particular camera angle that’s visible on their screen. In the Televerse we are connecting in the same space—it’s a shared environment, meaning we can pick up on more subtle communication and have better situational awareness in remote environments. It’s an advanced platform that provides more immersive, intelligent and interactive communication between people wherever they are.”
He says it’s a hugely challenging project. “To engage with the Televerse you have to capture, model, stream, perceive, visualise, and interact with it. And this all has to happen in real time, in a way that’s much richer and more immersive than regular videoconferencing. We have to think differently. It’s not just sending our avatar to the other person; it’s bringing the other world to us so we feel as though we’re there even when we’re physically not.
“In addition to the visual information that will be shared in the Televerse, there are other digital interfaces we can use. For example, we can capture the chemical signals of different smells in data form, meaning they can be sent from one place to another where that chemical mixture can be reproduced. We can also share tactile feedback between locations with touch sensors and temperature regulation. Telerobotics is another tool, allowing us to do physical tasks remotely. The Televerse brings all these kinds of technologies together. They can be controlled from a distance by the data and electronic signals that can be digitised, encoded and decoded in different locations. This is a lot more straightforward than molecularising our bodies and quantum teleporting!”
Associate Professor Rhee says Wellington—and the University—is the perfect place to build this kind of forward-looking technology. “I worked in the tech industry for 17 years. I contributed to the manufacturing of devices and equipment, and also built platforms to use that gear. My academic research now is exploring what kinds of equipment or devices are required for making the Televerse framework, and to design the architecture and solve the algorithmic problems to make it work. We have our technical base in Wellington because it is a powerhouse of visual effects filmmaking, contributing to the global film industry in a huge way. We’re building our core framework of the Televerse using that visual effects knowledge base. Visual effects augment the live-action films with virtual assets, and we will augment human life with digital transformation in the Televerse. It’s given us a huge advantage.”
Associate Pofessor Rhee says the main axis of the Televerse is the three ‘I’s: immersive, interactive, and intelligent.
“It’s a bit of a cheesy phrase but I think my vision is to use the Televerse to unlock the power of teleportation—to give people the power to go any place in the world, any time they want to, and connect with people from all over the world without having to physically travel there.”
“It’s not about trying to eradicate physical travel in favour of the virtual, but telepresence will produce fewer carbon emissions than flying to a location. It will definitely help us to reduce unnecessary trips, and make life better for all of us by improving how we connect with people, while also being good for the environment.”
An obvious impact of the Televerse will be in areas such as tourism or for remote medical diagnosis and treatment, he says. “It will also be hugely helpful for the education sector—international students affected by COVID-19 restrictions, for example, would potentially be able to attend Victoria University of Wellington using the Televerse. It would provide an immersive media platform so they feel they’re in New Zealand.”
The entertainment industry will also benefit from the technology the Televerse will provide. Imagine being able to watch live concerts or sports matches from front-row seats, or jump into the action of your favourite film or video game and interact with the characters.
He says the Televerse will require a great deal of collaboration. “We need a lot of heads together to make this happen—a single institute or organisation can’t do it alone because they’re limited to using their own hardware or media platform. This is an open innovation project. We are keen to share what we’ve developed as much as possible, because we’re going to need a lot of collaboration to bring the Televerse to life. If we want this to contribute to as many people’s lives as possible, we need to work together to build it in a unified way.”
Associate Professor Rhee is adamant the Televerse is not about replacing the real world with a digital one.
“Where the Metaverse is trying to create a whole new, virtual world, the Televerse is an enhanced, augmented version of the real world. Real life is our base—we won’t change it. We just want to make life better.”
He says the improved communication that will result from better global connection will enhance people’s empathy and understanding. “We want to fulfill that fundamental demand for human contact—it’s a universal desire. And when the Televerse is up and running, the first place I’ll go is to my home country. I just want to hug my mum.”
Related links
Built with Shorthand