Javier Molina and his VR team at NYU Tandon decided that our very own virtual reality software, LiveAction was the best solution to deliver the believable character motion needed for their latest VR theatre production, To Be With Hamlet. Streaming natural behaviours from actor to 3D character in real-time was imperative for greater storytelling, as well as a convincing experience for audience participation during live shows.
Equipped with HTC Vive headsets, participants can see as well as interact with the virtual space, whether in the specially designed black-box theatre, or from a remote location. The actor’s movements are tracked using motion capture technologies, and the data is real-time fed through IKinema’s LiveAction to produce realistic, quality motion from actor to 3D character.
“By immersing [the audience] in the setting — allowing them to virtually share Hamlet’s world for a short time — we hope that they might gain a more visceral understanding of Hamlet’s experience”.
David Gochfeld | Director | To Be With Hamlet
The audience wear HTC Vive headsets to see and interact with the virtual space
GDC Vault, 2017
In this talk, the production team behind ‘To Be With Hamlet’ explored the technological and aesthetic considerations behind performing Hamlet live in a detailed, multi-user virtual reality environment. Attendees learnt how the team simultaneously utilised 3D scanning, photogrammetry, facial motion capture, and real-time skeleton-streaming technology to fuse theatre’s sense of intimacy with virtual reality’s powerful sense of immersion. The project’s director shared how he learned to direct live performance in VR and create a world that enabled audience members to both explore the environment and engage with the story.
IKinema’s LiveAction retargeting tool
Animation graph: live-stream MoCap data, retarget to IKinema rig for output of final animation pose via Unreal Engine
However, the NYU aren’t strangers to LiveAction – back in 2015 they harnessed it’s incredible digital puppetry capabilities to drive virtual Adam in The Return, an interactive art installation where visitors to The New York Metropolitan Museum could interact and ask Adam questions about the sculpture’s (Tullio Lombardo) creation, travels and return to the gallery.
The Met: The Return – taking centre stage, actors’ live performance captured, streamed and retargeted through LiveAction to virtual Adam for visitor interaction
To Be With Hamlet is a non-profit organisation, and we’re proud to donate LiveAction towards this team’s incredible efforts to forge new and innovative experiences in the space of virtual storytelling. You too can join the campaign by donating via the gofundme portal.