Virtual / Mixed / Augmented Reality (XR)
XR is revolutionizing the way we interact with computers. Since trying Google Cardboard for the first time and buying an HTC Vive on launch day I've been enthralled with the prospect of exploring virtual worlds. Working at Microsoft I gained experience in working in mobile AR and saw its potential for mass adoption.
I've been experimenting with the mediums for the past few years, trying every experience I can and learning the development process in Unity.
Prototyping interactions in VR / AR. Built in Unity. Above - a memory palace idea showcasing some photogrammetry scans and photos from a trip to Morocco.
VR Light Painting
If you've ever light painted before, you know that it's iterative since one can only imagine the path the light takes through the scene. You run around in the dark with a flashlight, and then peer at the camera to see if the shot turned out.
We love doing this, but for VR, we wanted to create an experience that doesn't exist in reality: slightly eerie, otherworldly, and supernatural.
This project is an ongoing labour of love, but we're sharing our process here to give people a sense of where we are headed.
Our goal is to makes it feel as if you are painting with light on a foggy beach at night with the stars all around you, as though you're inside a long exposure shot.
I created the video above (Tiltbrush + Cinema4D + Octane) to show a possible tone and visual target, and Michelle worked on creating a corresponding interactive VR experience in Unity.
For reference and inspiration, and to create both the rendered scene and the Unity environment, we went to Tofino, BC and shot a variety of 360 videos, panos, and timelapses, and reconstructed parts of the beach using photogrammetry.
I took hundreds of shots of the beach to recreate a large section of Long Beach. Our photogrammetry tool of choice, Reality Capture, produces photorealistic results with an extremely high poly count. We worked with 3D artist Reece Mills to research how to bring down the size of the model and texture, while preserving its perceived quality.
drawing with light in vr
Michelle implemented our light-painting to be similar to what you might experience with a very basic Tilt Brush. Once she had a "flashlight/wand" painting a trail, she started exploring both how to make it look smoother/more fluid, and what else the player could do with the light once it was in the scene.
We asked ourselves when you paint, how does it affect the scene? Why do you paint—does it reveal the next element in the scene, or move a story along? Is it simply a fun mechanic that you can play with, and should it have physics? How physical is this paint?
We are currently working on some of the story-based options, but in our tests, we also looked at how it could help create the right ambience (more on that below).
creating ambience with light and fog
Long Beach in Pacific Rim National Park (near Tofino, BC) is one of my favourite places, especially at moonset, when the stars come out. The shadows and fog make it feel like you are in another world. We wanted to see if we could use fog and lighting in Unity to create a similar feel in our experience.
Light painting in Tofino is just one instance of a memory we both really enjoy. As we work on this project, we're always trying to understand better how this gets us closer to reliving a memory, or perhaps experiencing someone else's interpretation of that memory.
We are always wondering: What other things can we create that would help us revisit an event from the past, or just take us back to a happy place?
We'll share more as we learn. The road is long, and we look forward discovering it.
3D Spherical Video
One of the problems with current VR tech is the sheer computational overhead that you deal with when rendering a scene twice at 90fps. With pre-rendered stereo content from Octane, it becomes possible to create photorealistic effects and much higher quality rendering than could be easily achieved in Unity or Unreal Engine.
Let's go for a VR hike
Photogrammetry and VR go together like peanut butter and jelly - one of the main reasons I started learning photogrammetry was to bring it into the Virtual Reality. On a hike to Hanging Lake (just outside of Glenwood Springs, CO) I was inspired to capture a selection of scenes along the trail. I'm currently working on compiling those scenes into a single stereoscopic spherical video to publish on Youtube, Vimeo & Facebook's 360 platforms.
Boat Life VR
One of my favorite ongoing projects is to create an interactive VR scene using the photogrammetry scans of boat I lived on last year. The aim of the project is to attach photos and videos to relevant points of the model and create a sort of holistic documentary experience. You'll be able to walk through the various cabins as well as interact with multimedia content in that space. For now, I've compiled a 3D Spherical video as a preview.
Experiments in Synthesis
Interesting possibilities lie at the intersection of mediums - these works represent the results of combining VR-created artwork with videos, 3D graphics, and photogrammetry. Above - a marble run track drawn in Tilt Brush is used for a rigidbody simulation in Cinema 4D.