As we march into the tail end of 2016, “The Year of VR”, let’s take some stock in what has happened this year, look ahead at how the market will develop, and then drill down to see what this means for employment and business prospects within the sector. The dawn of consumer VR happened this year, three major premium headsets launched and we are now talking about millions of premium VR customers for the first time.
Oculus Rift & HTC Vive
It began with CES in January 2016, Oculus Rift announced its pre-orders for their headset. It would include an XBOX 360 controller and basic positional tracking up to 9 feet with a single camera. Mid summer, about the time that Oculus Rift should have been shipping its CV1 headsets to customers, HTC Vive announced that its headset was shipping and that they had plenty of supply and could guarantee delivery within three weeks. HTC had jumped on what looked like a manufacturing and distribution problem for the young Oculus. Component shortages and other manufacturing delays put some Oculus customers almost 2 months past their shipping window. Oculus was kind enough to refund the shipping costs to those who had to wait, but the delay was costly in terms of public sentiment. What arrived by early summer was a lightweight, high-definition VR headset with built-in speakers, and a modest selection of games including 2 exclusives for pre-order customers: Eve Valkyrie & Lucky’s Tale. The problem was that a better headset was out, when the hype cleared, HTC Vive had beat them on capability, supply and quality.
Unlike Oculus, the Vive came with robust positional tracking in the form of Lighthouse, complete with a “chaperone” system to keep players from running into walls. This enabled impressive walk-around VR experiences which were hard to match on the Oculus. Also, HTC Vive came with 2 positionally tracked 6 DOF (Degree of Freedom) controllers at launch. Something Oculus has promised, but has yet to deliver. There were some nice exclusives from HTC Vive available with pre-orders as well: Fantastic Contraption, Tiltbrush, and Job Simulator.
On the platform side, one difference between Oculus and HTC Vive was the nature of their digital distribution platforms: the Oculus Store and Steam. Steam declared itself headset agnostic and had happily supported both Vive and Rift versions of games on its online store, as well as supporting the headset in its Steamworks SDK and even allowing the tracking of Oculus Touch Controllers using SteamVR. Oculus on the other hand has kept its store exclusive to Oculus only versions of games and apps. By July, we had 2 premium consumer headsets, 2 new VR platforms. They were both good and both finding their feet among consumers.
Sony Enters VR
The Playstation VR launched in the US in October to largely positive results. It has the advantage of being a much cheaper solution than the premium HTC Vive or Oculus Rift headsets. An install base of over 40 million Playstation 4 owners only require an additional $400-500 investment to experience a very robust VR experience. Furthermore, Sony brought the games. The Sony library of complete game experiences rivaled both HTC Vive and Oculus right out of the gate. It is still very early in this product’s release, but it looks promising, and while not the premium PC VR experience, it is a major step up from the mobile headsets and very accessible to a large market.
Google and Microsoft
Google’s success with getting some massive early adoption of their cardboard headsets encouraged them to go deep on a mid-tier mobile headset and platform they dub “Daydream”. It is a lightweight headset with bluetooth capability, 3 DOF tracking using your phone, and it includes a 3 DOF remote controller allowing pointing in games and other VR experiences. On the PC high-end side of things, Microsoft surprised the VR community late this year with some VR headset announcements of their own: They announced a whole hierarchy of tethered VR headsets which will be on par with current gen premiums (Oculus and HTC Vive) in terms of visual fidelity and will not rely on external cameras for tracking, rather they will use inside-out tracking. Demonstrations of this looked promising. Furthermore, Microsoft announced that these headsets would be opensourced and developed not by them but rather by 3rd party developers. Finally they mentioned that the PC specifications would be very reasonable, well below the initial specs from HTC Vive and Oculus. It looks like the VR headset market will be getting a little more crowded next year.
Social VR took a few steps forward this year, but don’t cancel your snapchat account just yet. Notably, there were really nice initial offerings from vTime, RecRoom, Surreal and BigScreenVR all which allow players to take on an Avatar persona and to chat online with other people in VR spaces. Recently we saw a nice glimpse into the future of Social VR with the Oculus demo at OC3, where Mark Zuckerberg donned a Rift in front of an audience, connected with some of his co-workers in a virtual chat room, immersed in a 360 video surrounding of his home, where they played games, brought in a video call from his wife at her work and then took a VR photo of himself, his wife on video conference, and his dog who was in a live 360 video feed sitting on the couch at the Zuckerberg estate, and then he posted it to Facebook.
Both Unity and Unreal Game engines tried their hands at creating a development environment in VR. Between these two, the Unreal toolkit is a little further along, but both need a lot of design and improvement and neither of these seems like it has surpassed the productivity of a traditional development environment. Nonetheless, they both hold some promise for a time when we will prefer to develop with the headsets on.
Cryengine is also worth noting in the Game Engine space. While its toolkit is arguably less friendly than some of its competitors, it has put forth a couple solid VR offerings which have both been competitive and visually compelling: The Climb & Robinson: The Journey.
Out-of-Home VR Experiences
This year there have been several VR experiences only to be had on location. Samsung rigged up several rollercoasters at multiple Six Flags theme parks with Gear VR experiences. Universal hosted VR haunted houses at their theme parks. The VOID demonstrated their technology in Times Square with their Ghostbusters experience. IMAX partnered with Starbreeze to bring VR experiences to theaters. Just recently, HTC Vive began rolling out VIVE Arcades across China and has announced plans to do this in the US and Europe as well. What the future of these out-of-home experiences will be remains a mystery, but there is clearly a lot of investment and interest in this space right now.
Cutting the Wires
We have mobile wireless VR already with cardboard, Gear VR, and a hundred headsets in between which all use your phone as the screen and computer to drive the experience. These have been successful in getting a basic VR experience out to a large audience(on the order of 20 million cardboard-type headsets and over 2 million Gear VR headsets). However, for anyone who has tried one of the premium tethered experiences like the Vive or Oculus, the current mobile experience falls short in a couple key areas. Notably, it does not yet have 6DOF (six degrees of freedom), the ability to track both positional and rotational movement of the user. Phones have many sensors like gyros and accelerometers which make tracking rotational data possible, but the push lately has been to figure out the positional tracking for mobile VR. There are several solutions on the market already, we’ve seen offerings from Intel, Occipital in the past year, and Oculus just demonstrated their “Santa Cruz” headset to reporters at OC3 which uses “inside-out” tracking, similar to Microsoft’s hololens. Graphic processing is the other major drawback to current Mobile experiences, they just cannot compete with their PC-tethered counterparts. Framerate being so critical to VR, to achieve this on mobile, it requires many sacrifices to lighting, textures and geometry in a VR experience to achieve a passable experience. Mobile processors are getting better all the time, but there is still a large gap to fill to marry the PC and Mobile experiences in terms of fidelity. Wireless may hold the promise of solving both of these problems, but it remains to be seen. There are millimeter wave solutions to tracking which may allow for very accurate tracking of high end VR experiences in large open spaces, also we may bridge the gap in wireless transmission to allow for a 90 FPS experience to stream from a nearby computer allowing the user to be untethered and still using a high end GPU to drive the experience. There are several companies working on both of these solutions right now and next year will see many of them come to market.
Tap the Senses
There are a few areas of sensory involvement within VR which have seen a lot of activity this year. While using audio and visuals with some basic haptic feeback has been a mainstay within videogames for decades, VR developers are now trying to figure out ways to get at the other senses and to enhance our experience of the ones we already have access to.
On the visual spectrum, there are companies like Magic Leap and OTOY working to bring lightfield synthesis within the headsets. Lightfield Synthesis is the “other” data in the visual spectrum we’ve largely ignored in PC experiences thus far. We haven’t missed it because we haven’t experienced it yet outside the real world. It is the capturing of how light interacts in an environment, how it bounces off of objects, and what angles the photons are presented to the eye, this is critical to creating a lifelike and believable scene, one that your brain accepts as real. A company called Lytro is at the forefront of creating a 360 degree camera that captures and stitches all of this complex data. While the promise of this being included in all future VR experiences is exciting, there are still hills to climb here to optimize this massive visual data set to run in a real time environment at an acceptable framerate.
There have been many attempts to improve our sense of touch in VR this year. While last year it was the “Tesla suit” and “UltraHaptics”, this year we were treated with “Skinterface” and “Sub-Pac”, solutions which use sound to simulate physical interactions. Several companies are working on gloves and other localized haptic solutions to help enhance our sense of touch. Both the HTC Vive controller and the upcoming Oculus Touch controllers have some basic haptic vibration built into them, while this is not new, it does help with some sense of presence in VR. A company called Tactical Haptics has put together a very convincing set of controller enhancements using “strafe” haptics to give a sense of weight and tension to VR interactions.
The offerings this year for smelling in VR have not really gone much beyond the “Smell o Vision” experiments of the early 1960’s. The “Nosulus Rift” is a real thing and thanks to the creators of South Park, we can now smell farts in VR. The Feel Real “Nirvana” is another peripheral which adds smell, as well as wind and heat to your VR experience. University of Tokyo has an amazing virtual cookie experiment which can fool you into tasting flavors using VR. These efforts, while interesting, make it clear that smell has got a long way to go before it finds a home in the consumer VR market.
There are three major categories of effort in simulating movement in VR: Treadmills, Vestibular Stimulation and Real World Locomotion.
The first is the treadmill solution: products like Omni, and Infinitrak are making tethered environments that allow you to walk and run in any direction with a headset on, while remaining in a very small area. These work, and in some cases are not too bulky or expensive, but all of these solutions have limitations which reveal their tethering and there are some safety issues even with the devices that have harnesses to keep you contained.
The second solution for movement is vestibular stimulation. This is a technique which originated with electrode stimulation just behind the ear which fools your inner ear into perceiving movement. Samsung has demonstrated a means of doing this using only sound waves and a pair of headphones. Imagine simulating falling, flying, driving, earthquakes while in VR without the player requiring a lot of space or special contraptions. The promise here is huge for immersion, but we are still early on this technology, and it will be exciting to see what next year brings in this space.
The third solution is actually having you move around in a mixed-reality setting. You will walk, drive or ride somewhere all while staying in your virtual environment. To get this to a viable state involves very accurate tracking in open environments, using techniques similar to driver-less car technology, and will also require some advancements in scanning and processing technology. This is probably the furthest technology out there because of the number of dependent solutions needed to get there, but like the inevitable path the AR and VR will take in the years ahead.
Google Tango, SoftKinetic and Occipital brought depth sensing to AR and VR experiences in the past two years, but this year we are seeing the first phones to have it built-in. The technology used here is called “Time of Flight”, and it is a solution which uses light beams to establish distances of objects within the visual spectrum. LIDAR is an example of how this has been used in the past, now it is being used in real-time to scan environments, objects and people. Capable of not only capturing the shape of things, but also their colors and texturing them very realistically, this technology has the power to re-create what it sees in the visual spectrum. The value of scanning the shapes of things means that we can interact with them in VR. For example, scanning in a keyboard, or a pen would allow someone in VR, without seeing the “real” world to pick up and interact with objects in their environment without leaving VR. Also, elements within a VR experience can interact with the environment. Microsoft’s Hololens Project X-Ray demonstrated this year how gaming and software elements can interact with furniture and architecture. It will be a few years before we can scan, optimize and render everything in real-time, but the goal is there and there are many companies working towards it. Michael Abrash used the term “Augmented Virtual Reality”, in his keynote at OC3 to describe this process.
VR clearly has the ability to engage us, and it is not just the entertainment sector which is benefitting from this. Of course games and movies will be advancing heavily in VR this year, but the benefits of VR reach far beyond entertainment.
Directors like Steven Spielberg (Ready Player One) and Jon Favreau (Gnomes and Goblins) are dipping their toes into full VR movie experiences and validating this medium. Hollywood is taking this seriously. They recognize it as a new medium, one which evolves their current storytelling, venturing off into new areas which the rules are not yet defined, and one rich with new experiences to be explored.
Architects, City Planners, and anyone building anything can benefit from VR visualization of their proposed developments. Stakeholders can walk around imagined spaces alongside their architects and builders to experience large developments long before the first shovel has hit the dirt. Allowing for very realistic presence in those environments and a real sense of what the final product will be, a far cry from the conceptual drawings and physical models that have been the mainstay of visualization in this sector.
Visualization of existing spaces, like homes and commercial spaces is greatly serviced by VR. High resolution 360 photography as well as video allow for buyers to walk around homes that may be thousands of miles away, to try before they buy. This is really taking off and we will see this evolve into “the” method for home buying over the next few years.
Music, Sports and Politics
This year we saw major sporting events in VR, we could attend music concerts in the front row and backstage, and we could be on stage with political candidates while they were having a live debate. While all of these experiences had some shortcomings, they were all firsts, and experiments and all promising of a future where we have control of how we want to experience an event. The future here promises social interaction too, you will be able to go to a ballgame with your dad, who is on the other side of the country, you will be able to go to see your favorite band with all of your high school friends even though you are all now grown up and live in different states.
The health field has several areas that will benefit from VR in the near term: Diagnostic, Pain Management and Fear Management. We have yet to see much in terms of diagnostic, but the ability for your doctor to give you an exam remotely doesn’t seem too far off. Distraction has been an early success for VR in health. A good distraction might be more effective than a sedative or painkiller. VR has been used to help with chemotherapy sessions and skin stretching for burn victims, proving as or more effective than painkiller counterparts. For fear management, the simulation and presence capability of VR allow very real human reactions to virtual situation allow people to approach their fears in a safe, controlled environment. This year was saw live streamed surgery in VR, and VR is used to train surgeons too with interactive elements. Mindmaze brought us a means to help stroke victims recover in VR. Simulation in VR is also being used to help treat PTSD, the presence and realism allowing for safe environments with which to deal with the trauma of warfare. The health benefits of VR are really just starting to be tapped, and the future here is very bright.
One of the largest umbrellas of VR potential is in training. Simulating scenarios, machines, tools, environments and people all propel VR ahead of traditional in-person or video based training solutions. While the fidelity of these experiences and quality of the instructional design are subject to vary based on the developer, this is one area that VR can safely say it has a handle on for the future. Whether you want to fix a car, repair a furnace, solve a personnel problem, learn a culture, VR has the ability to engage learners at a level never before possible. The training experiences to date have been lackluster, but there are few who doubt this will evolve into the medium for training and instruction in the future.
So with all of this amazing progress in the VR space, where are the jobs? In the next few years the major jobs will be leveraged from the gaming industry, many of the skills needed to make VR experiences are already present in video game development. Modeling, Rigging, Animation, Interface Design, Graphics, Real Time Processing, Control and Networking are among the core competencies of game development which translate immediately into VR development of all sorts. Content creation within this space will be an order of magnitude larger than we have seen in the video game industry, so while game developers are a good starting point, the demand for content across all sectors will require a new generation of developers to learn these skills for more than just gaming to make the environments needed for training, simulation, gaming, movies, health and every other conceivable VR use case.
Movies are the other area of massive VR job growth. The skills of Hollywood, from Cinematographers and Directors to Actors and special effects experts, all of these skills are needed to create compelling stories in VR. There are efforts for direct translation of Hollywood methods, such as those by Uncorporeal, whose aim is to get 3D film of actors into CG environments. Their efforts in “Fluffy” and “Alcatraz Island Lofts” show the emotional value of real human actors in VR environments. As experimentation progresses, we will start to see more and more synthesis of traditional Hollywood and their CG counterparts. One company Baobab studios, founded by former Pixar and Hollywood talent, put together a great short “Invasion” showcasing good seated VR storytelling, 20th Century Fox took notice and has invested heavily in them. This is just a single example, but there are dozens of studios and teams working on the next generation of VR movies and the jobs in this sector will make the current hollywood blockbuster credits look short by comparison.
Capturing 360 video in 8K resolution per camera is something that is coming. Current Offerings like the GoPro 18 camera rigs, alongside the all-in-one Nokia Ozo are creating a new breed of filmmaker for VR and the rules are yet to be written on how to film, what to film or even how to present that to a viewer. These skills will also draw heavily from Hollywood, but require a new lens and a new generation to explore and show us what VR film can be for Movies, News, Sports and Music.
As all of these areas of VR are explored, there are people all over the world which will soon be capable of working on projects from wherever they are. Collaborative VR development and distributed teams are something many developers are embracing. The Unity Game Engine, for example, has two services in this vein: “Collaborate” and “Connect” which aim to get developers from opposite ends of the globe working on the same project together in the same environment, and also offering up “yelp” style job postings and developer databases to have people bid on freelance projects from anywhere in the world.
We are into our first year of VR, and there have been a lot of stumbles, but those have been outshadowed by the successes and ultimately the fervor of what is to come. We know there will be better headsets and better experiences in the not too distant future, and anyone who has tried some of the premium offerings in VR this year already has a good sense of where we are headed, strap in, it’s going to be a fun ride.