The Khronos Group recently launched an initiative to standardise the way VR applications access the many available hardware platforms that have arrived on the scene over the last couple of years. On its website Khronos identifies industry demand as driving the initiative and judging by the number of companies that have added their names to the announcement and the industry leaders who have supplied quotes, that is no exaggeration. Khronos supplies a graphic to show the extent of support:
And a selection of quotes will give a flavor of just how enthusiastic the industry is for this standard to come into being.
“With VR on the verge of rapid growth across all of the major platform families, this new Khronos open standards initiative is very timely. We at Epic Games will wholeheartedly contribute to the effort, and we'll adopt and support the resulting API in Unreal Engine,” Tim Sweeney, founder & CEO, Epic Games.
“Khronos’ open APIs have been immensely valuable to the industry, balancing the forces of differentiation and innovation against gratuitous vendor incompatibility. As virtual reality matures and the essential capabilities become clear in practice, a cooperatively developed open standard API is a natural and important milestone. Oculus is happy to contribute to this effort,” John Carmack, CTO, Oculus VR.
“Open standards which allow developers to more easily create compelling, cross platform experiences will help bring the magic of VR to everyone. We look forward to working with our industry colleagues on this initiative,” Mike Jazayeri, director product management, Google VR.
There are more (and you can see them in full on the announcement page) but just those three cover a huge chunk of the industry, and include three companies that are critically important to the success of this move.
2017 is being billed as a crucial year for the VR industry, with headsets and software at last coming together in a way that might just provide a breakthrough - although in my opinion the jury hasn't even left the room yet. There are rumours of new headsets, new technologies including eye tracking with foveated rendering and optical navigation systems which require new sensor technologies and new software to make use of them so standardisation can only help. A second graphic from Khronos helps to explain the problem and their proposed solution:
Their basic point is that the market is fragmented wth multiple proprietary runtimes and driver interfaces and that this impedes the creation of widespread VR experiences that can easily run across multiple platforms. Developers can only afford to support a limited number of platforms so fragmentation leads to less content choice for consumers, and slower VR market growth. Khronos’ proposed standard will include cross-platform APIs for the various sensors, tracking devices, controllers and displays that go into a VR system, solving that problem and stimulating growth of a VR software ecosystem.
This is not the first time that Khronos has moved quickly to head off market fragmentation in the face of rapid innovation; the timely introduction of the Vulkan graphics API last year and the swift move to provide API and file format support for deployment of deep learning based vision systems are two other recent examples of this responsiveness.
This is unusual in a collaborative industry body of this type and is a testament to the organisation and its members. A standard like this can only help, and if the adoption is anything like as widespread as Vulkan, it could turn out to be one of the crucial factors in making this market work.
Augmented reality is usually seen as a successor technology to virtual reality, a different application with a different (and harder) set of problems to solve which will come into its own in a few years.
VR is here today, the problems are being rapidly knocked down, billions of dollars are being invested and there is a huge user base just waiting for the hardware to cross that availability threshold and Bam! the whole thing is going to explode.
That’s the buzz, anyway but there are a number of measured arguments being made which cast serious doubt on the viability of VR as a mass consumer technology for today, and there are good reasons to expect that the problems with VR can only be solved by merging it with AR.
Most of the arguments predicting limited success for the current crop of VR implementations are non-technical, which makes sense now that many of the technical barriers which doomed previous generations of VR are being rapidly demolished.
We have now had plenty of opportunity to try out a whole range of headsets and it is clear that the classic problems: field of view, lens correction, responsiveness, are either solved or well on the way.
Issues like user isolation, the difficulty of mapping established interactive paradigms onto VR and the problem of providing an equivalent visual experience have not gone away, or even really been addressed in a meaningful way and are the focus of these arguments.
And then there’s the headset.
Those last two issues: equivalent visual experience to the best of what is already available and the need for a headset, were the twin nemeses of 3D TV. VR boosters bristle at the mention of that epic failure, arguing that these are two different things but that is not the point. People are the same, which is what matters, and visual quality is important to them so we do need to pay attention to the lessons learned from that failure.
3D TV has the problem that, because of the need to deliver a different image to each eye, the visual quality is degraded by comparison with a standard video. This in itself has been enough to dissuade many viewers but gamers are more resilient in that regard than general consumers so why is it still a problem for VR? The answer lies in the extreme nature of the requirements and the gap between where graphics hardware is today and where it needs to be to deliver a true, no compromise experience.
The driving constraint of VR is latency, what is referred to as motion–to-photon delay, so that in addition to delivering a left/right pair, the system must also meet a very tight deadline for getting the pixels on the screen. From sensing a motion of the head, modern systems aim to get the resulting image on screen within a frame time and they aim to do it consistently, without the variations which are acceptable in a standard gaming system.
Standard game systems observe the twitch limit, the tenth of a second or so which represents the limit of human reaction time. This means that the various stages of the game can be pipelined: each stage must stay within the target frame rate but it’s OK to have the final displayed image several frames behind, as long as the overall delay stays within the twitch limit.
VR systems don’t have that luxury. Everything, including responding to the sensor plus animating the scene and then rendering it must happen an order of magnitude faster, with a corresponding reduction in the sophistication of the models and quality of the rendering. This is not a trivial point: it means that GPU performance must increase by 10 before the visual quality of today’s PC is available on a VR system and according to one study the last order-of-magnitude increase took eight years to happen. There’s a reason the best looking VR games are mostly static, and this is it.
VR has to offer a compelling reason, even to gamers, if they are to adopt it. Immersion is not enough: game play and realism are at least as important. If these are substandard then you are just immersed in a crappy game. Fortunately, VR has much more to offer than visual immersion so we are seeing games like “The Climb” from Crytek which tries to leverage that but in fact has been more useful in exposing more of the shortcomings of current equipment. Which brings us back to the headset.
If polarized glasses were an issue for 3D TV, the isolation and discomfort of a headset is the same issue in spades for VR.
I believe the hardware manufacturers when they say that the enormous sums being spent on design of lighter and more ergonomic headsets will result in devices no more inconvenient than a pair of sunglasses. But that only gets us to the same status as 3D glasses, which is still a problem, and it overlooks the real issue which is isolation. Isolation from the real environment and isolation from real people.
Once in a VR world, the user is blind to the real world, and now that we are starting to see apps and demos which encourage moving around, the disadvantages of that are starting to become obvious.
A world which gives no indication of its extent is not a comfortable place to be. Add a second VR user to the same space and the problem gets even worse; as long as the headsets are unaware of each other’s positions, there is the potential for collisions. This is a very poor user experience, no matter how compelling the immersion feels otherwise.
Without a solution to these problems, VR remains a solo activity conducted while seated or constrained to very limited movement (Google’s Tilt Brush app works nicely within those constraints but is hardly the basis for mass adoption of the technology). As long as that is the case, its consumer appeal will never cross over from enthusiast to mainstream.
Now, I am not going to argue that VR will fail completely but I am convinced that mass deployment is much further away than a lot of the hype would suggest. This is because the required technologies are coming from AR and are in a much earlier phase of development, which puts back the adoption curve by several years.
To be fair to the VR hardware companies, they are aware of the need for positioning and they have come out with systems like HTC’s Lighthouse to address it. The problem is that these are partial solutions and positioning is not the only problem.
Vision based systems can provide robust solutions to a lot of the problems outlined above, and we are starting to see VR headsets which include forward facing cameras (as well as internal, eye tracking cameras – an approach to solving the GPU performance problem).
With a camera the headset can autonomously map its environment, including stationary and moving objects such as other users, and can also eliminate the need for handheld controllers via gesture control.
More importantly, vision can convincingly locate the user within the virtual world in a way that is not possible today. The spooky disembodied hands in “The Climb” can be correctly connected to the user’s body, giving a feeling of immersion that goes well beyond the visual.
The point at which that becomes possible is the point a compelling user experience has been created. What makes it possible is vision; until now seen as the domain of augmented reality.
The downsides of including vision are that it adds expense, for the sensors plus a new vision-specific processing unit, as well as delay. Google’s Tango project plus recent events in the auto industry have shown us how much more there is to do on the software side but my opinion is that the benefits outweigh the costs, especially since I am convinced that current generation ‘blind’ headsets will soon be regarded as incomplete products.
So my conclusion is that we are not there yet for VR. It will take another hardware cycle to get the equipment right, and the succesful products will leverage a whole new set of skills coming from a different set of companies.
We’re in the build up to Siggraph now; only a couple of weeks to go before the grandaddy of all graphics conferences gets under way in Anaheim and I’ll be there as always to take it all in. What will I be looking for this year? well, the thing which is impossible to miss is the continued excitement around VR.
Now, I’m the first to be sceptical about VR as a mainstream consumer technology; I simply can’t see legions of people wearing headsets for substantial periods of time. That has been tried multiple times and even when the headsets are reduced to the size of a pair of glasses – think 3D TV – it was unsuccessful. It is a fact that people simply don’t like to have to wear those things but, given enough incentive, they will make the effort which is why the important thing is the content.
If enough compelling content can be created then we can throw out the lessons learned from the failure of 3D TV and movies, as well as the experience of previous generations of VR all of which failed to produce a compelling reason to don that headset.
That’s why, even though I will be dutifully making the rounds of the hardware vendors and learning about their latest gizmos, I will be especially interested in the content creation tools and how they are shaping the development of that all-important content. I’ll be looking at Quill from Oculus and hopefully talking to its designer and I’ll also be looking at technologies for 3D printing and health care delivery.