Visualise The World

Last week’s news that Apple is ending its reliance on Imagination Technologies’ PowerVR GPU designs  for all iOS devices sent IMG’s stock price into a tail spin and has many people questioning the company’s ability to survive the shock. There seems to be an emerging consensus that Apple was planning this all along and, now that they have put their dastardly plan into action, Imagination is doomed; I subscribe to neither of these theories (you can see my reasoning in this blog post) and in fact, there are upsides to the situation that could result in some interesting benefits for mobile graphics as a whole.

The Apple contract has always been a doubled-edged sword for Imagination: for a company whose business depended on being way out ahead of the market, the constant pressure to put the brakes on and stay in line with the commercial realities of today has sapped their drive to innovate and resulted in a much more conservative approach to technology development. Freed from constraint, can they now start to deliver on some of the technology ambitions they had in the past? The answer is that they might, and they have been laying the groundwork for that for some time.

Imagination is sitting on some truly pioneering technology that they acquired when they bought Caustic Graphics a few years back. Real time ray tracing has been a dream of graphics hardware designers for decades now but the computing power to handle such a heavy and incoherent task has always lagged behind what was needed to compete with the ‘fake it and ship it” approach of raster graphics. This is for very good reasons, handling the heavy workloads in chunks that have been localized into triangles results in the kind of computational efficiencies that make GPUs practicable. All the quality and workflow trade-offs needed to preserve that optimization have been seen as worth it and it’s hard to argue with that point of view when you see the results.

The thing is, there is a lot of raytracing already being used to implement dynamic lighting and shadowing, it’s just being done on the CPU or as offline compute loads on the GPU, then baked and textured onto the rasterized graphics. There are huge quality, workflow and performance benefits to being able to handle that directly in the rendering pipeline and Imagination knows how to do that.

For mobile graphics in particular, performance within a strict power budget is the key to success and Imagination has demonstrated very advanced lighting and shadows at double the performance and half the power of the equivalent rasterizing technique. Even better, they have shown it running from within the immediate mode programming paradigm of current graphics APIs. Real time ray tracing AND OpenGL – game developer’s heaven! Also, as I reported in this article the technique is being quietly introduced into the major game engines. All they need is a deployed platform to turn it on.

This could be a major win for mobile graphics, and a major boost to indie game developers who will find their production budgets stretching much further with a whole asset creation step removed from the chain. It will take some determination from Imagination plus a mobile chip company with the nerve and ambition to drive the industry into a new era. I don’t know if these two things will come together but I know it’s possible and I certainly hope it happens.

Today we got our first in-detail look at the internals of Google’s Tensor Processing Unit, the hardware inference accelerator introduced last year but whose details have been kept under wraps until now.

The design team behind the Chip, led by Norman Jouppi, have released a paper that will be presented at the International Symposium on Computer Architecture this coming June in Toronto and, while the reveal isn’t complete (the paper at one point refers the reader to a thicket of patent filings for further details, how’s that for a sense of humor?) there is enough to get a good sense of the chip.

The basic design philosophy is simple and direct: Neural nets don’t need fancy scheduling and inference doesn’t need floating point so junk all of that and jam enough multipliers and memory onto a chip to handle the biggest inference model you can think of without needing to go back to the CPU. This diagram (shamelessly lifted from the paper) shows what they did to achieve that end.

The heart of the die is a systolic array of 64K (216) 8-bit multiplier-accumulators connected to 4096, 256-element, 32-bit accumulators and a 24Mbyte memory array. The MAC array is set up to produce 256 results per cycle, with a pipeline delay that is dependent on the instruction (matmul or conv) being executed. A matrix operation takes in variable size B*256 input and multiplies it by a 256x256 constant weight input, taking B pipelined cycles to complete. 16x8 (or 8x16) multiplies run at half speed; 16x16 drops to one quarter. Interconnect widths between blocks are 256 bytes in order to keep the array busy and the results flowing back to memory.

The chip performs two basic operations: Matrix multiply/convolution and the various flavors of activate; the rest is moving data in and out of memory and keeping the multipliers busy. It acts as a coprocessor, being fed by the CPU (rather than seeking its own work like a GPU) and is programmed using Tensorflow.

Google doesn’t give a die size, although it does reveal that the chip uses a 28nm technology and is less than half the size of a Haswell E5 2699 V3 die, although there is no saying how much less. The claimed performance for the TPU running at 700MHz is 92 TOPS for 8-bit operands while dissipating 75W.

Naturally, the benchmarks quoted show the TPU blowing away both the CPU and GPU for both outright performance and performance per watt.

The paper finishes up with a discussion of where TPU design might go in the future, concluding that higher performance is all about memory bandwidth. Interesting.  

The news this week that Apple intends to start using its own GPUs starting in eighteen month’s time has left a lot of people wondering if IMG can survive this latest blow. Last year, remember, IMG cut loose a lot of employees, including CEO Hossein Yassaie (and myself) and then posted a huge loss before selling off chunks of its business, such as the Pure digital radio business.

As usual, there is a lot of hyperventilation from people who you might expect to have steadier nerves but this is a defining moment for IMG, which has never really found itself after the ill-advised acquisition of MIPS back in 2013. Apple makes up about half of IMG’s royalty revenue and it would be difficult for any company to recover from the sudden removal of that sort of income but that’s not what is happening. Let’s look into the details a little.

First some background; Apple has been hiring GPU architects and designers for years now. They snagged some big names, such as Bob Drebin and Raja Koduri from AMD way back in 2009, and there have been persistent rumors (although not from people who know anything about the subject) that Apple has been busily designing its own GPUs all these years. There’s no doubt that they are capable of doing such a thing and have now announced they will be ready within a couple of years to do just that but, 8 years later and some of the best GPU architects on the planet are only just starting to produce results? Not very likely.  In fact, Koduri went back to AMD in 2013 - with a great job offer, no doubt - but also because I believe he was not getting to design GPUs at Apple.

The role of Drebin and company seems to have been to drive the GPU suppliers of choice, Intel and IMG, to continue to excel at their jobs, and to ensure that the GPUs they produced were properly tailored to Apple’s needs. (I believe this was one of the sources of IMG’s problems; more on that later). Although they do seem to have been adding functions adjacent to graphics, Apple were taking IMG drivers and using them unmodified, certainly until last year and probably still to this day. That fact alone should put a stake through the heart of any argument that they have been using non-PowerVR graphics (or whatever it is you need to do to kill off zombie ideas that refuse to lie down).

So, something happened fairly recently that caused Apple to decide they needed to get into the GPU business themselves and that event is fairly obvious. Things weren’t going well at IMG for several years and it is clear that numbers of key employees were looking for places to go. It’s inconceivable that they didn’t approach Apple (in fact a key senior person did join Apple in late 2015) and then last year the  IMG board pushed out a CEO that Apple felt they could rely on. Two and a half years from design start to chip integration for a new architecture GPU sounds about right to me: that right there – April 2016 - is when Apple kicked off their GPU design project.

Oh, by the way, if you ask me what all those engineers and architects have been doing in the meantime? Building vision accelerators, of course! In common with almost everyone else, they see that as the prime strategic investment and held off diverting resource from it for as long as possible until IMG’s problems last year forced their hand. This also gives the lie to the narrative of Apple as the faithless monster; I’m sure they are as ruthless as any other business but no more so and not in this case.

So, what now for IMG? All that guff about patents is just hot air; sure, they have lots of patents but with at least a dozen other GPU companies successfully negotiating the so-called patent minefield, to suggest that Apple is incapable of doing the same is just laughable. That really doesn’t matter, though because they are not going to implode and with a steady hand can turn this around – although I do think that acquisition in about three years is the best they can aim for.

First off, Apple revenues will continue unchanged for the eighteen months it will take them to introduce their own device and there will be a long tail of royalty-producing products stretching out after that. The only immediate effect will be the loss of the annual license fee (and they might even be able to protect that) so IMG need do absolutely nothing right now and in fact, if they have the nerve, they can seize the moment.

Remember I said earlier that being driven by Apple has been a huge problem? Always being pulled to service their biggest customer, always being pushed to design GPUs that satisfy Apple‘s needs, and to design them first, has been a huge burden for the company. IMG’s business had always been run on a relationship basis and this suffered under the stress of working for Apple. Multiple opportunities were sacrificed in this way.

Free of that burden, IMG can now refocus their GPU design effort on products the rest of the market wants and also take the enormous engineering resource sucked up by Apple support and put it into vision - the strategic investment they have consistently failed to address.  

There are a few other things the company needs to do, such as finish cutting loose businesses they have no chance of succeeding at but overall, I don’t see them as being dead yet. The big unknown is the management: this is not a time for caution. IMG is back into startup mode and this is when big ideas and big risks come into play.  It could be there are too many suits and not enough rolled-up sleeves.

GDC last week was stiff with VR headsets and other virtual - and augmented – reality equipment, some of which I have reported on already but there was one interesting item I have had to keep under wraps until today’s official launch date.

The PowerVR team at Imagination briefed me on ‘Furian’ which they are calling the first new PowerVR architecture since Rogue was launched in 2010 and there is enough there to make this a very interesting event indeed. Imagination is aiming this new generation explicitly at the 7nm process node with VR, ADAS and notepad PCs in mind.

Where the Rogue architecture was designed for a 28nm process shooting for 60FPS gaming on 2MPixel screens, Furian* is looking for 120FPS on a 4K screen; capable enough to drive a low latency VR setup in a mobile configuration. In support of this, Imagination tells me that they have basically doubled everything: twice the fillrate and twice the SIMD width per cluster, which doesn’t get us to the 8x performance boost implied by the new frame rate and resolution targets but they assure me that with the improved scalability and performance density, that sort of performance is now achievable by adding more clusters within the power limits.

There are a lot of changes just to get the improved graphics performance (more of which elsewhere) but purely from the vision processing point of view, there are some intriguing developments. One of the more interesting is that down in the guts of the cluster where all the compute gets done, Imagination has split out what was a unified cache into separate texture, instruction and data cache memories, allowing them to take advantage of data memory banking to implement a scatter-gather scheme similar to that found on vision-specific DSPs. The rogue architecture was already able to implement lane swizzling and shifting, so this new feature brings Furian into a much more direct feature comparison with those cores.

The base ALU has also changed. Where Rogue has all its instructions going through a single path containing multiple arithmetic units as well as logic and other functions, Furian splits these out into an arithmetic-only path with a separate parallel path for everything else. Imagination does not claim that this path is simultaneous issue, so we’re not looking at some sort of hybrid SIMD/VLIW architecture but given that this secondary path is used for multi-cycle instructions, some degree of parallelism may be available and the simplified structures will certainly lend themselves to higher clock rates and/or lower power operation.

Vision algorithms typically work on fixed point and integer operands so any GPU that plans to compete needs to take that into account. The PowerVR series 7XT GPUs were equipped with separate integer ALUs operating in Vecn mode depending on whether 32, 16 or 8-bit operands were chosen and the launch materials for Furian specify that there is a Vec2 mode for FP16 operands but there is nothing on integer resources. Privately, Imagination has assured me that Furian inherits “some aspects” of series 7XT integer handling but without giving details. That’s a loose end that will need to be tidied up before we can really say how this generation will perform as a vision processor.  

Imagination has been going through some tough times recently. They are still holding on to their prized position as sole GPU supplier to iOS products but that is both a blessing and a curse as the rest of the market gradually slips away to competitors with greater freedom of action. Will the new Furian architecture help them regain their lost market share? With projected headset sales in the single digit millions,  VR is not going to do it for them even if they manage to deploy a ray tracing version but vision functions in mobile and automotive will only get more important and Imagination have traditionally had loyal support from TI, Renesas and Socionext.

According to Imagination, multiple licenses have already been taken and the designs delivered to customers so we should be seeing product announcements within a year. As usual, there is no hint as to who these customers might be but given that this is a high end XT architecture that really doesn’t play in the mid-range handset market of their usual customers, we should expect something from one of those automotive players.

*Yes, the name is something to do with Vin Diesel but for the sake of decency, I shall draw a veil over the mental process that linked Riddick to this GPU

 If you are as keen on ray tracing as I am, a couple of very interesting things have happened over the last few months but with so little fanfare that you would be forgiven for missing them.  Both of these things have their origin in Imagination Technologies' efforts to deploy their hardware accelerated ray tracing technology so I'll quote a from a 28th February blog post and March 1st press release; first the blog post:

The demo ... at our GDC 17 booth this year is an expanded version of one of the regular (Unreal) engine tutorial scenes. The engine can be switched between raster mode and full ray traced mode on the fly. While in ray traced mode, one can then select between raster and ray traced shadows and probed and ray traced reflections, again on the fly...we demonstrate the flexibility of ray tracing with respect to alternative camera models with a ... render mode, which renders the scene using a true 360-degree spherical lens in a single render.

Future work will include dynamic geometry, support for point lights, and a reimplementation of our hybrid single-ray soft shadow algorithm.

Which is interesting enough in itself, but there is more:

With the upcoming Unity 5 lightmapping feature that uses PowerVR ray tracing for pre-baked lighting, developers and artists can use our ray tracing technology to get unbiased physically-based global Illumination (GI) and interactive feedback on any changes they make to the scene, including materials, geometry or lights.

In other words, there is a project to provide UE devs with a fully ray traced rendering path that can take advantage of hardware accelerated ray tracing on a mobile GPU (check out Imagination's previous work on this). And for Unity shops, there is a full, commercial grade ray tracer available for production of light maps. These seem like two different things but the common factor is that they both use Imagination's ray tracing extensions to Khronos rendering APIs. How long would it take to switch out the software renderer and add the hardware assisted version in Unity? Imagination claims that the UE mods they have made play completely transparently with the standard raster based renderer; what's the downside for Epic to deploy something like that?

Now that we have seen a mobile GPU with better than 150 million rays per second, the common objection whenever the idea of ray tracing comes up is that it's impossible to break the chicken and egg situation. Lighting is a solved problem, even if its all a kludge and artists despise it, so there is no incentive for GPU makers to change their hardware; without hardware, why would devs write for it? Who's going to square that circle?

It's almost two years since Imagination rolled out Gandalf, their first serious attempt at this, and at the time they were predicting a vastly upgraded version within this timeframe. Since Moore's law is still ticking along nicely in defiance of all the doomsayers, the cost of 150 million rays is almost literally nothing on a 7nm process so it can't be long before adding this as a differentiator becomes a no-brainer.

With the two big engines quietly acquiring the capability to turn on ray tracing in anticipation of hardware showing up, and the cost/performance of significant amounts of ray tracing moving very quickly in the right direction, that circle is starting to look pretty square.

Full story »