How does AirPlay mirroring work?

One of the exciting features coming in iOS 5 is AirPlay mirroring which lets iPhone and iPad owners send sound and picture to an Apple TV wirelessly. But how does it actually work? With cleverness and magic.

Disclaimer: Outlandish Ideas has no insider information on this. Nor have we reverse engineered the protocol. This technical explanation is gathered from research, common sense and some informed guesswork.

My first thought when I saw a demo was that Apple had worked out a way to stream full screen uncompressed video over a wireless connection. But that’s clearly not the case. A competing technology, WirelessHD, does exactly that but it requires direct line of sight between sender and receiver and uses the 60GHz spectrum to achieve the necessary transmission rates of over 4Gbit/s.

AirPlay is more subtle than that. One clue comes from the fact that it is only possible to mirror displays between iPads, iPhones and (2nd generation) Apple TVs all of which run the same iOS operating system. That means they are capable of running exactly the same apps even if installing apps on Apple TVs is normally not permitted.

What you’re actually seeing in the video above (not ours, by the way) is the iPad desktop running on the Apple TV in parallel with the iPad. It’s not sending the video frames acros the air, it’s just sending the touch gestures and some extra stuff to keep the picture in sync. When you see it load a web page, it’s just sending the HTML over the air and rendering it on the Apple TV. When you run an app, it sends the app over the air and runs it on the Apple TV hardware. When you play a 3D game, it renders each frame independently on both the iPad and the Apple TV. When you watch an HD movie, it sends the compressed video stream, buffers for a few seconds, then plays it on both devices.

The exception to this is the camera app where the raw image data has to be sent over the air since it’s coming live from the iPad camera. And in the demo above (just over 1 minute in) there’s a very visible lag and the frame rate goes way down, as you would expect.

This is clever stuff. And it explains why AirPlay mirroring is not available on Mac laptops: the Apple TV is just not capable or running Mac OS applications.

It will be interesting to see what happens when there’s a mismatch between the processor speed of the iPad/iPhone and the Apple TV. Will it still be possible to synchronise the running of apps between an iPad 3, 4, 5 gen and the current 2nd generation Apple TV with its 1GHz A4 processor? And will the Apple TV eventually be upgraded to allow mirroring some (or all?) Mac OS applications? Time will tell.

What’s clear is that the AirPlay interaction model of sending content from a portable device to a TV is much better than the flawed idea of installing and running apps on your TV.