Video Compositing on the iPhone: OpenGL with alpha mask on top of a full motion video

During the iPhone game project I have written about before we had to implement a highly dynamic 3D scene in great detail here at itemis. From a third-person perspective, the player looks at a semi-transparent ball rolling through a gully towards a waterfall. To implement this with the limited resources of the mobile device we came up with an adaption of the rear projection technique as known from film productions.

Layer compositing

The game screen has been built with a full motion video behind the HUD and the game layer.

The game screen as seen by the player internally consists of different layers. The top-most layer contains 2D graphics and displays information such as speed or remaining time. Behind this layer (or plate) the dynamic 3D game scene is being rendered. This layer contains the actual game elements the player controls. It has been built with OpenGL and displays the player’s character inside a rolling ball.

Now, in addition to this quite common setup we implemented a third layer in the back that plays a full motion video of a pre-rendered game scene. Even though this technique had been used for games such as Rebel Assault on the PC between the late 80′s and the early 90′s I am not aware of any iPhone game that integrates the game layer as tight as we did with matting and alpha compositing.

Game elements (a) with animated matting (b) appear inside (c) or even "behind" a background movie as illustrated on (d).

In no case we wanted to deliver a separate, affixed game layer above the movie. Instead, we tried to create the impression of embedded game elements (a) as an integrated part of the pre-rendered movie. With the help of an animated alpha mask (holdout matte) that corresponds to the wave movements of the river (b) we were able to let the water appear above the ball (c). And by using a matte with a true alpha channel we were even able to let submerged parts of the scene shine through the water surface as you can see on (d).

NIVEA FOR MEN WaterfallRace is available on the App Store for free and I will write another post including a screencast once the scheduled update for known bugs is out.

We were able to make this compositing feature work on the 1st generation iPod Touch and iPhone with the SDK 3.0. Please let me know if you are interested in the technical details on how to configure the EAGLView and MPMoviePlayerController or how to sync with the movie on the targeted iPhone OS 3.1. Also, please comment on this post if there’s another development aspect of this project you want to learn more about.

Support my Work

Writing an article like the one you have just read takes me quite an amount of my personal time. Way too often, I invest this time in different interests and decide against another blog post. On the other hand, you can motivate me with your feedback, your thoughts and your ideas. Please leave a comment below or flattr this post if you think it's worth it.

Comments

  1. [...] Update: Read the follow-up post about the NIVEA FOR MEN WaterfallRace. Different functions to separate actual and displayed speed values [...]

  2. Ash says:

    I had a look at your Nivea Waterfall for Men app, and saw that you were able to get a holdout matte to make the water appear to be in front of the rolling ball. I am working on a App for in which a person appears as a cutout figure in a movie with a black background, we would like to have some sprites appear to float around and behind him. Up to now I have been unable to get any kind of alpha or chroma key to work with the video; and believe it’s actually not possible.

    Thank you very much!

  3. Hey Ash,

    I am glad I could inspire you with my blog post about video composition. In general, you can place any UIView on top of the rendering view of a MPMoviePlayerController as you can read on different online tutorials. If your particular UIView contains transparent areas the content of the movie shines through.

    What you want to achieve is the opposite, though. Parts of your movie should be rendered transparently based on its content (chroma keying). I share your opinion that this is currently not possible. Neither can you place a UIView behind the movie (until SDK 3.2) nor can do you have live access to the frames. What you could do though, is storing the individual frames as sprites if this fits in memory (you can render thumbnails of random frames to UIImages, since 3.2).

    Hope this helps.

    Cheers,
    Heiko

  4. Ash says:

    Hi Heiko,

    Thanks very much for your reply. You summarised what I would like to do exactly.

    So if I understand correctly, to achieve your affect of having the water flow in front of the sphere, you took random frames of the playing movie (using thumbnailImageAtTime:timeOption) which returned a UIImage; which you then convert to a mask and put in a UIView. You then convert the UIView to a EAGLView? This is a very useful tip!

    We are using Cocos2D because we also have sprites to draw in another layer. Do you add the mask layer as a node to the sprite layer? How do you blend the two layers to mask the sprites? I am sorry for the rudimentary questions – I am new to OpenGL programming.

    Thank you again for your invaluable help.

    Best regards,
    Ash

  5. Hey Ash,

    The rolling ball loop has been rendered as movie plus mask by our artists and we converted it to 192 transparent pngs offline. We then created 3 atlases of 64 frames each loaded as textures of the size 1024×1024. Using thumbnailimageAtTime was just an idea you might follow to do this online – directly in your app.

    To apply masks with OpenGL ES 1.1 (this applies to cocos2D) you can use FBOs to change the alpha channel in a second step after you had rendered to them. Cocos2d supports this with CCRenderTexture directly.

    Cheers,
    Heiko

  6. James Norton says:

    I am interested in how you set up the EAGLView to transparently overlay the MPMoviePlayerController view.

    Thanks,

    James

  7. [...] want to keep to the budget. Also, if you carefully look on page 31 you might notice the featured NIVEA Waterfall Race I wrote about before from a technical perspective. mobile business 03.11 – you can read my German article "mobile [...]

  8. Pedro says:

    Hello, I have read all your post and I am interested in doing something similar.

    I would like to have an alpha mask over a video recording, and when you finish recording, then the final video has this mask over. Do you know what I mean?

    But I have being looking for it and I have not found nothing.

    Please could you help me?

    I leave you my email.

    thaks a lot!

  9. Perdo, this has to be done in a different way. To modify frames of a video and to render it onto disk, have a look at the AV Foundation Framework and on assets speficially.

Leave a Reply