In July, 2000 we started a half-year project "Advanced 3D rendering technology" with Samsung Advanced Institute of Technology (http://www.sait.samsung.co.kr), with the task of preliminary research into image-based representations (IBR) for 3D graphics and animation.

The project was a success, and was followed by a one year project (from April, 2001 to April, 2002) "Advanced Methods of 3D Rendering and Animation", devoted to development of several IBR formats and their implementation as MPEG-4 nodes for the Animation Framework eXtension (AFX) of the new version of MPEG-4 (http://mpeg.telecomitalialab.com/ ) international standard. This lead to a family of 3D still and animated formats unified under the name DIBR (Depth Image-Based Representations), currently undergoing the formal process of acceptance into MPEG-4. The third project on the same theme is in progress since July, 2002.

Depth image-based model of 3D object consist of a set of images ('photographs') of the object taken from several viewpoints so that to cover its visible surface, each image accompanied by a depth map, i.e. set of distances from pixels to the object surface. This representation is illustrated in the figure below. Gray scale images are depth maps.

This main idea can be generalized in various ways. For example, images and depth-maps may be multivalued, or depth maps may be unified into a single octree structure. Both these options were implemented in our DIBR formats.

• image-based formats for still and animated 3D objects
  • Depth Image (DI), a union of arbitrary number of Simple Textures (ST), where Simple Texture is a single pair 'image + depth map'
  • Point Texture (PT), a multivalued image with corresponding depth map, obtained by projection of the object onto a single plane (this format is known as Layered Depth Image, LDI, in the literature)
  • Binary Volumetric Octree (BVO), which consists of octree-represented union of depth maps, together with a set of reference images.

  DI and BVO have animated versions. In animated DI, images and depth maps are replaced with proper videostreams. In animated BVO, images are replaced by videos, and a single additional stream of binary trees is introduced. Unification mechanisms of MPEG-4 allow to combine formats of different types, providing a great flexibility for optimal representation of an object.

• new lossless compression method for the BVO representation
  • the method is based on adaptive arithmetic coding with multiple context tables
  • usage of the orthogonal invariance assumptions allows to compress already very compact binary linkless octree structure about 1.5 - 2 times
• simple and effective rendering techniques were developed for the DIBR formats
  • splats (small color patches) of adaptively chosen size are used as rendering primitives
  • OpenGL-based implementation allows to make use of hardware accelerator at rendering stage
  • real-time rendering speed is provided for still and animated objects

For the detailed description of the formats, compression and rendering methods, design of nodes and volume/speed characteristics, see publications below.

• Y. Bayakovski, L. Levkovich-Maslyuk, A. Ignatenko, A. Konushin, D. Timasov, A. Zhirkov, Mahnjin Han, In Kyu Park, "Depth Image-based Representations for Static and Animated 3D Objects", accepted for ICIP'2002 (IEEE International Conference on Image Processing, http://www.icip2002.com )
  .pdf (618kb)
• A. Zhirkov, "Binary Volumetric Octree Representation for Image Based Rendering", Graphicon 2001 (http://www.graphicon.ru)
  .pdf (129kb) .ps.gz (122kb)

• Leonid Levkovich-Maslyuk (Senior Scientist, The Keldysh Institute of Applied Mathematics RAS)
  (levkovl@spp.keldysh.ru)

• Alexey Ignatenko (currently PhD student):
  Depth Image format, rendering, nodes specifications
• Anton Konushin (currently PhD student):
  DIBR models construction, nodes and streams specifications, rendering
• Dmitry Timasov (currently postgraduate)
  MPEG-4 reference software, specifications, rendering
• Alexander Zhirkov (currently PhD student):
  BVO format, compression, rendering

Collaborators at the SAIT side: Mahnjin Han (Multimedia Lab, SAIT , Co-Chair of Ad Hoc Group on AFX PDAM/VM editing and Core Experiments) and In Kyu Park (Multimedia Lab, SAIT).

Left to right: D. Timasov, A. Konushin, A. Zhirkov, A. Ignatenko, L. Levkovich-Maslyuk, Mahnjin Han (May, 2001, Moscow University, Graphics&Media Lab).