Live light field recording and display

The project aims at developing an integrated system for real-time 3d light field recording and display. Applications of such a system include live 3d endoscope views in medicine, live 3d microscopy and 3d optical inspection.

The LiveRay project will create the first integrated light-field camera & light-field display system that enables the direct capturing of 3D light fields with a purpose-built camera, and visualize it in real time on a glasses-free light-field 3D display. Light-field is an emerging concept for representing rich 3D visual information that is able to capture real world phenomena with unprecedented image quality. A light field describes visual information as a set of light rays that pass a (sensor / display) surface, but apart from capturing the position and color of each light ray, the direction of each light ray is also recorded. Both, light-field cameras and light-field display exist in the market as of today (from Raytrix and Holografika, respectively). While light-field displays can reproduce hologram-like, full color, real time representations of anything that can be captured in light-field format, light-field cameras can capture 4D information of a subject, which can be used to reconstruct the depth of the recorded scene for every pixel, as well as to reconstruct all-in-focus images, all from a single shot. This project will be the first to combine the two technologies to generate completely new and exciting business opportunities. One of the main advantages of Holografika 3D light-field displays compared to standard stereo 3D display systems is that it offers a very natural, glasses-free 3D impression. This is very important for professional applications, where users need to use the display for hours at a time without their eyes tiring or hurting. The Raytrix 3D light-field cameras directly record light fields with a single camera, a single lens and in a single shot. Raytrix light-field cameras have unique advantages over other 3D recording systems in particularly when looking at small or even microscopic scenes. For example, a Raytrix light-field camera can record 3D microscope images at frame rates of up to 180 fps. Combining a Holografika 3D light-field display and a Raytrix 3D light-field camera is therefore the natural next step. One application is a completely new type of 3D microscope (or endoscope) where a number of users can look at a live, full color 3D microscope (or endoscope) image in real-time. This has many applications in medicine, medical research, biology, research and education, industrial quality inspection and countless other use cases where 2D microscopes are currently used. One example is the use for minimally invasive surgery which is performed with endoscopes. Having a live 3D view through the endoscope, visible for all personnel involved would be a great advantage. Another application area is security. Using a Raytrix light-field camera on a telescope has been shown to give 3D information even at a couple of kilometers distance. Combining such a camera system with a Holografika light-field display will therefore offer a completely new view of distant areas for security personnel, since the relative 3D positions of objects can be made out much better. On use case of such a system is the surveillance of borders. The second part of this project will consider the combination of a number of light-field cameras to generate a single view with a wide viewing angle for the Holografika display. A single light-field camera only captures the light rays within a fairly small angular range. To capture the light in a 180 degree angular range a number of light-field cameras will be needed. Currently, such wide viewing angle scenes are recorded by a large number of standard 2D cameras. The number of cameras needed can be reduced considerably when using light-field cameras instead. How to combine the light-fields of the separate cameras and how to display the combined light-field on a Holografika display will have to be developed. One application of such a system is the realistic display of faces for video conferencing systems. Two light-field cameras capture the face from different angles, which is then displayed on the light-field display. This will give a much more personal experience when communicating. Another application is the inspection of objects over a large viewing range. If not all interesting parts of an object can be seen with a single light-field camera, a number of light-field cameras can be installed to give a 3D view over a larger viewing range. Such images can also be stored for documentation and later 3D inspection. The proposed combination on light-field cameras and displays and the innovations required to make them seamlessly work together as an integrated system will bring groundbreaking advantages in light-field microscopy and visualization, contributing to better research, medicine, manufacturing and quality inspection processes. Spatial relationships in the micro-world will be made apparent, and thanks to light-field display technology, can be observed without wearing any 3D glasses.
Project ID: 
9 919
Start date: 
Project Duration: 
Project costs: 
660 000.00€
Technological Area: 
Imaging, Image Processing, Pattern Recognition
Market Area: 
Medical imaging

Raising the productivity and competitiveness of European businesses through technology. Boosting national economies on the international market, and strengthening the basis for sustainable prosperity and employment.