The researchers developed this study to explore the potential of the human eye as a source of information about the surrounding world. They leveraged the reflective nature of the human eye to collect multiple views of a scene outside the camera’s direct line of sight through the reflections in the eyes. The main challenge was to reconstruct a 3D scene beyond the camera’s line of sight using portrait images containing eye reflections. This task was challenging due to the difficulty of accurately estimating eye poses and the entangled appearance of the eye iris and the scene reflections.
The researchers proposed a method that jointly refines the cornea poses, the radiance field depicting the scene, and the observer’s eye iris texture. They also introduced a simple regularization prior on the iris texture pattern to improve reconstruction quality. The feasibility of their approach was demonstrated through various experiments on synthetic and real-world captures featuring people with varied eye colors.
The primary contributions of the study are:
- Presenting a novel method for reconstructing 3D scenes of the observer’s world from eye images, integrating earlier foundational work with the latest advancements in neural rendering.
- Introducing a radial prior for iris texture decomposition in eye images, significantly improving the quality of the reconstructed radiance field.
- Developing a cornea pose refinement procedure to alleviate the noisy pose estimates of eyes, which overcomes the unique challenge of extracting features from human eyes.
These advancements extend the current capabilities of 3D scene reconstruction through neural rendering to handle partially corrupted image observations obtained from eye reflections. This opens up new possibilities for research and development in the broader area of accidental imaging to reveal and capture 3D scenes beyond the visible line-of-sight.
Publication Date: June 15, 2023
Project Page: https://world-from-eyes.github.io/
Paper: https://arxiv.org/pdf/2306.09348.pdf
