Convert 2D photos to 3D in seconds: Nvidia showcases impressive AI technology

Just in time to celebrate the 75th birthday of Polaroid, Nvidia has introduced a new technology that can create a 3D environment from 2D images using artificial intelligence calculations. This technology is called Neural Radiation Fields (NeRF) and should be able to train within a few seconds with dozens of images.

Buy a graphics card: Geforce offers from RTX 3050 to RTX 3090 available with discounts of up to 24%

David Luebke, Vice President of Graphics Research, compares NeRFs to bitmaps on the grounds that traditional 3D representations such as polygonal networks are comparable to vector images. It sees the use of technology primarily in the creation of avatars and scenes of the virtual world, for example b- 3D recording of participants in a video conference or to reconstruct digital 3D scenes. For illustrative purposes, as part of GTC 2022, Nvidia took a mockup of iconic artist Andy Warhole and converted the 2D image into a 3D scene using Instant NeRF.

Neural networks are used to train NeRF, which are trained using the input. In doing so, the neural network must be provided with a few dozen images from different perspectives about the corresponding scene and the exact camera position must also be entered. If there are people or other moving objects in the pictures, they should be taken at very short intervals, otherwise the corresponding parts of the 3D rendering will become blurred due to the large differences between the pictures. Due to many different angles and perspectives, NeRF can complete colors and lighting due to training and thus create a 3D scene.

The technology that Nvidia relies on has long been known, but the main problem of long training and correspondingly long display times has made the technology very demanding. Instant NeRF aims to change that with the so-called Multi-Resolution Hash Grid Encoding, which aims to produce decent results very quickly with the help of a small neural network. This technology supports Nvidia GPUs and uses Tensor cores. The manufacturer also sees practical applications outside of computing in training robots or self-driving cars, which can use 2D images with the help of NeRF to determine the sizes and shapes of real obstacles or other objects.

source: nvidia