DIY Photography

Hacking Photography - one Picture at a time

  • News
  • Inspiration
  • Reviews
  • Tutorials
  • DIY
  • Gear
Search

Submit A Story

This crazy new NVIDIA tech turns 2D photos into fully 3D scenes in seconds

Mar 25, 2022 by John Aldred Leave a Comment

  • Share
  • Tweet
  • Flipboard
  • WhatsApp

Creating 3-dimensional worlds and objects from flat 2-dimensional photographs isn’t a new concept. It’s been happening for years. In the early days (and still often, today), we used photographs inside 3D modelling software as reference images for our 3D creations. A few years pass and along comes photogrammetry, allowing the automatic creation of 3D models and scenes from a couple of hundred photos.

Photogrammetry often takes hours, but this new tech from NVIDIA can do it in seconds. And you don’t need hundreds of images in order to do it, either. Just a few dozen. The tech demo above illustrates the principle where NVIDIA reproduces an iconic image of Andy Warhol shooting a Polaroid camera and with just a relatively handful of images, it gets turned into a 3D scene with a virtual camera that can fly around it with ease.

The new technology was revealed during a keynote presentation at NVIDIA GTC this week. Naturally, it uses AI to get this level of performance using neural radiance fields (NeRF). It’s a relatively new technique, with NVIDIA’s new implementation of it being dubbed “Instant NeRF” due to the short amount of time required to synthesise the 3D scene. NVIDIA says it’s the fastest NeRF technique to date.

The bit you want to watch is about the Omniverse Digital Twins, which starts at 51:04 in the video. The new technique offers speed increases of over 1,000x in some cases, with a model requiring just seconds to train on a few dozen still photos. It still requires some basic information about the cameras, too, like the angles from which the images were shot – like regular photogrammetry – but then can render the 3D scene “within tens of milliseconds”.

A millisecond is a thousandth of a second. And “tens of milliseconds”… Well, if it can do it less than four tens of milliseconds (0.04 seconds) then that’s pretty much real-time image generation at 24fps from a 3D scene made from still photographs that itself only took a handful of seconds to create. That’s pretty instant!

To get the level of speed that it does, NVIDIA says that it relies on a technique they developed called multi-resolution hash grid encoding which, naturally, is optimised to run on NVIDIA GPUs. The new encoding methods allow researchers to achieve high quality 3D results using tiny and fast neural networks. It was developed using the NVIDIA CUDA Toolkit and the Tiny CUDA Neural Networks library and can be trained and run on a single NVIDIA GPU – as long as it’s an RTX card with Tensor cores.

NVIDIA says that the Instant NeRF tech could be used to create avatars or scenes for virtual worlds, to capture video conference participants and their environments in 3D or to reconstruct scenes for 3D digital maps. I can also see this being combined with some of the Unreal Engine tech used in virtual sets like those used to shoot The Mandalorian or the BBC Olympics coverage to place actors and talent in projected 3D worlds that could have been shot mere minutes or seconds before being used.

Outside of creative uses and looking at more practical everyday applications, NVIDIA suggests that it might be used to train robots and self-driving cars to construct “real” 3D worlds from the 2D flat images they capture to provide more context to its environment, ultimately making them safer and more reliable.

There’s no word yet on when something will be released that we can have a go with ourselves at home. I did ask NVIDIA and they said that they “can’t comment on any unannounced products”. So, it could be weeks or even years before something is released to the general public. But when it does, that’s gonna be a lot of fun to play with!

You can find out more and see the full keynote on the NVIDIA website.

FIND THIS INTERESTING? SHARE IT WITH YOUR FRIENDS!

  • Share
  • Tweet
  • Flipboard
  • WhatsApp

Related posts:

NVIDIA’s GauGAN AI landscape generator can now create scenes from scratch from written descriptions Nvidia’s new “computational zoom” tech lets you change composition in post Diveroid turns your smartphone into a fully fledged dive camera, computer and logger “Reverse Prisma” AI turns paintings into photos and turns winter to summer

Filed Under: news Tagged With: AI, GTC 2022, Nerf, Nvidia, Photogrammetry, Tech Demo

About John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter - and occasional beta tester - of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

« More content moderators sue TikTok over psychological trauma
These trippy “micro landscapes” are made only with light reflected in opals »

Submit A Story

Get our FREE Lighting Book

DIYP lighting book cover

* download requires newsletter signup
DIYPhotography

Recent Comments

Free Resources

Advanced lighting book

Recent Posts

  • Sony’s ZV-E1 is a cut-down FX3 in a vlogging form factor
  • Watch: How good (or bad) is an $8.50 tripod?
  • How to light and photograph Lego building interiors
  • Lighting Setup: How to light your portraits with £50 LED tubes
  • Review: Insta360 announces its first gimbal – The AI-tracking Insta360 Flow

Alex Baker is a portrait and lifestyle driven photographer based in Valencia, Spain. She works on a range of projects from commercial to fine art and has had work featured in publications such as The Daily Mail, Conde Nast Traveller and El Mundo, and has exhibited work across Europe

Dave Williams is an accomplished travel photographer, writer, and best-selling author from the UK. He is also a photography educator and published Aurora expert. Dave has traveled extensively in recent years, capturing stunning images from around the world in a modified van. His work has been featured in various publications and he has worked with notable brands such as Skoda, EE, Boeing, Huawei, Microsoft, BMW, Conde Nast, Electronic Arts, Discovery, BBC, The Guardian, ESPN, NBC, and many others.

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter - and occasional beta tester - of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Dunja Djudjic is a multi-talented artist based in Novi Sad, Serbia. With 15 years of experience as a photographer, she specializes in capturing the beauty of nature, travel, and fine art. In addition to her photography, Dunja also expresses her creativity through writing, embroidery, and jewelry making.

Copyright © DIYPhotography 2006 - 2023 | About | Contact | Advertise | Write for DIYP | Full Disclosure | Privacy Policy