Nvidia has developed neural network that could revolutionise video streaming

Oct 7, 2020

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Nvidia has developed neural network that could revolutionise video streaming

Oct 7, 2020

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Join the Discussion

Share on:

YouTube video

One of the big problems with video, especially when watching it online, is the amount of bandwidth it often takes up. This problem of bitrates is particularly a problem when it comes to things like live streaming and video conferencing. Researchers at Nvidia think they’ve found a way around the limitations of existing video streaming codecs, though, with the development of a new neural network engine.

The new engine works by completely ignoring every traditional video codec out there in order to bring the amount of bandwidth required for video streaming down to a fraction of what it might normally use with something like h.264.

The new technology is possible due to Nvidia Maxine, their cloud-AI video streaming platform for developers. Nvidia posted a video explaining the new technology to YouTube, which you can see above.

It essentially works by sending along the usual keyframes like you’d expect with h.264, but instead of generating whole in-between frames and sending those down the pipe, as would happen with h.264, it creates a sort of mask of the subject, focusing on key parts of their face. The movements of these key locations is then sent instead of image data for each frame, and the recipient sees an AI-reconstructed image.

This means that the bandwidth is drastically reduced as only a relatively tiny data set is sent to the recipient instead of entire images or blocky chunks of changed pixel data.

Nvidia says that this technology also offers some other advantages for video conferencing, too, such as being able to reposition the subject’s head to face the camera. For most of us, looking directly at the camera isn’t possible. Our cameras are usually either on top of the monitor that’s displaying the other side of the conversation or they’re off to the side – so we’re not looking directly at the camera. The new Nvidia technology solves that problem, by helping to make sure subjects are looking at each other while they talk.

And, naturally, because it’s able to understand the shape and structure of your face, you’re able to swap yourself out for things like virtual characters – although that isn’t really anything new. My phone’s been able to do that for a couple of years already now.

The technology is still in its early days, and you can see that there are some issues with this technique in the video up top. When you compare the quality vs the amount of bandwidth used, though, there’s really no comparison when it comes to something like video conferencing.

As time progresses and the technology gets more advanced and more reliable and cameras and screens keep increasing in resolution, this or a similar technology could potentially become the key to streaming super high resolution and high frame rate data of all types into our homes without having to go through major hardware upgrades to increase the available bandwidth.

[via DPreview]

Filed Under:

Tagged With:

Find this interesting? Share it with your friends!

John Aldred

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Join the Discussion

DIYP Comment Policy
Be nice, be on-topic, no personal information or flames.

Leave a Reply

Your email address will not be published. Required fields are marked *

10 responses to “Nvidia has developed neural network that could revolutionise video streaming”

  1. g_discus Avatar
    g_discus

    bandwidth for sound?

    1. Kaouthia Avatar
      Kaouthia

      What about it?

      1. g_discus Avatar
        g_discus

        hmmm… I think it’s more than video. Therefore, the benefits are questionable.

        1. Kaouthia Avatar
          Kaouthia

          Audio doesn’t use anywhere near the amount of bandwidth that video does. Video bitrates are counted in MEGAbits per second. Audio bitrates are KILObits per second. 1 Megabit = 1024 Kilobits.

          1. g_discus Avatar
            g_discus

            Ok. Let’s calculate nvideo bitrate + audio bitrata. 0.000000001 kb (video of nvidia) + 10 kb audio. Main information is the sound, it needs much bigger bandwidth than new video. Where is the advantage?

          2. Kaouthia Avatar
            Kaouthia

            What ARE you talking about?

          3. g_discus Avatar
            g_discus

            audio data is too big

          4. Kaouthia Avatar
            Kaouthia

            Audio data IS SMALLER THAN VIDEO

          5. g_discus Avatar
            g_discus

            yes, smaller than current video. new technology is good for tiny video bitrate, but there is big new problem: audio data much more bigger than video

        2. Kaouthia Avatar
          Kaouthia

          Look, never mind. Just stop.