Results 1 to 10 of 10

Thread: Testing AI Processing on Earls Court 1980 to clean up generation loss artifacts.

  1. #1
    Join Date
    Nov 2018
    Posts
    79
    Thanks
    1
    Thanked 591 Times in 9 Posts

    Default Testing AI Processing on Earls Court 1980 to clean up generation loss artifacts.

    ESRGAN is an AI software designed to try to clean up an image using a model. This model is developed by taking a low quality version and a high quality version and lining them up, then the computer tries thousands of different modifications to the image to try to bring the low quality image as close to the high resolution image as it possibly can. Then you can use that model on images that it doesn't know to achieve similar results.

    I decided to try to experiment with the software and with a bit of help from a user on a restoration Discord named Tanks, I was able to get the software to build the model and to use the models working. My plan was to try to get the AI to restore the Earl's Court VHS footage, by comparing it against clips from the Behind The Wall documentary. I left the model running overnight for the results and here they are.




    More examples here
    https://imgsli.com/NDE1NTE
    https://imgsli.com/NDE1NTQ
    https://imgsli.com/NDE1NTY
    https://imgsli.com/NDE1NTI
    https://imgsli.com/NDE1NTM

    Now the model has it's issues, some may thing that the image is a bit too blurry, also it adds this weird blue tint to some colors. However what I find surprising is that it got rid of that weird black aberration, which has always been a pain to anyone who tried to restore this. It also cleans up a good bit of the crunchiness caused by the horrible MPEG2 compression that was used when recording the tape onto DVD.

    However the main problem with this technique is the speed at which it runs. The software uses CUDA, which is a type of compute unit present in NVIDIA graphics cards, I have a pretty good graphics card all things considered (an RTX 3060ti) however it takes about 4-7 seconds per frame to process. Which doesn't sound to long until you realize that this is NTSC Video, so after being deinterlaced it runs at 60 frames per second. The entire show is about 388,000 frames which doing the math would take about 3-4 weeks of compute time to fully render out. So obviously this would take a while to finish. I made a test of about 5500 frames and that took about 4 1/2 hours to finish, which you can see here. https://drive.google.com/file/d/1MB4...ew?usp=sharing

    If anyone wants to try using the model for the Earls Court footage or any other similarly damaged tapes, here is the model I made: https://drive.google.com/file/d/15Je...ew?usp=sharing
    I would recommend using the software Cupscale, as it's the most user friendly interface. But make sure to have Python 3.8 installed and preferably an NVIDIA GPU, also make sure to go into the settings and install the resource files and embedded version of Python if you want to try it out.
    Last edited by Notelu; 2021-02-20 at 10:27 PM.

  2. #2
    Join Date
    Oct 2016
    Location
    PA
    Posts
    403
    Thanks
    17
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by Notelu View Post
    ESRGAN is an AI software designed to try to clean up an image using a model. This model is developed by taking a low quality version and a high quality version and lining them up, then the computer tries thousands of different modifications to the image to try to bring the low quality image as close to the high resolution image as it possibly can. Then you can use that model on images that it doesn't know to achieve similar results.

    I decided to try to experiment with the software and with a bit of help from a user on a restoration Discord named Tanks, I was able to get the software to build the model and to use the models working. My plan was to try to get the AI to restore the Earl's Court VHS footage, by comparing it against clips from the Behind The Wall documentary. I left the model running overnight for the results and here they are.




    More examples here
    https://imgsli.com/NDE1NTE
    https://imgsli.com/NDE1NTQ
    https://imgsli.com/NDE1NTY
    https://imgsli.com/NDE1NTI
    https://imgsli.com/NDE1NTM

    Now the model has it's issues, some may thing that the image is a bit too blurry, also it adds this weird blue tint to some colors. However what I find surprising is that it got rid of that weird black aberration, which has always been a pain to anyone who tried to restore this. It also cleans up a good bit of the crunchiness caused by the horrible MPEG2 compression that was used when recording the tape onto DVD.

    However the main problem with this technique is the speed at which it runs. The software uses CUDA, which is a type of compute unit present in NVIDIA graphics cards, I have a pretty good graphics card all things considered (an RTX 3060ti) however it takes about 4-7 seconds per frame to process. Which doesn't sound to long until you realize that this is NTSC Video, so after being deinterlaced it runs at 60 frames per second. The entire show is about 388,000 frames which doing the math would take about 3-4 weeks of compute time to fully render out. So obviously this would take a while to finish. I made a test of about 5500 frames and that took about 4 1/2 hours to finish, which you can see here. https://drive.google.com/file/d/1MB4...ew?usp=sharing

    If anyone wants to try using the model for the Earls Court footage or any other similarly damaged tapes, here is the model I made: https://drive.google.com/file/d/15Je...ew?usp=sharing
    I would recommend using the software Cupscale, as it's the most user friendly interface. But make sure to have Python 3.8 installed and preferably an NVIDIA GPU, also make sure to go into the settings and install the resource files and embedded version of Python if you want to try it out.
    I could see it used for small artifacts like those whitelines that appear often. Nothing too big though like full scale restoration attempts. It would just take actual months to render every single song.
    Concerts:

    Roger Waters - Philadelphia, 2017.

    Ozzy Osbourne - Camden, NJ, 2018.

  3. #3
    Join Date
    Sep 2010
    Posts
    140
    Thanks
    270
    Thanked 779 Times in 16 Posts

    Default

    I prefer the original
    some contemporary modifications aimed at "improving" have the same effect as sound compression.

  4. #4
    Join Date
    Jan 2008
    Posts
    759
    Thanks
    53
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by voices2010 View Post
    I prefer the original
    some contemporary modifications aimed at "improving" have the same effect as sound compression.
    Agreed. Trying to remove VHS artifacts, etc, without just resorting to using a fancy blurring technique, is a bit like trying to push toothpaste back into the tube. OP's 'improved' photos just look different. A lot of the definition and surviving detail has been lost, the colours don't look any better, and some VHS muck has still passed through.

    I know that there is this belief that AI and machine learning will solve all of our problems but, for example, I'm getting a bit bored of seeing artificially colourised videos all the time. The latest one I saw was that clip of King Crimson performing 'Cat Food' on Top of the Pops. The footage had been made much darker (presumably to improve the results), and the AI used had fairly faithfully mapped overly-tanned skin tones to faces and hands. Like all of these experiments, it didn't know what to do with clothes so it just mapped a constant shifting pink/blue/green/pink/blue to fabric that changed with any movement going on. Part of the problem is maybe the assumption that everything has to have some sort of colour, all of the time.


  5. #5
    Join Date
    Jul 2014
    Posts
    2,246
    Thanks
    329
    Thanked 984 Times in 11 Posts

    Default

    Quote Originally Posted by Orgone Accumulator View Post
    Agreed. Trying to remove VHS artifacts, etc, without just resorting to using a fancy blurring technique, is a bit like trying to push toothpaste back into the tube. OP's 'improved' photos just look different. A lot of the definition and surviving detail has been lost, the colours don't look any better, and some VHS muck has still passed through.

    I know that there is this belief that AI and machine learning will solve all of our problems but, for example, I'm getting a bit bored of seeing artificially colourised videos all the time. The latest one I saw was that clip of King Crimson performing 'Cat Food' on Top of the Pops. The footage had been made much darker (presumably to improve the results), and the AI used had fairly faithfully mapped overly-tanned skin tones to faces and hands. Like all of these experiments, it didn't know what to do with clothes so it just mapped a constant shifting pink/blue/green/pink/blue to fabric that changed with any movement going on. Part of the problem is maybe the assumption that everything has to have some sort of colour, all of the time.

    I agree, most video processing tends to make the image look different and not necessarily better. It all depends on the source material, of course.

  6. #6
    Join Date
    Jul 2006
    Location
    Assen, Holland
    Posts
    419
    Thanks
    450
    Thanked 9 Times in 2 Posts

    Default

    Well, it can be done.
    I really like what they did with the WW1 footage in They Shall. Not Grow Old: https://youtu.be/zRed-Ri9IpI

  7. #7
    Join Date
    Oct 2016
    Location
    PA
    Posts
    403
    Thanks
    17
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by wbeens View Post
    Well, it can be done.
    I really like what they did with the WW1 footage in They Shall. Not Grow Old: https://youtu.be/zRed-Ri9IpI
    Well, with the WW1 footage they had access to the actual film reels used in the war. They have access to professional restoration programs.
    Concerts:

    Roger Waters - Philadelphia, 2017.

    Ozzy Osbourne - Camden, NJ, 2018.

  8. #8
    Join Date
    Mar 2008
    Posts
    219
    Thanks
    78
    Thanked 0 Times in 0 Posts

    Default

    In some ways it looks better, in other ways it looks worse.
    The soft focus is problematic in and of itself.
    You may have to go next-level, like the Star Wars team did with 35mm theatrical prints, and restore frame-by-frame.
    If you can find the best source of this show. Sadly, it'll probably be generational vhs.
    Depends how much you're willing to invest in this sort of thing.
    I'm still working on the audio of this...and you can't really set & forget software as a blanket approach. Every second has to be manually scrutinized. It's a labour of love for sure.

  9. #9
    Join Date
    Feb 2013
    Posts
    143
    Thanks
    35
    Thanked 150 Times in 3 Posts

    Default

    Quote Originally Posted by Roscoe59 View Post
    I agree, most video processing tends to make the image look different and not necessarily better. It all depends on the source material, of course.
    Though both of those are true, ESRGAN heavily depends on the model you trained and the time you spent on it and its not like any past video edit process. I've seen impossible things being achieved with it, though this is definitely an early quick attempt at the process.

  10. #10
    Join Date
    Jan 2007
    Posts
    386
    Thanks
    11
    Thanked 0 Times in 0 Posts

    Default

    Looks like the AI here does a good job of getting rid of certain types of crap - compression artefacts, artificial sharpening etc. But of course it doesn't add anything in their place, so the output looks a bit smoothed-out. Nothing you'd actually want has been taken away, it just sort of feels like it has.

    Presume re-balancing the colours could get rid of the blueish tinge?

    I posted about this on another thread a while back and no one directly responded - but it looks as if there may be a product called Topaz that could be useful in a subsequent stage of filling in the content that's missing: https://topazlabs.com/video-enhance-ai/

    There are some pretty impressive examples of what it can do in terms of genuinely bringing back lost details, including from old VHS footage. How much it could do with the somewhat degraded sources here I'm not sure, but might be worth looking at?

    Some examples of what it can do, admittedly with much higher-res source material, here: https://www.extremetech.com/extreme/...ance-ai-review

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •