Author Topic: Upscaling Technique  (Read 33654 times)

bnvm

  • Member
  • ***
  • Posts: 159
Upscaling Technique
« on: September 05, 2013, 11:34:58 PM »
Image 1


Image 2


Ok, so here are 2 images, 1 was up-scaled 200% can anyone guess which one? The process is described here in the link below. In addition to up-scaling it also reduces color moire artifacts as well.

http://ns2.scarletuser.com/showthread.php?2489-Proper-processing-upres-ing-and-the-future-5-years-of-resolution

For and easier description of the process check out post #112.

Maybe someone here can port this technique to a standalone app to make it more accessible to everyone.

EDIT by SDX: 900px size, please

Adjusted images to 900px thanks SDX.

Midphase

  • Hero Member
  • *****
  • Posts: 712
Re: Upscaling Technique
« Reply #1 on: September 09, 2013, 08:21:09 AM »
Interesting, I assume picture 2 is the scaled one since it looks sharper?

I don't use Nuke, anyone care to explain how to achieve this in Resolve or AE?

deleted.account

  • Member
  • ***
  • Posts: 154
Re: Upscaling Technique
« Reply #2 on: September 09, 2013, 08:40:19 AM »
Avisynth, Vapoursynth or Imagemagick. Any camera raw app that works in Lab colour space.

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #3 on: September 09, 2013, 05:03:58 PM »
Nuke has a node called TVIScale, it is a special type of scaling and only does 200% scaling. It is better than any up-scaling filtering that I have seen in any other app and I don't think it is possible in AE or Resolve, never used resolve though. Every other process listed in the link I posted is for removing color moire, if you look closely at the hair in image 2 you can see some magenta/green artifacts, they are gone in image 1.

Image 1 is the scaled one, both are being viewed at 100%.

Here was my process, sorry I don't know of any other app besides nuke that can do all of this but that doesn't mean there isn't one. If anyone can translated these steps to other apps that would be great.

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) sharpen the Y channel
5.) convert to RGB
6.) TVIScale 200% (this is also called lambda scaling, afaik)
7.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)

So far these steps have given me the best results. In most cases when viewing at 100% there is only a barely visible change between the up-scaled and reference images and the difference can only noticeable when toggling back and forth.

PressureFM

  • Member
  • ***
  • Posts: 171
Re: Upscaling Technique
« Reply #4 on: September 09, 2013, 05:28:41 PM »
As great as that is, Nuke is £2,500 without VAT  :-\

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #5 on: September 09, 2013, 05:45:45 PM »
Unfortunately yes, but this process may be able to be extended to others apps I am going to give it a shot in AE and will post if it works.

EDIT: I tried it in AE and got comparable results, Nuke is just a tiny bit better.

For AE the process is the same, used the channel combiner to convert to YUV which is the same as YCrCb, R->Y, G->Cr, B->Cb. AE required sharpening post scale so and nested comps. I also had to separate out the red channel for sharpening and add it back in. One other side effect was the green pixels on the border but that is easily cropped out. Also I think I pushed the sharpness a little to hard on this one.

1 to 6: same.

7.) un-sharp mask.


zuzukasuma

  • Member
  • ***
  • Posts: 104
  • in Canon we Trust.
Re: Upscaling Technique
« Reply #6 on: September 09, 2013, 06:40:50 PM »
CS6 works like a charm for me for last 1 year, but CC's resizer better, unfortunately I don't know how long does it take 1hr footage from 1280x720 to 1920x1080.
in a complicated relationship with eos m.

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #7 on: September 09, 2013, 07:44:55 PM »
Probably around 8 hours or so depending on your machine. I believe I get around 3 FPS doing that.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: Upscaling Technique
« Reply #8 on: September 09, 2013, 07:49:59 PM »
Probably around 8 hours or so depending on your machine. I believe I get around 3 FPS doing that.

Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #9 on: September 09, 2013, 08:38:47 PM »
Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

Well not exactly that is how long it would take to scale and write out a 1 hr movie including debayering in ACR, etc.., and that guess may have been optimistic. Writing 1472x1104 raw took 35 minutes to write a 22.5 second long mp4 with Adobe Media Encoder. No scaling or CC, all that was doing was doing was loading it with a 2.0 PAR. It may end up being more like an hour a minute. While the nuke method has produced the best results for me it is way slower than AE->media encoder. The TVIScaling in nuke takes ~3 seconds per frame to compute at 1440x1080.

The nuke method takes would take about 1.25 hrs per minute of video, on top of the time it takes to export an image sequence to go from AE to Nuke.

I always apply the sharping to the Y channel before scaling as I posted above. I tried doing the AA before the scaling and sharpening but most of the time it blurs the image too much. With the limited testing I have done so far I have had better results with it at the end.

Raw video is great but processing it takes a lot of time.

zuzukasuma

  • Member
  • ***
  • Posts: 104
  • in Canon we Trust.
Re: Upscaling Technique
« Reply #10 on: September 09, 2013, 09:14:25 PM »
Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

remember the xeon I'm talking about?

now imagine that next gen after effects will support quadro k2000 and TSX-NI, that speed will be around 25fps/Live.
in a complicated relationship with eos m.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: Upscaling Technique
« Reply #11 on: September 10, 2013, 02:44:54 AM »
I always apply the sharping to the Y channel before scaling as I posted above. I tried doing the AA before the scaling and sharpening but most of the time it blurs the image too much. With the limited testing I have done so far I have had better results with it at the end.

AA works better after scaling because there is more information to work with, the lines are larger and it's easier to to blur the fine details without softening the image to much.  The scaler is doing it's own slight AA also.  Rather then sharpening those fine aliased edges only to blur them again later on, you should try sharpening after the AA (when you have the fine edge detail that you want).

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) sharpen the Y channel
8.) Final down scale to output resolution

remember the xeon I'm talking about?

Go for it.  Get a Blue Gene and it will be even faster ;)

zuzukasuma

  • Member
  • ***
  • Posts: 104
  • in Canon we Trust.
Re: Upscaling Technique
« Reply #12 on: September 10, 2013, 02:51:33 AM »
... Go for it.  Get a Blue Gene and it will be even faster ;)

I have rights to access miniature blue gene, don't worry.
in a complicated relationship with eos m.

3pointedit

  • Senior
  • ****
  • Posts: 263
  • 550D
Re: Upscaling Technique
« Reply #13 on: September 10, 2013, 03:20:56 AM »
As silly as this sounds, the freeware called Blender has a node compositor that may achieve these steps for less than Nuke. Also operates in float space.
550D on ML-roids

zuzukasuma

  • Member
  • ***
  • Posts: 104
  • in Canon we Trust.
Re: Upscaling Technique
« Reply #14 on: September 10, 2013, 04:17:31 AM »
As silly as this sounds, the freeware called Blender has a node compositor that may achieve these steps for less than Nuke. Also operates in float space.

not silly at all, after effects does it a bit faster than blender, I always have blender on my pc next to gimp.
in a complicated relationship with eos m.

x4kep

  • New to the forum
  • *
  • Posts: 7
Re: Upscaling Technique
« Reply #15 on: September 10, 2013, 08:29:04 PM »
Can you upload nuke script? Thanks.

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #16 on: September 10, 2013, 10:40:40 PM »
1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) sharpen the Y channel
8.) Final down scale to output resolution

I will give your work flow a shot and see how it goes, thanks for the advice. Also the steps would go like this since you are sharpening Y last, I assume the TVIScale works better if its RGB.

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) convert the image to YCrCb.
8.) sharpen the Y channel
9.) convert to RGB
10.) Final down scale to output resolution

Also for you AE users processing raw with AE is 5X faster than encoding with Media Encoder.

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #17 on: September 10, 2013, 11:06:52 PM »
Can you upload nuke script? Thanks.

Its pretty simple here are the nodes.

1.) either use a "Blur" with channels set to rgb with only b checked, or use a "DenoiseBlue" , or both.
2.) "ColorSpace" set out to YCrCb
3.) "Blur" channels set to rgb g,b checked
4.) "ColorSpace" set in to YCrCb out to Linear
5.) "TVIScale"
6.) "Matrix" enter 3 and 3, check normalize, then set like so

1 2 1
2 4 2
1 2 1

7.) "ColorSpace" set out to YCrCb
8.) "Sharpen" channels to rgb, r checked
9.) "ColorSpace" set in to YCrCb out to Linear
10.) "Reformat" to whatever you want 2K for example.

Rewind

  • Senior
  • ****
  • Posts: 356
Re: Upscaling Technique
« Reply #18 on: September 10, 2013, 11:25:42 PM »
Step 3. blur G and B perhaps? (R from RGB becomes Y in YCrCb)

x4kep

  • New to the forum
  • *
  • Posts: 7
Re: Upscaling Technique
« Reply #19 on: September 10, 2013, 11:30:07 PM »

6.) "Matrix" enter 3 and 3, check normalize, then set like so

1 2 1
2 4 2
1 2 1


That's what I wanted to know. Thanks!

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: Upscaling Technique
« Reply #20 on: September 11, 2013, 12:22:22 AM »
I will give your work flow a shot and see how it goes,

I'd skip the extra color space conversions at the end also bnvm, to minimize rounding errors.
With the denoising of Cr/Cb channels earlier, it should be fine to just sharpen the entire signal.

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #21 on: September 11, 2013, 06:18:06 AM »
Step 3. blur G and B perhaps? (R from RGB becomes Y in YCrCb)

That's correct it was a typo, thanks.

Quote
I'd skip the extra color space conversions at the end also bnvm, to minimize rounding errors.
With the denoising of Cr/Cb channels earlier, it should be fine to just sharpen the entire signal.

I will try that as well, thanks again. Also the YCrCb conversion and bluring of the g,b channels successfully removes color noise from shadows with no negative side effects as far as I can tell.

DFM

  • Freshman
  • **
  • Posts: 86
Re: Upscaling Technique
« Reply #22 on: September 11, 2013, 08:59:53 AM »

bnvm

  • Member
  • ***
  • Posts: 159
Re: Upscaling Technique
« Reply #23 on: September 11, 2013, 04:55:46 PM »
Hopefully this will help..  ;)

http://tv.adobe.com/watch/adobe-at-ibc-2013/after-effects-cc-detailpreserving-upscale/

Looks good from the video, I just got CC at work so I will test it out once I get it installed. Thanks.

zuzukasuma

  • Member
  • ***
  • Posts: 104
  • in Canon we Trust.
Re: Upscaling Technique
« Reply #24 on: September 11, 2013, 04:58:32 PM »
Looks good from the video, I just got CC at work so I will test it out once I get it installed. Thanks.

I just did, CC hasn't got that module yet.
in a complicated relationship with eos m.