Upscaling Technique

[bnvm]

Image 1


Image 2


Ok, so here are 2 images, 1 was up-scaled 200% can anyone guess which one? The process is described here in the link below. In addition to up-scaling it also reduces color moire artifacts as well.

http://ns2.scarletuser.com/showthread.php?2489-Proper-processing-upres-ing-and-the-future-5-years-of-resolution

For and easier description of the process check out post #112.

Maybe someone here can port this technique to a standalone app to make it more accessible to everyone.

EDIT by SDX: 900px size, please

Adjusted images to 900px thanks SDX.

[Midphase]

Interesting, I assume picture 2 is the scaled one since it looks sharper?

I don't use Nuke, anyone care to explain how to achieve this in Resolve or AE?

[deleted.account]

Avisynth, Vapoursynth or Imagemagick. Any camera raw app that works in Lab colour space.

[bnvm]

Nuke has a node called TVIScale, it is a special type of scaling and only does 200% scaling. It is better than any up-scaling filtering that I have seen in any other app and I don't think it is possible in AE or Resolve, never used resolve though. Every other process listed in the link I posted is for removing color moire, if you look closely at the hair in image 2 you can see some magenta/green artifacts, they are gone in image 1.

Image 1 is the scaled one, both are being viewed at 100%.

Here was my process, sorry I don't know of any other app besides nuke that can do all of this but that doesn't mean there isn't one. If anyone can translated these steps to other apps that would be great.

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) sharpen the Y channel
5.) convert to RGB
6.) TVIScale 200% (this is also called lambda scaling, afaik)
7.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)

So far these steps have given me the best results. In most cases when viewing at 100% there is only a barely visible change between the up-scaled and reference images and the difference can only noticeable when toggling back and forth.

[PressureFM]

As great as that is, Nuke is £2,500 without VAT 

[bnvm]

Unfortunately yes, but this process may be able to be extended to others apps I am going to give it a shot in AE and will post if it works.

EDIT: I tried it in AE and got comparable results, Nuke is just a tiny bit better.

For AE the process is the same, used the channel combiner to convert to YUV which is the same as YCrCb, R->Y, G->Cr, B->Cb. AE required sharpening post scale so and nested comps. I also had to separate out the red channel for sharpening and add it back in. One other side effect was the green pixels on the border but that is easily cropped out. Also I think I pushed the sharpness a little to hard on this one.

1 to 6: same.

7.) un-sharp mask.

[zuzukasuma]

CS6 works like a charm for me for last 1 year, but CC's resizer better, unfortunately I don't know how long does it take 1hr footage from 1280x720 to 1920x1080.

[bnvm]

Probably around 8 hours or so depending on your machine. I believe I get around 3 FPS doing that.

[Audionut]

Probably around 8 hours or so depending on your machine. I believe I get around 3 FPS doing that.

Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

[bnvm]

Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

Well not exactly that is how long it would take to scale and write out a 1 hr movie including debayering in ACR, etc.., and that guess may have been optimistic. Writing 1472x1104 raw took 35 minutes to write a 22.5 second long mp4 with Adobe Media Encoder. No scaling or CC, all that was doing was doing was loading it with a 2.0 PAR. It may end up being more like an hour a minute. While the nuke method has produced the best results for me it is way slower than AE->media encoder. The TVIScaling in nuke takes ~3 seconds per frame to compute at 1440x1080.

The nuke method takes would take about 1.25 hrs per minute of video, on top of the time it takes to export an image sequence to go from AE to Nuke.

I always apply the sharping to the Y channel before scaling as I posted above. I tried doing the AA before the scaling and sharpening but most of the time it blurs the image too much. With the limited testing I have done so far I have had better results with it at the end.

Raw video is great but processing it takes a lot of time.

[zuzukasuma]

Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

remember the xeon I'm talking about?

now imagine that next gen after effects will support quadro k2000 and TSX-NI, that speed will be around 25fps/Live.

[Audionut]

I always apply the sharping to the Y channel before scaling as I posted above. I tried doing the AA before the scaling and sharpening but most of the time it blurs the image too much. With the limited testing I have done so far I have had better results with it at the end.

AA works better after scaling because there is more information to work with, the lines are larger and it's easier to to blur the fine details without softening the image to much.  The scaler is doing it's own slight AA also.  Rather then sharpening those fine aliased edges only to blur them again later on, you should try sharpening after the AA (when you have the fine edge detail that you want).

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) sharpen the Y channel
8.) Final down scale to output resolution

remember the xeon I'm talking about?

Go for it.  Get a Blue Gene and it will be even faster

[zuzukasuma]

... Go for it.  Get a Blue Gene and it will be even faster

I have rights to access miniature blue gene, don't worry.

[3pointedit]

As silly as this sounds, the freeware called Blender has a node compositor that may achieve these steps for less than Nuke. Also operates in float space.

[zuzukasuma]

As silly as this sounds, the freeware called Blender has a node compositor that may achieve these steps for less than Nuke. Also operates in float space.

not silly at all, after effects does it a bit faster than blender, I always have blender on my pc next to gimp.

[x4kep]

Can you upload nuke script? Thanks.

[bnvm]

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) sharpen the Y channel
8.) Final down scale to output resolution

I will give your work flow a shot and see how it goes, thanks for the advice. Also the steps would go like this since you are sharpening Y last, I assume the TVIScale works better if its RGB.

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) convert the image to YCrCb.
8.) sharpen the Y channel
9.) convert to RGB
10.) Final down scale to output resolution

Also for you AE users processing raw with AE is 5X faster than encoding with Media Encoder.

[bnvm]

Can you upload nuke script? Thanks.

Its pretty simple here are the nodes.

1.) either use a "Blur" with channels set to rgb with only b checked, or use a "DenoiseBlue" , or both.
2.) "ColorSpace" set out to YCrCb
3.) "Blur" channels set to rgb g,b checked
4.) "ColorSpace" set in to YCrCb out to Linear
5.) "TVIScale"
6.) "Matrix" enter 3 and 3, check normalize, then set like so

1 2 1
2 4 2
1 2 1

7.) "ColorSpace" set out to YCrCb
8.) "Sharpen" channels to rgb, r checked
9.) "ColorSpace" set in to YCrCb out to Linear
10.) "Reformat" to whatever you want 2K for example.

[Rewind]

Step 3. blur G and B perhaps? (R from RGB becomes Y in YCrCb)

[x4kep]

6.) "Matrix" enter 3 and 3, check normalize, then set like so

1 2 1
2 4 2
1 2 1

That's what I wanted to know. Thanks!

[Audionut]

I will give your work flow a shot and see how it goes,

I'd skip the extra color space conversions at the end also bnvm, to minimize rounding errors.
With the denoising of Cr/Cb channels earlier, it should be fine to just sharpen the entire signal.

[bnvm]

Step 3. blur G and B perhaps? (R from RGB becomes Y in YCrCb)

That's correct it was a typo, thanks.

I'd skip the extra color space conversions at the end also bnvm, to minimize rounding errors.
With the denoising of Cr/Cb channels earlier, it should be fine to just sharpen the entire signal.

I will try that as well, thanks again. Also the YCrCb conversion and bluring of the g,b channels successfully removes color noise from shadows with no negative side effects as far as I can tell.

[DFM]

Hopefully this will help.. 

http://tv.adobe.com/watch/adobe-at-ibc-2013/after-effects-cc-detailpreserving-upscale/

[bnvm]

Hopefully this will help.. 

http://tv.adobe.com/watch/adobe-at-ibc-2013/after-effects-cc-detailpreserving-upscale/

Looks good from the video, I just got CC at work so I will test it out once I get it installed. Thanks.

[zuzukasuma]

Looks good from the video, I just got CC at work so I will test it out once I get it installed. Thanks.

I just did, CC hasn't got that module yet.