Magic Lantern Forum

Using Magic Lantern => Post-processing Workflow => Topic started by: bnvm on September 05, 2013, 11:34:58 PM

Title: Upscaling Technique
Post by: bnvm on September 05, 2013, 11:34:58 PM
Image 1
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FtgtOopC.png&hash=a1073dac7677620ca67856461bbd1be2)

Image 2
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2F4EFIU1L.png&hash=2b6e1dc2584f6dfe6b64d6af53ac5e17)

Ok, so here are 2 images, 1 was up-scaled 200% can anyone guess which one? The process is described here in the link below. In addition to up-scaling it also reduces color moire artifacts as well.

http://ns2.scarletuser.com/showthread.php?2489-Proper-processing-upres-ing-and-the-future-5-years-of-resolution (http://ns2.scarletuser.com/showthread.php?2489-Proper-processing-upres-ing-and-the-future-5-years-of-resolution)

For and easier description of the process check out post #112.

Maybe someone here can port this technique to a standalone app to make it more accessible to everyone.

EDIT by SDX: 900px size, please

Adjusted images to 900px thanks SDX.
Title: Re: Upscaling Technique
Post by: Midphase on September 09, 2013, 08:21:09 AM
Interesting, I assume picture 2 is the scaled one since it looks sharper?

I don't use Nuke, anyone care to explain how to achieve this in Resolve or AE?
Title: Re: Upscaling Technique
Post by: deleted.account on September 09, 2013, 08:40:19 AM
Avisynth, Vapoursynth or Imagemagick. Any camera raw app that works in Lab colour space.
Title: Re: Upscaling Technique
Post by: bnvm on September 09, 2013, 05:03:58 PM
Nuke has a node called TVIScale, it is a special type of scaling and only does 200% scaling. It is better than any up-scaling filtering that I have seen in any other app and I don't think it is possible in AE or Resolve, never used resolve though. Every other process listed in the link I posted is for removing color moire, if you look closely at the hair in image 2 you can see some magenta/green artifacts, they are gone in image 1.

Image 1 is the scaled one, both are being viewed at 100%.

Here was my process, sorry I don't know of any other app besides nuke that can do all of this but that doesn't mean there isn't one. If anyone can translated these steps to other apps that would be great.

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) sharpen the Y channel
5.) convert to RGB
6.) TVIScale 200% (this is also called lambda scaling, afaik)
7.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)

So far these steps have given me the best results. In most cases when viewing at 100% there is only a barely visible change between the up-scaled and reference images and the difference can only noticeable when toggling back and forth.
Title: Re: Upscaling Technique
Post by: PressureFM on September 09, 2013, 05:28:41 PM
As great as that is, Nuke is £2,500 without VAT  :-\
Title: Re: Upscaling Technique
Post by: bnvm on September 09, 2013, 05:45:45 PM
Unfortunately yes, but this process may be able to be extended to others apps I am going to give it a shot in AE and will post if it works.

EDIT: I tried it in AE and got comparable results, Nuke is just a tiny bit better.

For AE the process is the same, used the channel combiner to convert to YUV which is the same as YCrCb, R->Y, G->Cr, B->Cb. AE required sharpening post scale so and nested comps. I also had to separate out the red channel for sharpening and add it back in. One other side effect was the green pixels on the border but that is easily cropped out. Also I think I pushed the sharpness a little to hard on this one.

1 to 6: same.

7.) un-sharp mask.

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2Fsv4XIdU.png&hash=1f88772330e8e97cface94729e91b101)
Title: Re: Upscaling Technique
Post by: zuzukasuma on September 09, 2013, 06:40:50 PM
CS6 works like a charm for me for last 1 year, but CC's resizer better, unfortunately I don't know how long does it take 1hr footage from 1280x720 to 1920x1080.
Title: Re: Upscaling Technique
Post by: bnvm on September 09, 2013, 07:44:55 PM
Probably around 8 hours or so depending on your machine. I believe I get around 3 FPS doing that.
Title: Re: Upscaling Technique
Post by: Audionut on September 09, 2013, 07:49:59 PM
Probably around 8 hours or so depending on your machine. I believe I get around 3 FPS doing that.

Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?
Title: Re: Upscaling Technique
Post by: bnvm on September 09, 2013, 08:38:47 PM
Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

Well not exactly that is how long it would take to scale and write out a 1 hr movie including debayering in ACR, etc.., and that guess may have been optimistic. Writing 1472x1104 raw took 35 minutes to write a 22.5 second long mp4 with Adobe Media Encoder. No scaling or CC, all that was doing was doing was loading it with a 2.0 PAR. It may end up being more like an hour a minute. While the nuke method has produced the best results for me it is way slower than AE->media encoder. The TVIScaling in nuke takes ~3 seconds per frame to compute at 1440x1080.

The nuke method takes would take about 1.25 hrs per minute of video, on top of the time it takes to export an image sequence to go from AE to Nuke.

I always apply the sharping to the Y channel before scaling as I posted above. I tried doing the AA before the scaling and sharpening but most of the time it blurs the image too much. With the limited testing I have done so far I have had better results with it at the end.

Raw video is great but processing it takes a lot of time.
Title: Re: Upscaling Technique
Post by: zuzukasuma on September 09, 2013, 09:14:25 PM
Ouch.  Just for the scaling process.

Wouldn't it be better to apply the Y channel sharpening after the anti-aliasing and before the final scale to render?

remember the xeon I'm talking about?

now imagine that next gen after effects will support quadro k2000 and TSX-NI, that speed will be around 25fps/Live.
Title: Re: Upscaling Technique
Post by: Audionut on September 10, 2013, 02:44:54 AM
I always apply the sharping to the Y channel before scaling as I posted above. I tried doing the AA before the scaling and sharpening but most of the time it blurs the image too much. With the limited testing I have done so far I have had better results with it at the end.

AA works better after scaling because there is more information to work with, the lines are larger and it's easier to to blur the fine details without softening the image to much.  The scaler is doing it's own slight AA also.  Rather then sharpening those fine aliased edges only to blur them again later on, you should try sharpening after the AA (when you have the fine edge detail that you want).

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) sharpen the Y channel
8.) Final down scale to output resolution

remember the xeon I'm talking about?

Go for it.  Get a Blue Gene (http://en.wikipedia.org/wiki/Blue_Gene) and it will be even faster ;)
Title: Re: Upscaling Technique
Post by: zuzukasuma on September 10, 2013, 02:51:33 AM
... Go for it.  Get a Blue Gene (http://en.wikipedia.org/wiki/Blue_Gene) and it will be even faster ;)

I have rights to access miniature blue gene, don't worry.
Title: Re: Upscaling Technique
Post by: 3pointedit on September 10, 2013, 03:20:56 AM
As silly as this sounds, the freeware called Blender has a node compositor that may achieve these steps for less than Nuke. Also operates in float space.
Title: Re: Upscaling Technique
Post by: zuzukasuma on September 10, 2013, 04:17:31 AM
As silly as this sounds, the freeware called Blender has a node compositor that may achieve these steps for less than Nuke. Also operates in float space.

not silly at all, after effects does it a bit faster than blender, I always have blender on my pc next to gimp.
Title: Re: Upscaling Technique
Post by: x4kep on September 10, 2013, 08:29:04 PM
Can you upload nuke script? Thanks.
Title: Re: Upscaling Technique
Post by: bnvm on September 10, 2013, 10:40:40 PM
1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) sharpen the Y channel
8.) Final down scale to output resolution

I will give your work flow a shot and see how it goes, thanks for the advice. Also the steps would go like this since you are sharpening Y last, I assume the TVIScale works better if its RGB.

1.) slightly blur/denoise the blue channel.
2.) convert the image to YCrCb.
3.) slightly blur the Cr and Cb channels (this removes color moire)
4.) convert to RGB
5.) TVIScale 200% (this is also called lambda scaling, afaik)
6.) apply an anti-aliasing filter to the image (this is done with a 3x3 matrix in nuke)
7.) convert the image to YCrCb.
8.) sharpen the Y channel
9.) convert to RGB
10.) Final down scale to output resolution

Also for you AE users processing raw with AE is 5X faster than encoding with Media Encoder.
Title: Re: Upscaling Technique
Post by: bnvm on September 10, 2013, 11:06:52 PM
Can you upload nuke script? Thanks.

Its pretty simple here are the nodes.

1.) either use a "Blur" with channels set to rgb with only b checked, or use a "DenoiseBlue" , or both.
2.) "ColorSpace" set out to YCrCb
3.) "Blur" channels set to rgb g,b checked
4.) "ColorSpace" set in to YCrCb out to Linear
5.) "TVIScale"
6.) "Matrix" enter 3 and 3, check normalize, then set like so

1 2 1
2 4 2
1 2 1

7.) "ColorSpace" set out to YCrCb
8.) "Sharpen" channels to rgb, r checked
9.) "ColorSpace" set in to YCrCb out to Linear
10.) "Reformat" to whatever you want 2K for example.
Title: Re: Upscaling Technique
Post by: Rewind on September 10, 2013, 11:25:42 PM
Step 3. blur G and B perhaps? (R from RGB becomes Y in YCrCb)
Title: Re: Upscaling Technique
Post by: x4kep on September 10, 2013, 11:30:07 PM

6.) "Matrix" enter 3 and 3, check normalize, then set like so

1 2 1
2 4 2
1 2 1


That's what I wanted to know. Thanks!
Title: Re: Upscaling Technique
Post by: Audionut on September 11, 2013, 12:22:22 AM
I will give your work flow a shot and see how it goes,

I'd skip the extra color space conversions at the end also bnvm, to minimize rounding errors.
With the denoising of Cr/Cb channels earlier, it should be fine to just sharpen the entire signal.
Title: Re: Upscaling Technique
Post by: bnvm on September 11, 2013, 06:18:06 AM
Step 3. blur G and B perhaps? (R from RGB becomes Y in YCrCb)

That's correct it was a typo, thanks.

Quote
I'd skip the extra color space conversions at the end also bnvm, to minimize rounding errors.
With the denoising of Cr/Cb channels earlier, it should be fine to just sharpen the entire signal.

I will try that as well, thanks again. Also the YCrCb conversion and bluring of the g,b channels successfully removes color noise from shadows with no negative side effects as far as I can tell.
Title: Re: Upscaling Technique
Post by: DFM on September 11, 2013, 08:59:53 AM
Hopefully this will help..  ;)

http://tv.adobe.com/watch/adobe-at-ibc-2013/after-effects-cc-detailpreserving-upscale/
Title: Re: Upscaling Technique
Post by: bnvm on September 11, 2013, 04:55:46 PM
Hopefully this will help..  ;)

http://tv.adobe.com/watch/adobe-at-ibc-2013/after-effects-cc-detailpreserving-upscale/

Looks good from the video, I just got CC at work so I will test it out once I get it installed. Thanks.
Title: Re: Upscaling Technique
Post by: zuzukasuma on September 11, 2013, 04:58:32 PM
Looks good from the video, I just got CC at work so I will test it out once I get it installed. Thanks.

I just did, CC hasn't got that module yet.
Title: Re: Upscaling Technique
Post by: Audionut on September 11, 2013, 05:04:01 PM
I just did, and the conclusion is all advertisements are fake. clearly they're resizing the post smaller size at 240dpi or so then resizing it up to 60dpi with greater resolution. nice trick though.

DPI only affects print detail.  It has no affect on video.  Video displays have a fixed number of DPI that is tied directly to resolution.
Title: Re: Upscaling Technique
Post by: zuzukasuma on September 11, 2013, 05:22:52 PM
DPI only affects print detail.  It has no affect on video.  Video displays have a fixed number of DPI that is tied directly to resolution.

tell me something I don't know. they are marketing guru's.
Title: Re: Upscaling Technique
Post by: DFM on September 14, 2013, 05:58:36 PM
It's part of the October update. Full details are available here (http://blogs.adobe.com/aftereffects/2013/09/after-effects-cc-12-1-whats-new-and-changed.html).

I just did, CC hasn't got that module yet.
Title: Re: Upscaling Technique
Post by: bnvm on September 24, 2013, 12:14:05 AM
I just tried adobe's new scaling method. It is not out for AE yet but it exists in PS so I did it there again both are scaled 200%. I tried to get the best result possible but generally speaking it over sharpens the image considerably.

As things stand now, Nuke is the winner in my opinion.

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2F6elRFnZ.jpg&hash=eb854eb5f6b0e9322182fdd2c519b16e)
Title: Re: Upscaling Technique
Post by: _no_id on August 25, 2014, 02:11:52 PM
As great as that is, Nuke is £2,500 without VAT  :-\
actually it happens to be a poor man's version of TVIScale Nuke node. Points 1-4 and 6-8 can be done in any software that can work with Lab color, and the substitute for Nuke lambda scaling is plugin for AviSynth called nnedi3_rpow2. Nnedi3 is originally a deinterlace plugin, and rpow2 is its version for enlarging images by powers of 2. Speaking of Magic lantern RAW, captured frames (if you are not using x5 zoom) are not downscaled from full sensor, but recieved by lineskipping, which causes aliasing artifacts, and after upscaling they often get worse, but not with nnedi, besause of its deinterlacing nature it actually fixes lineskipping artifacts. Here are some examples, look closely at sharp high contrast edges of cars and snow. And the most interesting thing is its actually a couple of times faster than nuke's tviscale node and you need only avisynth to use it.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2F2J2xvTT.png&hash=c06792d96cc55d657fb9e61c197d6507)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FiXBCupn.png&hash=0056ab0206b562437df1783c7378e0ad)