ProRes 4444 XQ

[reddeercity]

Just some Fact about Apple ProRes 4444 and XQ
Apple ProRes White Paper

• Apple ProRes 4444: This codec features full-resolution, masteringquality
4:4:4:4 RGBA color and visual fidelity that is perceptually indistinguishable
from the original material. Apple ProRes 4444 is a high-quality solution for storing
and exchanging motion graphics and composites, with excellent multigeneration
performance and a mathematically lossless alpha channel up to 16 bits.
....target data rate of approximately 330 Mbps for 4:4:4 sources at 1920 x 1080 and 29.97 fps

variable bit rate (VBR)......Apple ProRes 4444 support image sources up to 12 bits
and preserve alpha sample depths up to 16 bits.

• Apple ProRes 4444 XQ: The highest-quality version of Apple ProRes for 4:4:4:4 image
sources (including alpha channels), with a very high data rate to preserve the detail
in high-dynamic-range imagery generated by today’s highest-quality digital image
sensors. Apple ProRes 4444 XQ preserves dynamic ranges several times greater than
the dynamic range of Rec. 709 imagery—even against the rigors of extreme visual
effects processing, in which tone-scale blacks or highlights are stretched significantly.
Like standard Apple ProRes 4444, this codec supports up to 12 bits per image channel
and up to 16 bits for the alpha channel. Apple ProRes 4444 XQ features a target data
rate of approximately 500 Mbps for 4:4:4 sources at 1920 x 1080 and 29.97 fps.

This is biggest reason to use XQ for ml raw as prores4444 will clip high lights & shadows because of the rec709 color space.
where as XQ is linear , I guess closer to rec2020 wide gamut . There is a big difference between them.

alexa xrxt supports newprores 4444 xq

"ProRes 4444 XQ is a fantastic choice for high-end mastering and archiving," says Henning Rädlein,  ARRI's Head of Digital Workflow Solutions.
"ProRes 4444 XQ offers 12-bit RGB encoding with a low compression ratio of 1:4.5 that maintains the superior tonal range of Log C

HDCAM SR vs ProRes 4444

ProRes 4444 330Mb/sec or 5.6:1 compression

Now FFmpeg prores 444 is just a 10bit YUV (RGB) I would put it on the some par as
Apple ProRes 422 or Blackmagic RGB codec for quality , but you are  still constrained to a limited color space.

Finally If you were one of the lucky one that got the Miraizon ProResXQ Codecs for PC before Apple
shut him down , then you could encode ProRes4444XQ on Windows PC .
I had a chance to view a video file encoded with the "miraizon prores4444xq" in windows "Hex Editor" and from what I see it looks
identical to a Apple encode file just has more metadata info , if you load a ProRes file in a Hex editor look at the "moov" header its at the back of the file .
I can see why apple shut it down it has the "ap4x" codec id where as the FFmpeg Prores444 uses  "ap10" ap4h(YUV444p10le)
10 is the 10bit
My conclusion is that it is 12/16bit file .

[Danne]

When selecting the following syntax. 

Code: [Select]

ffmpeg -vcodec prores_ks -pix_fmt yuv444p10 -n -r
You get following from terminal. This is also the highest quality ProRes I could cream out of FFMpeg.

Code: [Select]

[swscaler @ 0x7fc234800600] deprecated pixel format used, make sure you did set range correctly
[prores_ks @ 0x7fc234013200] Autoselected 4:4:4:4 profile because of the used input colorspace. It can be overridden through -profile option.
Output #0, mov, to 'hej.mov':
  Metadata:
    major_brand     : qt 
    minor_version   : 537331968
    compatible_brands: qt  CAEP
    encoder         : Lavf56.40.101
    Stream #0:0(eng): Video: prores (prores_ks) (ap4h / 0x68347061), yuv444p10le, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 59.94 fps, 60k tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 2016-01-17 12:08:10
      timecode        : 12:28:08;03
      encoder         : Lavc56.60.100 prores_ks
    Stream #0:1(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2016-01-17 12:08:10
      timecode        : 12:28:08;03
      encoder         : Lavc56.60.100 aac

Code: [Select]

Autoselected 4:4:4:4 profile because of the used input colorspace.

Code: [Select]

prores (prores_ks) (ap4h / 0x68347061), yuv444p10le, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 59.94 fps, 60k tbn, 59.94 tbc (default)



Now when piping dng files through dcraw to FFmpeg that is the bigger issue. How would you for example use the highlight settings to match extreme lighting conditions which seems to be handled perfectly in ACR.
FFmpeg prores is fine for general use and the files are really high quality and ACR workflow will do this even better. What would it look like if ACR could be piped to FFmpeg ?

Since we,re at lossy/lossless codec why not work with this

Code: [Select]

-c:v huffyuv -pix_fmt rgb24
It turned a 250mb file into 1.5gb and I only got it to play in mpv player https://mpv.io/installation/

[dfort]

Going back to the post that rebooted this discussion:

Hello everyone,

     I am doing a 48 hour film challenge for fun this upcoming weekend, and am using my 7D for raw video. My Macbook Pro is okay for offloading with MLVFS and converting to ProRes 4444 XQ with After Effects and Camera Raw/VisionLog/Vision Log to REC 709 LUT. However, my friend just made a really beefy computer, but it is Windows platform. He has Adobe CC 2015. Is there any way to make it possible for Windows to support ProRes 4444 XQ video files to edit in Premiere Pro? The only possible way I see so far is Cinemartin (costs money) or somehow making the files I dragged from Final Cut Pro X (or Motion, don't remember which one it was - RedDeerCity showed us on this forum post) compatible on Windows. If anyone has any suggestions I would greatly appreciate it. This would be a lot easier to edit in Premiere Pro on a beefy system with all the ProRes 4444 XQ files. Thanks!

- Paul Harwood

I think the answer would be that if you want to create a "real" ProRes 4444 XQ you'll have to use the Apple codec and do it on a Mac.

So is the 48 hour film challenge over? What did you end up doing?

[PaulHarwood856]

Hello everyone,

     I apologize for the delay in response. I really appreciate everyone's feedback, it was very helpful. The 48 hour is this weekend, but Sunday we did a test shoot. I really am glad we did. I have come to the conclusion ProRes 4444 XQ is not going to be possible on my friend's pc, like dfort stated, but I believe I've found an alternate solution. I have Avid's DNxHR 444 codec which produces, to what I've read, the same as ProRes 4444 XQ. The only difference I mainly saw was the inability to pick Trillions of Colors+, but there was still the option for Millions of Colors+. If anyone has any information on this I would greatly appreciate their input. Basically I need to export a codec on my Macbook Pro that my friend will be able to edit with on his pc. The Macbook Pro is for offloading, and he wants to use his pc he just built. I'm mainly looking to be able to edit with a codec best suited for Magic Lantern raw, at the least possible compression on both a PC and Mac, intermittently.

     I did experience some issues when shooting raw on Sunday, and was hoping for some feedback on this as well. I ran into vertical banding while using a Sandisk 128 gb 120 mb/sec, 85 mb/sec write speed, and found this is actually a problem with the 7D. I read online there is a solution to prevent this in camera before shooting. If I set the digital ISO to ex. -0.2 EV and black level to ex. -10 the lines will go away. Now I read this was in low light, so I wanted to figure out if there were settings to prevent vertical banding under good lighting or daylight. Does anyone know of good settings for removing vertical banding? Maybe expose to the right? I have researched this topic, but only really found some solutions to the 5D Mark III. The vertical banding even happens with the VAF-7D Anti Aliasing and Moire filter from Mosaic Engineering. I realize this isn't quite about the ProRes 4444 XQ codec, but I figure since it's related to the 48 hour it's okay to at least ask.

     Also, if I'm using Cinemascope crop bars in the 7D, should I have the aspect ratio set at 2:35:1 or would there be less distortion at 16:9? I mean where the faces get widened a little. I was able to make it work with 16:9 by using two layers of black video, and cropping the top on one by 88%, not 87%, and the same with cropping the bottom, with 88%, not 87%. I had a flash of white in one clip and a corrupted frame in the other, and some audio stops in mid waveform. I'm thinking to prevent this, I could film at 2:35:1 at 23.976 with sound. We are also recording sound separately with a clapboard and using Plural Eyes.

     I was using a 7 inch MustHD monitor, and was able to get it to work properly with the 16:10 monitor setting. The 7D froze when zooming with focus ring setting was turned on, and I learned to turn Global Draw off before shooting to prevent issues. I'm just looking to see if anyone would like to share settings that have worked well for them when shooting.

     I really appreciate all Magic Lantern has done for DSLR filmmakers, and I am absolutely loving the 7D and T3i with this firmware add-on. If anyone can let me know how to make the 7D work as smooth as possible, I would greatly appreciate it. Thanks so much.

- Paul Harwood

     

[dfort]

Interesting stuff but yeah, you're drifting way off topic. Maybe start a new topic? Seems that this might be especially interesting to Windows users. I've never used Avid's DNxHR 444 codec but looking up the specs it looks like it is very similar to the other Avid HD codecs I used which lock you into standard video frame sizes so no 2.35:1 option, you'll have to letterbox. No big deal but if you shoot 16x9 you can reframe vertically if needed. Of course hitting that 48 hour deadline you're better off making things as simple as possible and concentrate on the story.

Back on topic, one of the advantages ProRes has over Avid is the flexibility of using non-standard image formats. This is especially helpful when shooting 2.35:1 -- which is a standard aspect ratio in the movie industry. Of course manufacturers are constantly updating their codecs so maybe Avid will come out with something for wide screen productions.

Good luck!

[arrinkiiii]

For vertical banding (patterns of noise in vertical?) ...you can use MLVProducer that you can upload black frame for subtract and get all the noise that exist in the 7D.

DNxHR or ProRes in 444 is more then good. They say that 4444XQ is like working with RAW...

[PaulHarwood856]

Thanks so much dfort! I appreciate all of your feedback. Do you think ProRes 4444 would be a better idea versus DNxHR? I believe it is still possible to get the ProRes 4444 codec on Windows. Do you know?

- Paul Harwood

Hey arrinkiiii,

Do I need a black frame already recorded? I'm using MLVFS and the vertical stripe fix is on. However, no difference with it turned on or off, the vertical stripes are still there. Yes, they are patterns of vertical noise, vertical banding. DNxHR seems to be weird when exported, I'll try again.

- Paul Harwood

[DeafEyeJedi]

Paul,

If I remember correctly (will have to check my 7D) but I think they show up on certain ISO's (some worse than others) so which ISO were you shooting in?

Anyway to stay on topic ... I prefer 4444XQ over 4444 or let alone 422HQ. Why? Because I don't like getting the highlights/shadows clipped and personally I would like to retain DR as much as possible which is key to XQ over standard 4444 (highlights may seem like it's not clipped as much but it's definitely noticeable) if you will.

Which is why life sucks having sacks of External Drives everywhere all loaded w archived files in XQ's along with a flat log (at least for me I use Cinelog-C to make it easier in the long run) applied to use as starting points.

Many may feel that this is slightly overkill. TBH if we are going to risk our lives flying high with ML then we might as well just duplicate what the RAW sensors are actually giving us by using XQ, right? [emoji6]

Re: ProRes on Windows ... Absolutely this is possible ... Miraizon used to sell them as plugins but they got shut down by Apple.

Try this one (a bit pricey) but you can use their demo trail version...

http://www.cinemartin.com/cinec/

[PaulHarwood856]

Hey DeafEyeJedi,

     Thanks so much for your reply. I was using ISO 400 with good lighting. Do you know if I should shoot in increments of 60 versus 100? I think I my friend, the crew, and I might be able to afford the student discount for $108.50. However, they don't have ProRes 4444 XQ yet. From my understanding, as long as Quicktime Player Pro 7 is installed on Windows, ProRes support up to 4444 is allowed for decoding. All I need is his pc to support ProRes 4444 XQ files that would be converted from dngs from MLVFS in After Effects. I just want to make sure because if I export ProRes 4444 XQ, and his pc won't support them, well, yeah, that would be bad. The other solution I possible saw was taking the ProRes 4444 XQ codec files from the Quicktime folder on my Macbook Pro (from reddeercity's video) and placing them in the Quicktime folder for Quicktime Player Pro 7 on my friend's pc. Is this possible. I tried Avid DNxHR and couldn't stand it. ProRes 4444 looked ok, but yes, 4444 XQ is super legit, especially since CC 2015 supports it. I definitely agree with you about using ProRes 4444 XQ to get the best possible compression out of Canon DSLRs. Maybe there's a version of XQ with GoPro Cineform?

- Paul Harwood

[reddeercity]

@ PaulHardwood856 , Ok so you want to do a cross platform edit right.
There is some options here to use .
1st: Extract with MLVFS  Windows PC ver. & MLVFS for MAC so  now you have 2 copies of your project in a virtual directories (Mac & PC)
From there put them in a real file folder , from there you can go right to Premiere Pro CC with Cdng's (native support)on pc & mac)  and edit etc... then
export a Edl or XML . From there you can use Blackmagic Resolve , load up the XML or EDL and re-link .
This can be done in either PC or MAC , from there you can do a round trip to FCPX . Go to the Mac platform  open resolve load the EDL or XML , re-link
setup a Round trip to fcpx (basically resolve will export proxies or full res , up to you plus a fcpxml this can be prores4444xq .
Edit sent back fcpxml to resolve grade etc...

Or you can 1st set up Resolve with Cdng's  round trip to fcpx or round trip to premiere pro on PC .

Or the last option , If your friend is adventures and the new PC is a Intel CPU i7 family then make a Hackintosh
Load MAC OSX and a way you go .......   Then they will be encoding prores xq

I have 2 , i7 Toshiba Laptop & 4770 i7 desktop workstation over clock to 4.7Ghz 

[PaulHarwood856]

Hey reddeercity,

     Thanks so much for your input. My friend is a bit biased against macs, so I'm not sure he will make it a Hackintosh. I've been wanting to build a Hackintosh, will happen at some point. I wish he would, since I'm pretty sure he has good enough specs for it. He doesn't have FCPX, and neither do I. I have tried it in a trial version, but it seemed to be like an upgrade from iMovie. I couldn't get into using that program for some reason, although I've heard it flows the best with Macs since it is utilized for mac hardware and the operating system. I've heard DaVinci Resolve is the best with color, but I'm not too familiar with it yet. It looks like I might have to settle with ProRes 4444, since if Quicktime Player Pro 7 is loaded on Windows the codecs are accessible. But, I do have a question. Do you know if it is possible to load the new codecs from Motion (in your tutorial) to a Windows PC? I have the Quicktime folder on my Mac with all codecs, including ProRes 4444 XQ. I'm wondering if I can put them on a flash drive and drag them on the Quicktime folder on his PC. The only problem I have with Premiere Pro is the colors are not too good importing cdngs natively, since it only supports up to 10 bit I believe (according to Adobe on the phone). Also, Adobe Camera Raw gets rid of dead pixels before exporting to ProRes 4444 XQ in After Effects, versus Premiere Pro where it doesn't. The colors from Adobe Camera Raw are not replicated in Premiere Pro right since the bit depth is lowered, at least from my experience. Also, do you know if there is another way to remove dead pixels besides in camera raw? Thanks again for all of your help!

- Paul Harwood

[dfort]

Thanks so much dfort! I appreciate all of your feedback. Do you think ProRes 4444 would be a better idea versus DNxHR? I believe it is still possible to get the ProRes 4444 codec on Windows. Do you know?

I'd say take the path of least resistance. I've worked with both the Apple ProRes and the Avid DNx codecs and both do the job. I don't even know if it is possible to encode a proper Apple approved ProRes file on Windows. ProRes encoded on Mac seem to playback fine on Windows but why complicate things?

[reddeercity]

@PaulHarwood856 , Yes PC Windows can read Apple ProRes4444 files encoded on Mac.
Please note you will clip hi-light and crush shadows so make sure compress the hi-light etc...
Remember the Color profile is Rec709 so you are constrained to that where as with ProRes4444XQ it linear.
That were a Log profile or lut like Vision Log would help a great deal , do some research you will be glad you did.
No you can't take the ProRes "ap4x" Codec from mac and install it , doesn't work that way in Windows .

Off Topic now , this should be discussed in a different thread

The only problem I have with Premiere Pro is the colors are not too good importing cdngs natively, since it only supports up to 10 bit I believe .

What Premiere Pro CC 2015 is doing is putting the Cdng in a Log/sRGB profile (that way you get real time play back) , just like resolve with BMD film , flat profile
this is what you are looking for, log image. And from there PP can direct import to Speedgrade  (I like it.)
I'm thinking of Grading my current project with Speedgrade even thou it's blackmagic cinema camera  2.4k footage.
Just think of the Cdng's in PP as Proxies and when you Color in speedgrade you will have the full 16bit image to work with.
ACR/A.E. is a whole different kettle of fish.
FYI: You can import a A.E. Comp in to Premiere Pro but it will be slower play back.
Here a vid I did some time ago about that very some thing , importing A.E. Comp in to PP 

[PaulHarwood856]

Hey dfort,

     Thanks for your reply. I've been trying to export with DNxHR and I keep getting video lag. It's really odd. I think I will stick with ProRes 4444 or try GoPro Cineform, which comes with Adobe CC automatically, which my friend also has. Cinemartin has a codecs, but not ProRes 4444 XQ yet. But you are definitely right, why complicate things?

- Paul Harwood

Hey reddeercity,

     Thanks for your reply. I use Visionlog in Camera Raw, and it is fantastic. So with just using ProRes 4444 and Vision Log that makes sense that I will have pretty good results. I watched your video, and it was very helpful. Definitely a good workflow to keep in mind. I didn't realize Speedgrade worked with 16 bit cdngs, which is really cool.

- Paul Harwood

[surami]

All I need is his pc to support ProRes 4444 XQ files that would be converted from dngs from MLVFS in After Effects. I just want to make sure because if I export ProRes 4444 XQ, and his pc won't support them, well, yeah, that would be bad.

Try this, maybe it works:
- render a short ProRes 4444 XQ movie on your MAC.
- give the file to your friend.
- open the file in VirtualDub's Hex editor (Tools -> Hex editor -> File -> Open)
- Search for "ap4x", rename it to "ap4h", save the file.
- Play the file on the PC.
- Come back and say something... is it working or not?

You could upload a short ProRes 4444 XQ movie somewhere and I could try too.

[PaulHarwood856]

Hey surami,

     Thanks so much for your response, and sorry for the late reply. If you could try this that would be fantastic! Also, if it works, is it possible to do batch processing? Thanks. Here is a link to a ProRes 4444 XQ file: https://drive.google.com/file/d/0B7-vwCkJCer5bmxvTUpSbUJxUGc/view?usp=sharing

[surami]

It works , you have to rename all "ap4x"s to "ap4h", but I don't know any batch processing, just search for "hex editor batch process", I'm sure you will find something usable.

update: I did a quick grade on Windows with AE CC 2015.

[PaulHarwood856]

Hey surami,

     Alright awesome, thanks so much. I'll make sure to search that, and post back here if I found a way to do it. Awesome grade!

- Paul Harwood

[PaulHarwood856]

Hey everyone,

     Here are links to the 48 hour film.

https://vimeo.com/paulharwood856/thosewhodig

    Post production took longer, but it was still written and shot in 48 hours. We transcoded in ProRes 4444 XQ using VisionLog to retain detail. If anyone has any questions please let me know. This was a great experience.

- Paul Harwood