crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[sylvain_c]

crop_rec_4k Apr29 not working here. I have followed exactly the video tutorial, but it result in random multicolored pixels (viewed in Mlrawviewer and raw2cdng1.7.9). also tested with and without fps overide.

dng sample:
https://mega.nz/#!DocmzRJJ!-A96kskYpm7ZqzFynl_4MZH9yd0cfc9O9bNCcrPlM5k

Edit: Ok sorry, I succeed with mlv_dump.exe

[Danne]

I would like to focus on 10bit12bit files in recent mlv_dump again. Seems the fix is very close but if it the problem passes unnoticed now it will probably be buried into other functions adding up in time.
I test mlv_dump with the following command(latest crop_rec_4k code)
mlv-dump --dng INPUT.MLV

12bit test file
https://bitbucket.org/Dannephoto/magic-lantern/downloads/12bit.MLV


The last commit that will export 10bit12bit files correctly is this
1738cb0
https://bitbucket.org/hudson/magic-lantern/commits/1738cb06e2652d9751f9407490de507fe30ef520?at=crop_rec_4k


Next commit is where things are getting wrong with 10bit12bit footage. Compiling from this commit will cause segmentation fault 11, however the next commit partly fix this but will export frames with missing information(see examples below)
92d1b4d
https://bitbucket.org/hudson/magic-lantern/commits/92d1b4d2958975f50ba428853545f64db8a61205?at=crop_rec_4k

Following commit, "re-adding 14bpp enforcement for DNG" produces black 10bit/12bit dng files cause it won´t fill the frame entirely(see examples). Dcraw exports the file and it shows where the image information is missing.
8b31c98
https://bitbucket.org/hudson/magic-lantern/commits/8b31c98380d5dc27573886d0f84ac7ed0c5d0f2b?at=crop_rec_4k

Looking at the code changes I see a lot of changes around frame_buffer_size, read, prev_frame_size so I guess somewhere in there this could be fixed. Unfortunately I couldn´t get this right with the example file provided but hopefully a dev could take a look or give a hint on how to fix it.


dcraw -T INPUT.dng gives:

Working commit
1738cb0




Not working

[Kharak]

Can I update from 113 to 123 Canon firmware, if I install Apr. 29 Build? Or do I have to update firmware via EOS utility? I am currently on 113... Its 2 years since I did firmware update 

[Walter Schulz]

You can install *Canon* firmware 1.2.3 using Canon menu. If you do this you have to run ML builds for *Canon* firmware 1.2.3.
ML is not firmware.
All builds should be available for 1.1.3 and 1.2.3. There is no need to upgrade Canon firmware if you don't have to for other reasons.

[Kharak]

I meant if the 123 firmware is also included in the Apr 29 build. As in i could update to 123 from 113 by updating the build and simaltaneously update the firmware?

[Walter Schulz]

Best way to semi-brick/stalling your cam.

[Savely]

@Danne thanks, you clarify issue pretty well. Just to remind you and those who develop some software solutions within magic lantern - currently mlv_dump didn't work properly with Dual ISO MLVs from crop_mode. Sometimes frames corrupted with looking-like-labirynth pattern and it seems randomly.

So it seems that currently there are just one reliable (but long) workflow for HDR-video when you shoot in crop_mode - MLV to DNG-sequence then DNG-sequence to MOV then MOV with Avysinth (in HDR-workflow pack) processed to A, B and C jpg-sequence then this C-sequence again assembled in some final video format.

[Kharak]

Best way to semi-brick/stalling your cam.

So the simple answer is: No. Thank you

[g3gg0]

12bit test file
https://bitbucket.org/Dannephoto/magic-lantern/downloads/12bit.MLV

where does that file come from? mlv_lite?

[g3gg0]

reason for asking is this:

Code: [Select]

Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 212.444000 ms
    Res:  960x540
    raw_info:
      api_version      0x00000001
      height           1181
      width            1888
      pitch            2832
      frame_size       0x003B8A48
      bits_per_pixel   12
      black_level      512
      white_level      4050
      active_area.y1   26
      active_area.x1   152
      active_area.y2   1181
      active_area.x2   1886
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1

Block: VIDF
  Offset: 0x0000027c
  Number: 9
    Size: 3344624
    Time: 550.688000 ms
   Frame: #0000
    Crop: 536x332
     Pan: 536x333
   Space: 0
   depth: 12 -> 14, size: 777600 -> 907200 (116.67%)
   black: 512 -> 2048
   white: 4050 -> 16200

Res:  960x540 -> 777600 bytes frame size.
but the frame seems 1181x1888 -> 3344592 bytes.

looks like the whole raw buffer was written instead of the specified frame size.
patching the fields results in proper frames.

any idea where this file came from?

[Danne]

It's an early build put into a 600D. Wonder if it was called raw_rec(mlv_lite). Stupid of me not to download a later build when testing. Right now I don't have the camera nor the firmware version until monday.
On another note. Is recent 10bit12bit files working with latest crop_rec_4k like it should? Last I tried it wouldn't work. If so, I guess there is no issue. Don't have any cam here unfortunately to verify.
By the way. That particular file will process with earlier builds. Guess there is some other buffer detectors in previous code?

[Danne]

Just tested with mlv files on my eos m coming from mlv_lite and regular mlv_rec and both transcodes 10bit12bit with wrong buffer(black bottom). Downloaded experimental raw_video_10bit_12bit 30 april build. Ran latest mlv_dump (crop_rec_4k) code.

Here is the thing. If I replace

Code: [Select]

if(read_size != (int)frame_buffer_size)with

Code: [Select]

if(read_size > (int)frame_buffer_size)later 10bit12bit files will process correctly. Is this the fix we could use? It still won´t work to compress 10bit12bit files(mlv_dump -c setting produces black output) but I guess they are already compressed being 10bit12bit so maybe compression for lower bits should be excluded?

[erek]

Here's my first 3072x1320 24 fps recording:

so with 14-bit lossless 3K i am seeing about 6.5 minutes of record time on a 64GB CF card... does this seem accurate?  also what is a recommended Aspect Ratio?

[g3gg0]

Here is the thing. If I replace

then you will get crashes

but thanks for your testing. just pushed a fix that should fix that issue.
after converting bit depth, frame size was updated, but not data size.

reason for having both is obvious when compression is used. frame size wont change with or without compression, whereas the resulting binary data varies in size.

[g3gg0]

Here's my first 3072x1320 24 fps recording:
so with 14-bit lossless 3K i am seeing about 6.5 minutes of record time on a 64GB CF card... does this seem accurate?  also what is a recommended Aspect Ratio?

3072 * 1320 * 24 * 60 * 6.5 * 14/8 * 0.58
gives ~38GiB

[Danne]

@g3gg0

Code: [Select]

                        frame_buffer_size = new_size;
Tested, working. Thanks a million.

[lostfeliz]

@erek, your shadows look a little noisy. maybe you underexposed or maybe you had a high ISO?

[bazidl]

What program are you guys using post production (after exporting via mlv_dump). I would normally use Premier pro cc 2015 but it doesn't recognize the .dngs whenever I use any new feature settings...although I can view the individual .dngs fine in photoshop or any image viewer.

[Danne]

Mlvfs or cr2hdr.app will produce cdng files

[bazidl]

I tried cr2hdr.app before and still got the same message as using mlv_dump. I'm prob missing a setting that allows for the proper export. Regardless just tried Mlvfs and it worked great!

Thanks for the help

[Danne]

I see. What settings did you use for it not to work? I would like to reproduce this issue. Do you have a sample file to test with?

[Danne]

@g3gg0

Code: [Select]

mlv_dump -c -c is not working healthy right now. dng files comes out with 0 black and white level for instance. I narrowed the issue down to this change.

Code: [Select]

                            if(new_depth != old_depth)reverting back to this and it works again.

Code: [Select]

                            if(new_depth)

There is also one of these beauties left

Code: [Select]

LJ92: Failed
which could go into nifty --relaxed to let mlv_dump produce compressed dng files.

Code: [Select]

                                /* set new compressed size and copy buffers */
                                frame_buffer = realloc(frame_buffer, compressed_size);
                                assert(frame_buffer);
                                memcpy(frame_buffer, compressed, compressed_size);
                                frame_buffer_size = compressed_size;
                            }
                            else
                            {
                                print_msg(MSG_ERROR, "    LJ92: Failed (%d)\n", ret);
                                goto abort;
                            }to

Code: [Select]

                                /* set new compressed size and copy buffers */
                                frame_buffer = realloc(frame_buffer, compressed_size);
                                assert(frame_buffer);
                                memcpy(frame_buffer, compressed, compressed_size);
                                frame_buffer_size = compressed_size;
                            }
                            else
                            {
                                print_msg(MSG_ERROR, "    LJ92: Failed (%d)\n", ret);
                                if(relaxed)
                                {
                                    goto skip_block;
                                }
                                goto abort;
                            }

[g3gg0]

first one done, really have to rework some code to make such changes happen just in one point.

the second thing, does this ever happen?

[Danne]

the second thing, does this ever happen?

You´re right. I´m working a version which turns on the compression switch when selecting -c -c. Not a problem in crop_rec_4k version.

[goldenchild9to5]

@g3gg0 @Danne Great job guy's.. ya'll working.