crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[thenickdude]

I'm having some troubles with mlv_dump and my MLV files on OS X High Sierra. My videos play fine on mlv_play, but while trying to unpack the MLV to DNGs with mlv_dump, I get a segfault:

Code Select Expand

XREF table contains 20 entries
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 9216039327435021128
    FPS         : 2.501000
    File        : 0 / 0
    Frames Video: 4
    Frames Audio: 0
    Class Video : 0x00000021
    Class Audio : 0x00000000
Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 0.754000 ms
    Res:  4096x3072
    raw_info:
      api_version      0x00000001
      height           3102
      width            4248
      pitch            7434
      frame_size       0x015FDF2C
      bits_per_pixel   14
      black_level      2047
      white_level      16200
      active_area.y1   30
      active_area.x1   146
      active_area.y2   3102
      active_area.x2   4248
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: RAWC
  Offset: 0x000000e8
  Number: 2
    Size: 32
    Time: 0.761000 ms
    raw_capture_info:
      sensor res      5760x3840
      sensor crop     1.00 (Full frame)
      sampling        1x1 (read every line, read every column)
Block: IDNT
  Offset: 0x00000108
  Number: 3
    Size: 84
    Time: 0.767000 ms
     Camera Name:   'Canon EOS 5D Mark III'
     Camera Serial: '4E58BA31B' (21031002907)
     Camera Model:  0x80000285
Block: EXPO
  Offset: 0x0000015c
  Number: 4
    Size: 40
    Time: 0.773000 ms
     ISO Mode:   0
     ISO:        1600
     ISO Analog: 104
     ISO DGain:  0/1024 EV
     Shutter:    19016 microseconds (1/52.59)
Block: LENS
  Offset: 0x00000184
  Number: 5
    Size: 96
    Time: 0.794000 ms
     Name:        '35-35mm'
     Serial:      '000000000' (no valid S/N)
     Focal Len:   35 mm
     Focus Dist:  0 mm
     Aperture:    f/2.80
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x00000028
     Flags:       0x00000000
Block: RTCI
  Offset: 0x000001e4
  Number: 6
    Size: 44
    Time: 0.810000 ms
     Date:        30.08.2018
     Time:        05:16:09 (GMT+0)
     Zone:        ''
     Day of week: 4
     Day of year: 241
     Daylight s.: 0
Block: WBAL
  Offset: 0x00000210
  Number: 7
    Size: 44
    Time: 1.446000 ms
     Mode:   1
     Kelvin:   5200
     Gain R:   491
     Gain G:   1024
     Gain B:   638
     Shift GM:   0
     Shift BA:   0
Block: VERS
  Offset: 0x0000023c
  Number: 8
    Size: 152
    Time: 599.243000 ms
  String: 'crop_rec built 2018-07-09 21:48:22 UTC; commit c76ef16 on 2018-07-04 12:58:01 UTC by alex: crop_rec: fix case fallthrough warnings '
Block: VERS
  Offset: 0x000002d4
  Number: 9
    Size: 148
    Time: 774.878000 ms
  String: 'ettr built 2018-02-17 21:49:58 UTC; commit 5aa7ec6 on 2018-02-16 23:44:52 UTC by alex: Merged 70D_merge_fw112 into crop_rec_4k '
Block: VERS
  Offset: 0x00000368
  Number: 10
    Size: 176
    Time: 809.217000 ms
  String: 'file_man built 2018-02-17 21:49:56 UTC; commit bcbabb3 on 2018-02-17 20:28:28 UTC by alex: Moved timer functions to timer.h (always included from dryos.h) '
Block: VERS
  Offset: 0x00000418
  Number: 11
    Size: 180
    Time: 999.310000 ms
  String: 'mlv_lite built 2018-07-03 23:13:14 UTC; commit f4213dd on 2018-07-03 23:10:22 UTC by g3gg0: mlv_lite: do not write all metadata again on every single chunk '
Block: VERS
  Offset: 0x000004cc
  Number: 12
    Size: 164
    Time: 1038.454000 ms
  String: 'mlv_play built 2018-06-20 20:32:09 UTC; commit 0e38b89 on 2018-06-18 22:11:42 UTC by g3gg0: mlv_play: use less RAM, proper cleanup on error '
Block: VERS
  Offset: 0x00000570
  Number: 13
    Size: 160
    Time: 1223.783000 ms
  String: 'mlv_rec built 2018-07-22 13:10:25 UTC; commit c1e44b8 on 2018-07-21 21:31:09 UTC by g3gg0: mlv_dump: pass INFO blocks as image description '
Block: VERS
  Offset: 0x00000610
  Number: 14
    Size: 160
    Time: 1296.649000 ms
  String: 'pic_view built 2017-08-19 10:52:04 UTC; commit fca6399 on 2017-08-19 06:39:25 UTC by alex: pic_view: lossless DNG support (experimental) '
Block: VERS
  Offset: 0x000006b0
  Number: 15
    Size: 148
    Time: 1331.059000 ms
  String: 'silent built 2018-07-09 21:48:10 UTC; commit 3d26697 on 2018-06-30 16:48:59 UTC by alex: lossless.c: fix warning in comment '
Block: VIDF
  Offset: 0x00000800
  Number: 16
    Size: 13493760
    Time: 2310.348000 ms
   Frame: #0000
    Crop: 152x30
     Pan: 152x30
   Space: 32
    LJ92: Decompressing
    LJ92: 2048x3072x2 14 bpp (25165824 bytes buffer)
    LJ92: 22020096 -> 22020096  (100.00% ratio)

Thread 2 received signal SIGSEGV, Segmentation fault.
dng_unpack_image_bits (input_buffer=0x10a2d5000, output_buffer=0x10c000000, max_size=<optimized out>,
    bpp=14) at dng/dng.c:739
739         uint32_t uncorrected_data = *((uint32_t *)&packed_bits[bits_address]);

The faulting video (54MB): https://s3.sherlockphotography.org/posts/2018/M30-0516.MLV

The video is 4096x3072 recorded at 2.5 FPS on the 5D III.

If I build mlv_dump using "USE_X64=1", some of my videos will decode without crashing, but some still crash, including this one. Running under Valgrind, I get:

Code Select Expand

==17989== Invalid read of size 4
==17989==    at 0x10000FE1F: dng_init_data (dng.c:739)
==17989==    by 0x1000084C6: main (mlv_dump.c:3486)
==17989==  Address 0x10bc1003e is 22,020,094 bytes inside a block of size 22,020,096 alloc'd
==17989==    at 0x1007243E4: realloc (vg_replace_malloc.c:788)
==17989==    by 0x1000061F8: main (mlv_dump.c:2917)
==17989==
==17989== Invalid write of size 4
==17989==    at 0x100010313: dng_save (dng.c:772)
==17989==    by 0x10000932D: main (mlv_dump.c:3632)
==17989==  Address 0x10d11103e is 22,020,094 bytes inside a block of size 22,020,096 alloc'd
==17989==    at 0x100723A36: malloc (vg_replace_malloc.c:302)
==17989==    by 0x10000FD75: dng_init_data (dng.c:823)
==17989==    by 0x1000084C6: main (mlv_dump.c:3486)
==17989==

It looks like dng_pack_image_bits is writing 2 bytes beyond the end of its buffer, because it writes 4 bytes at a time and that write can begin 2 bytes before the end of the buffer. I've submitted a PR to fix that by adding 2 bytes of padding to the allocated buffer size:

https://bitbucket.org/hudson/magic-lantern/pull-requests/944/mlv_dump-fix-segfault-while-writing-dngs/diff

dng_unpack_image_bits has a problem with invalid reads too, but I'm not sure of the best way to fix that one.

EDIT: The culmination of this morning's shooting with crop_rec, thanks for the great software!:

[tonij]

I have a 5d3 to play around with for the next couple days.
Can't seem to figure out crop rec, I'm interested in being able to record UHD.
Can somebody please tell me their working settings to get continuous'ish high res 24fps raw video?
Cheers

[zalbnrum]

I have a 5d3 to play around with for the next couple days.
Can't seem to figure out crop rec, I'm interested in being able to record UHD.
Can somebody please tell me their working settings to get continuous'ish high res 24fps raw video?
Cheers

First look at first post of this topic, there is a good and relevant video tutorial. And read text also. It is recomended that you read lets say last 10 pages also.

UHD is not continous. You will probably get something around 3520x1320 on 11...8bit lossless. That is under 3.5K in crop_rec menu.

If you will set FPS override, you will experience altered shutter speeds in crop mode, but you can fine tune even that, it works.

Your question is probably not being answered because this is experiments development page, but I can totally get you, I spent a few days just browsing through forum and didnt get it completely, but then after testing it for another few days... Latest developments here are even hard to follow and test, so huge respect to all of the people here, who actually move this developments forward.

It takes alot of time but it is defenitely worth it.

[theBilalFakhouri]

Now:
3072x1304 2.35:1 @ 23.976  &  2520x1440 @ 23.976  Is just working fine in 700D without any corrupted frames I didn't know why I got frames corruption before. Here the new settings in the spreadsheet and crop_rec module for testing.

I didn't go for 2560 instead of 2520 because the LiveView will be scrambled for only 40 pixels not a big deal, isn't it? Maybe if we could solve the distorted LiveView mystery the 2560 will be a great thing. 2520x1420 is True 16:9 in this case (in Camera 2520x1418 I don't know why but you can go up to 1440).

[Danne]

Welcome back Bilal!

[a1ex]

@Bilal,

Have you managed to test the source from reply #1824 on 700D?

Mind highlighting the differences between current and previous configuration?

[Levas]

@Bilal

If the 700d works about the same as the 6d, I'll think you can get even more vertical resolution in 2520 mode.
6d needs the B timer(6014) to be about 4F(hexadecimal) higher then the vertical resolution in 6804.

Can you try what happens if you make a small change in the 2520x1440 crop ?
Change the following in your 2520x1440 preset:
c0f06804 from 5BC0298 to value 6F80298
c0f0713c from 635 to value 6F9

This would give a vertical resolution of ~1753, only problem now is, you need to line out CMOS 7 value for this vertical resolution, to get normal frames again.
You can do this from within the crop_rec menu, use q button to enter crop_rec menu and change delta_head 4 timer.
Change delta head 4 timer 1 increment up and check your liveview/preview, if not correct, do another increment, check one increment at the time and see if your liveview/preview is back to normal again.
When liveview/preview is normal again, check the value in the menu you have for delta head 4.
Now to fix the crop_rec preset, you can change the value for CMOS 7 by adding the value of delta head 4.
Remember CMOS 7 value is hexadecimal and delta head 4 value is decimal.
So if you need delta head 4 value of 10. You can fix CMOS 7 by 0xaa9 + 10(decimal) = 0xab3

Curious if this works.

If this works, it means you probably also can get 3072 x 1440 resolution.

[theBilalFakhouri]

Thanks @Danne

@a1ex
Everything is working Just fine you are genius! Wow the new source code is a awesome and the way you can change the resolutions and mods is Perfect! I didn't notice any problem with all mods and there are no corrupted frames at all the only thing is you have to refresh LiveView manually as you mentioned.

The centering is almost if not PERFECT I will do more tests for this one (I noticed the centering change in x10 compared to other mods but not a big deal) .

Something is a bit annoying when pressing x5 mode the FPS stays 30fps I want to keep 2520 for not destroying LiveView and I want to increase vertical res I can get only 2520x1230 in 30fps and there is no way to decrease FPS to 23.976 for getting more vertical the solution is to exit x5 back to 1080p24 now the FPS is 23.976 with increased height up to 2520x1604 but the LiveView is scrambled (In this case I have to change X resolution to 2520).

It should be an option to set 23.976 FPS in x5 mode.

I tried also the max resolution in 3K (X res to 3072 Y res to 2000) I got 3072x1302 the same as what I got in adtg_gui using trial and error method. The only thing is missing is the higher SD write speeds and the lower bit-depths.

Alex.. You have unlocked the Canon cameras like a Monster! merciless!

@Levas
We don't have to override each register manually anymore! The new way is by Entering the X and Y Resolution you want and crop_rec will do the calculations for each register to get that resolution and if the resolution and FPS reach the limits it will not increase anymore and you will be in the safe frames area!

I can get with above way up to 2520x1604 @ 23.976 and the Max resolution in x5 16:9 is 2700x1500 @ 23.976.

[Levas]

I forgot about the experimental crop_rec Alex made for the 700d.
Have you tested max framerate in 720p mode ?
I got best results in 5x zoom mode in 720p50 canon mode.
In 1080p mode the 6d softbricks ad about 38 fps.

Tell us, can you get 75 fps, 100?

[a1ex]

No luck with high FPS on 700D from my side, but didn't try too hard. Long answer.

At least I'm starting to know where the limits come from.

[Levas]

Nailed it, 96fps, clean raw stream on the 6d 

It's done in 3x5 readout (Not in 5x zoom mode but in full view), resolution at 96 fps is 1440 x 360 -> which after correction for the squeeze factor, results in 1440 x 600 resolution.
Got it working by lowering some image registers, only the first will do. Added the other three registers later on for trying to lower A timer, but that didn't work

Code Select Expand


        case 0xC0F08184: return 0x1;
        case 0xC0F08188: return 0x1;
        case 0xC0F08194: return 0x1;
        case 0xC0F08198: return 0x1;


Two examples I shot today, 1440x360 base resolution at 96 fps, upscaled to 1920 x 800 on a 25 fps timeline:
I see the youtube compression does horrible things, better download the files for better view  Probably because 360p is only available at start, now 1080p available.
https://drive.google.com/open?id=1gbBmBiJ59Z75UeKc8tuTKb05a3gcwKfZ
https://drive.google.com/open?id=1qHHEr9C2PPEhHcB20VbUbDuBbOCJa2K4

Here are 2 framegrabs from DaVinci Resolve

[a1ex]

Awesome!

Mind sharing the full register configuration, or source code?

If you weren't able to lower timer A, I'd expect the horizontal resolution to be full width (1824 iirc).

You may also use other line skipping factors on the Y direction: 3, 5, 7, 9...

[Levas]

Have to go to work now, will send source later.

[Danne]

 
@Levas. Amazing, amaaaazing!!

[theBilalFakhouri]

Getting 5D3 video quality (No alisaing!) in 700D

Let's start with some samples: (There are no video processing or scripting in all samples 1736x736 2.35:1 @23.976 )





Video:

Link

Real life video samples: (I added sharpness more than usual for YT compression)

Link

This another and better example 1280x720 (Actually 1280x2160 in 1x3) @ 23.976 for better recording times:

Original file: Download

A close up look:




Details:
1x3 Sampling (Binning/Line skipping whatever) it was only for 5D3 I tried to get it using adtg_gui about three months ago and it worked and it was perfect with no aliasing like the samples above the problem was too wide aspect ratio 1736x386 (actually 1736x1160 but you have to stretch it to get the correct aspect ratio) I didn't realize I can increase the vertical resolution so I can get more vertical I was thinking 1x3 like 3x3 nothing to do with resolution..

Until a1ex made the new way for using crop_rec you can select a binning mode:



And X , Y resolution you want:


I thought okay let's make a try to increase vertical in 1x3 and it worked then I realized I was wrong and there is no vertical lines skipping so you can get the original height in sensor which is stretched 3478 pixels. To get 1736x736 2.35:1 the vertical should be 2220 in Y res you actually shooting 1736x2220 @ 23.976 FPS then stretch it down in post to get the right aspect which is 1736x736.


The problems are the write speed now about 80mb/s in 14-bit lossless (1736x736) so no continuous recording but this is the small problem and with some more tries to get higher SD writes speed and using Analog gain to get 10-bit lossless working in increased resolution will become not a big deal.

Annoying one: LiveView is very stretched! you can use ML Framing instead which is slow unfortunately.

@Levas
Nice! as a1ex said no much luck on 700D. Great Job!

[dfort]

Amazing developments. One of the biggest drawbacks to these cameras, aliasing, has been conquered!

I just had a facepalm moment. At first I thought that the 1x3 was reading every pixel horizontally and skipping vertically. That's because I'm more familiar with the way image sizes are expressed. Take 1920x1080 for example, always horizontal size first by vertical size second. The way ML is showing binning/skipping patterns is line by column.

For example, the mv720 pattern is described as:

Code Select Expand

sampling        5x3 (read 1 line, skip 4, bin 3 columns)
So that means sampling 5 vertically by 3 horizontally.

I haven't taken a close look at the crop_rec 1x3 pattern on the 700D yet but I'll update this post when I do.

[EDIT]

Code Select Expand

Block: RAWC
...
      sampling        1x3 (read every line, bin 3 columns)
...
Block: VERS
...
  String: 'crop_rec built 2018-09-15 16:23:15 UTC; commit c7130e1 on 2018-09-13 14:55:07 UTC by Dan: crop_rec: 700D test module from a1ex '

[/EDIT]

Ok--back to why 1x3 is working so well. All of the ML ported cameras are blending (binning) the pixels that are lined up horizontally but only the 5D3 bins the pixels that are lined up vertically (columns) so unless you are using a 5D3 you are probably going to experience some pretty serious aliasing, especially with the mv720 setting that samples every 5th pixel vertically. Now a head scratching moment--does the 5D3 always bin the columns even when in mv720 mode?

...I now have very strong reasons to believe that 5D3 does a 3x3 binning in 1080p, and a 5x3 binning in 720p (5 lines, 3 columns).

(if you shoot 720p on 5D3, the desqueezing factor - to correct the aspect ratio of your footage-  is therefore exactly 5/3 = 1.67x)

Back to what this means for non-5D3 users is that we should be able to get the same 5D3 raw image quality by shooting 1x3 then correct the aspect ratio of your footage in a way that it smoothly blends--no pixel skipping.

[a1ex]

Very nice! I was used to Octave notation (line/column) when writing the binning notation (5x3 etc), but relating these to resolution seems more logical; too late to swap them?

Now to figure out how to use the Eeko routines in LiveView and compress the footage to 3x3 (or at least 2x3) before writing it to card.

Binning: I believe all cameras are either reading out every column, or binning 3 columns of the same color. Some models apparently can bin more than 3 columns. Column binning is always configured from CMOS registers (model-specific).

[Danne]

Wow, bilal, a1ex. Epic. 700d, how about the eosm? Closely related registers?

[a1ex]

Likely identical; please give it a try (enable the is_700D flag in the experimental source from #1824 and let me know how it goes). 650D might work as well. 100D might need minor tweaks or might work out of the box, too. Need to check the CMOS registers for differences, as these are usually model-specific.

Code Select Expand


    /* default FPS timers are the same on all these models */
    if (is_EOSM || is_700D || is_650D || is_100D)


If timer A ends up too small (broken image or missing data on the right side), adjust it here:

Code Select Expand


    if (is_700D)
    {
        target_timerA = (272 + target_xres + skip_left) / column_factor;
    }




I forgot a crosshair in that bleeding-edge crop_rec source (used it for verifying the centering formulas)

[dfort]

EOSM is not happy using the settings @theBilalFakhouri recommends for the 700D:

Code Select Expand

ML ASSERT:
0
at crop_rec.c:327 (get_default_skip_top), task PropMgr
lv:0 mode:3

PropMgr stack: 17ce20 [17cf68-17bf68]
0xUNKNOWN  @ bac08:17ce90
0x00B2666C @ b26d20:17ce88
0x0009EBB4 @ b26c50:17ce50
0x0009E558 @ 9ec10:17ce20

Magic Lantern version : Nightly.2018Sep15.EOSM202
Mercurial changeset   : c7130e1e4c49 (crop_rec_4k_700D_test) tip
Built on 2018-09-15 17:02:43 UTC by rosiefort@RosieFoComputer.
Free Memory  : 226K + 3266K

I was used to Octave notation (line/column) when writing the binning notation (5x3 etc), but relating these to resolution seems more logical; too late to swap them?

Probably not necessary. Besides, it makes the "desqueezing factor" super easy to figure out -- 5/3 = 1.67x

[a1ex]

@dfort: enable the is_700D flag for the EOS M, or turn these conditions into is_700D || is_EOSM.

@Bilal: in the emulator I've got a vertical resolution limit of 2306 pixels for 1080p24 1x3; wondering why you've got 2220 as the upper limit (this screenshot). I've got 2200 in 25p.

Code Select Expand


[crop_rec] source: 1736x1160 @ 0.000 fps  # ignore this number; emulator bug
[crop_rec] target: 1736x2306 @ 23.976 fps
[crop_rec] ylimit: 1160,2306
[crop_rec] timerA: 528 -> 520
[crop_rec] timerB: 2527 -> 2566
[crop_rec] delta : X+0 Y+1146 [z=0]

[crop_rec] source: 1736x1160 @ 0.000 fps
[crop_rec] target: 1736x2200 @ 25.000 fps
[crop_rec] ylimit: 1160,2200
[crop_rec] timerA: 640 -> 520
[crop_rec] timerB: 2000 -> 2461
[crop_rec] delta : X+0 Y+1040 [z=0]


May I ask what timer values did crop_rec pick for your preset? Or maybe you tweaked the 260 from these formulas?

Code Select Expand


    desired_fps_x1000 = (uint64_t) fps_main_clock * 1000ull
                      / (uint64_t) target_timerA / (desired_yres + 260);
    ...
    int max_yres = (target_timerB - 260) & ~1;


[theBilalFakhouri]

@a1ex Yes this is correct but 2220 is stable and it's the closest number to get 1736x738 2.35:1 @23.976FPS (in 2306 you can get 30 more pixels not something huge) if I used the full 2306 I got some messages and black frame/s:

Code Select Expand


ML ASSERT:
0
at mlv_lite.c:2695 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1b1150 [1b11e0-1b01e0]
0x0007F5D0 @ b2328c:1b1180
0x0007EF78 @ 7f62c:1b1150

Magic Lantern version : Nightly.2018Sep11.700D115
Mercurial changeset   : d556a1022396+ (crop_rec_4k)
Built on 2018-09-11 08:15:57 UTC by Bilal4Fakhouri@WORKS.
Free Memory  : 124K + 3142K



Code Select Expand

ML ASSERT:
((mlv_vidf_hdr_t*)slots[slot_index].ptr)->frameNumber == (uint32_t) slots[slot_index].frame_number - 1
at mlv_lite.c:3421 (raw_video_rec_task), task raw_rec_task
lv:1 mode:3

raw_rec_task stack: 1acd10 [1acdd0-1abdd0]
0x0007F5D0 @ b31970:1acd40
0x0007EF78 @ 7f62c:1acd10

Magic Lantern version : Nightly.2018Sep11.700D115
Mercurial changeset   : d556a1022396+ (crop_rec_4k)
Built on 2018-09-11 08:15:57 UTC by Bilal4Fakhouri@WORKS.
Free Memory  : 124K + 3139K


I don't like to push it to the limits,, this is safer

[theBilalFakhouri]

May I ask what timer values did crop_rec pick for your preset? Or maybe you tweaked the 260 from these formulas?

Code Select Expand


    desired_fps_x1000 = (uint64_t) fps_main_clock * 1000ull
                      / (uint64_t) target_timerA / (desired_yres + 260);
    ...
    int max_yres = (target_timerB - 260) & ~1;


timer A=520
timer B=2566

I didn't tweak anything

[a1ex]

Okay, got it. These timer B limits are difficult to find because they are not hard (you don't see the image breaking as soon as you cross some limit, like it happens with timer A). Vertical resolution can usually be pushed further (i.e. closer to timer B) in low FPS modes. Not sure what's the best formula to use here. Writing down various resolutions and their outcome (stable or not) should reveal a pattern.

Selecting a higher margin (i.e. using 280 or 300 instead of that 260) may be a better option for high FPS modes, but might be suboptimal for e.g. full-res LiveView.

I did not try to record anything during my tests, only tried to get clean grayscale preview in mlv_rec.

[Levas]

Did think about this solution for aliasing too, it's just a matter of setting the line skipping factor to 0 (register 800c) in non zoom mode and expand the vertical resolution.
But because it's 3 times more data that needs to be recorded, so no continuous recording...I didn't see any use in it, hell I was just getting continuous recording times, because of the lossless development 

But I must say, the 1x3 files look a lot better, much more detail.
Can you share some sample DNG's of these examples. Both an 1x3 file and 3x3 dng file from the same subject ?
Curious how much the difference in detail is in a raw editor.