crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[MichaelVito]

With crop_rec_4k.2018Jun20.5D3123, though I've got mlv_snd enabled and turned on, and there are audio meters in the gui, I'm not getting audio recorded with clips. If I record with just mlv_lite loaded, I get a 30 second wav file of static regardless of clip length with mlv_dump. If mlv_rec is also loaded, any clips recorded with mlv_rec or mlv_lite don't generate wav files with mlv_dump or mlrawviewer.

[g3gg0]

resolved, thanks. the MLV file is fine - it was a merge error in mlv_dump which caused the WAV header to be skipped.

[togg]

I was wondering if this module could also basically being used to take full resolution silent pictures with fast shutter speeds. Never tested it until now! Maybe I should

[MichaelVito]

Thanks for the update. Checked sound with crop_rec_4k.2018Jun23.5D3123:

Clips recorded with mlv_lite are now good. I've got WAV files for 14-bit uncompressed and 14-bit lossless via mlv_dump. Yay!

Clips recorded with mlv_rec (14-bit uncompressed) still don't create a WAV with either mlv_dump or mlrawviewer.

Clips recorded with mlv_lite (14-bit uncompressed) and previewed with mlrawviewer playback video but still don't create a WAV. Now that I think about it, I never tested the earlier mlv_lite + mlv_snd builds to see how mlrawviewer would handle sound. Is a break in compatibility expected?

[g3gg0]

Clips recorded with mlv_rec (14-bit uncompressed) still don't create a WAV with either mlv_dump or mlrawviewer.

forget about mlv_rec, i will drop it soon. at least this is my goal

Clips recorded with mlv_lite (14-bit uncompressed) and previewed with mlrawviewer playback video but still don't create a WAV. Now that I think about it, I never tested the earlier mlv_lite + mlv_snd builds to see how mlrawviewer would handle sound. Is a break in compatibility expected?

did not try this tool, mainly worked with mlvfs
there might some things be different now, yeah. block order for example.
will check tonight.

thanks for feedback

[IDA_ML]

forget about mlv_rec, i will drop it soon. at least this is my goal

Hello g3gg0,

First of all, let me thank A1ex and you for your invaluable work on further improving 4k_crop recording. 

I briefly tested the June 23-rd build with the 5D3 of a friend of mine and here is what I found:

1) Sound works synchronously only with the 1920x1080 1:1 and 3x3 modes of the mlv_lite module when frame rate is set to 24 fps;

2) As soon as mlv_lite is switched to 3K 1:1 there is no synchronicity between sound and video - video lags behind by a few tenths of a second regardless of the resolution and bit rate set.  The lag is always different and clip length dependent.  However, there is no length proportional dependence.

For this reason, it may not be a good idea to drop mlv_rec before sound synchronicity is fixed with mlv_lite.  Maybe, returning the old condition, when mlv_rec was generating a synchronous sound file at 10/12/14 bits uncompressed is a better solution for now. As reported above, right now sound does not get recorded at all with mlv_rec.

Keep up this most important work!

[g3gg0]

hi,

that sync issue was always only 1-2 frames on my tests.
felt like it is quite static, thus is added the option to mlv_snd to correct this offset.
are you sure that mlv_rec was always in sync?

about the mlrawviewer issue, i suspect that the header field for audio frames could be related as it is zero.
but being set to zero means "autodetect" and the player should detect that on its own (as with early mlv_rec headers)

edit:
please check if toggling "sync beep" in mlv_lite menu helps with the variance

[MichaelVito]

Not to get off topic, but just for a reference point, is the original mlv_rec + mlv_snd implementation used in the Nightly builds a solid sync lock? I'd always assumed it was, but never tested and haven't been using it for anything where a variance would have been critical. To further confirm, was this same implementation used in crop_rec on steroids up until 2018 February, when work on expanding mlv_snd support began?

I totally understand getting things streamlined, eventually unifying everything under mlv_lite when sound issues are ironed out, and happy to test as many builds as it takes. But as IDA_ML says, it's helpful to have a fallback to legacy mlv_rec + mlv_snd for when you need it.

[a1ex]

If anyone wants to review the code:

- old implementation from ErwinH et al (crop_rec_4k_mlv_lite_snd): latest build on March 10

- new implementation from g3gg0 (crop_rec_4k_mlv_snd - started back in Feb. 2017): latest build currently on the Experiments page

From what I could tell, there is some divergence between them, although I'm pretty sure Erwin (7c51597 - 5f4ed21, October 2017) started from g3gg0's implementation (4a37b0e - 1ed4731, February-July 2017) or at least looked at it.

Other than a few tests back in February, I didn't record any raw videos with sound. Would be interesting to know whether the issues mentioned above were present in the old implementation (I can test it, but would be time-consuming and I'm not good at "seeing" the sync issues anyway).

I still have a MLV from back then, recorded with mlv_lite + mlv_snd; it does play in MlRawViewer. Some metadata:

Code: [Select]

File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 6497533523640690944
    FPS         : 25.000000
    File        : 0 / 0
    Frames Video: 164
    Frames Audio: 39
    Class Video : 0x00000021
    Class Audio : 0x00000001
Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 2.995000 ms
    Res:  1920x1290
    ...
Block: RAWC
  Offset: 0x000000e8
  Number: 2
    Size: 32
    Time: 3.005000 ms
    raw_capture_info:
      sensor res      5760x3840
      sensor crop     1.00 (Full frame)
      sampling        3x3 (bin 3 lines, bin 3 columns)

P.S. Here's my attempt to fix the audio sync issues (in Erwin's implementation); if that worked, it should probably be included in the newer version as well.

[g3gg0]

thanks didnt notice that the fixes weren't merged.
will look into them and merge into mlv_snd

[IDA_ML]

Hello all, following this exciting development,

Now that I had the chance to have the 5DMkIII for a few days in my hands, I decided to test the new June 20-th Experimental build from the Experimental download page and thought, it might be useful if I provide some feedback for those who use the 5D3 for shooting RAW video.  Here are my first impressions:

Things that I liked:
=============

1) Compared to the previous March 31 build, this one is very stable.  No more crashes, glitches, battery pulls, assert logs, etc.  It maintains excellent stability even when I switch modes.  One of the coolest features is that it is possible to quickly switch from video to photo mode just by flipping the video/photo switch and the mode dial of the camera from M to Av.  This can be done in fractions of a second and every time you switch back and forth from one mode to the other, all settings for both modes are stored and maintained.   Thus, when you film video and you quickly need to take a few stills or vice versa you can do this in a very fast and hassle free way without missing important moments.  Yesterday, I was filming the performance of my daughter's choir and I did this photo/video mode switching several times during the concert, without a single glitch.  Also, when filming FHD video with mlv-lite, I switch from 1920x1080 3x3 to 1:1 all the time.  This allows me to switch my EF 35/F2 IS lens from 35mm to 105mm in a very fast way, without swapping lenses or using slow zooms.  This is equivalent to the "Movie Crop Mode" on the 100D and yesterday, it worked perfectly on the 5D3 too, without a single glitch.   EXCELLENT WORK, A1ex !!!

2) The build is very convenient to work with.  In all video modes, in-camera previews work very well and scale nicely when resolution is changed in the 4K-crop recording modes.  This is very useful for correct framing during filming.  The previews are nice and bright even if bit rates are changed from 14 to 11 ... 8.  The feature that I absolutely love is that now focusing at 5x and 10x magnification works almost instantaneously.  This feature is extremely important especially when using fast primes for video shooting.  A quick and precise focus check and adjustment before pressing the record button is a must and it works very well on the 5D3 now, both - with half-shutter and back-button focusing.   I did not have a single misfocused clip during yesterday's filming.  Also in-camera playback works quite well and is fully usable, despite the fact that you may end up with a corrupt margin at the bottom and the right of the Live View screen if in the 3K 1:1 mode you reduce the resolution. 

3) Another cool feature concerns photo mode with the "HOLD" option activated in the Canon menu.  Immediately after a still photo is taken, it stays on the screen as usual but a tiny RAW histogram, focus peaking dots and under and overexposed areas appear on it.  In this way, just by taking a look at the picture, you know immediately if it is properly exposed and focused.  Very handy! 

4) All files that I recorded work flawlessly with MLVFS and open nicely in MLVApp and MLVProducer.

Things that I didn't like:
=================

These concern mostly my favorite 3K recording mode with mlv-lite.  There are two reasons why I use this mode on the 5D3 more often than any other modes when shooting  RAW video:

A. The resolutions of 3072x1920 or 3072x1728 (16:9) at 24 fps, 12  and 8 ... 11 bits lossless provide the highest video quality that I have ever seen even after upscaling to 4k.  Period.

B. At its maximum horizontal resolution of 3072p, this mode provides a crop factor of 1.87 which is reasonably small and greatly increases the functionality of the available arsenal of wide angle and normal focal length lenses for video shooting.  A 50/1.4 Sigma Art lens is converted into a 94/2.2 lens which provides a breath taking vision in terms of isolation of the main subject from the background, bokeh, smooth tonal transitions, etc.  At 12 mm focal length, my old Sigma 12-24 provides very impressive landscape videos at 3072x1728 resolution and 24 fps.  Fine detail and colors are just fantastic. 

There is one major limiting factor of the 3K mode though - the recording time is too short to be usable.  With a well lit scene (0.0 histogram value), Frozen LV and 8 ... 11 bits lossless I barely get 60 frames recorded with sound at 24 fps.  If I underexpose by one stop, I get about 85 frames and that's it.  If the scene contains a lot of highlights, the number of recorded frames can drop to 25 frames which is just 1s of recording time.  These data concern the fastest x1066 CF cards.  Of course, I could lower the resolution to say 2560x1440 where I can get continuous recording with sound and I do this all the time.  However,  this increases the crop factor.  Also, I could stay with 3072p resolution and reduce the vertical resolution but then I lose this fantastic full-screen vision and the video starts looking like a ribbon.  So, what is the solution?  In my opinion, there are two ways to solve the problem - through implementing the good old card spanning feature to crop recording with mlv-lite or through overclocking the CF card interface as this was successfully done with the 100D.  Or both   I think, more work in this direction is definitely worth considering.  A1ex, Danne, what do you think?

Things that need a fix:
================

Those concern the sound sync issues discussed above.  I hope, they will be fixed soon.  I never had any synchronicity issues with the old implementation of ErwinH applied to mlv-lite or the mlv-rec feature from the Nightly builds.  So, I guess, going back to these old sound implementations should fix the sync issues, at least for now, until G3ggO irons out his new implementations. 

Bottom line:
=========

The June 2018 builds for the 5DMkIII have turned this camera into a very powerful  RAW video recording machine which in my opinion is stable and convenient enough for serious work.  After the bugs mentioned above are fixed and problems are ironed out, I am confident that this camera will become the preferred choice for many professional videographers.  I wish, I had this stability and functionality in my 7D.

I have to return the 5D3 now and cannot test more features and modes but I will test the June 23-rd build for the 100D and will report on it as soon as I can.  I strongly encourage all 5D3 owners in this forum to run their own tests and report their findings here.  This will help the developers fix bugs and solve problems easier and faster, so more 5D3 owners can take advantage of this fantastic new functionality and stability of their camera.
 

[g3gg0]

thanks a lot for your feedback.

first of all, i missed the point that the branch "crop_rec_4k_mlv_snd" and "crop_rec_4k_mlv_lite_snd" are two different ones
so i felt safe to continue work on mlv_snd in crop_rec_4k_mlv_snd, thinking it was the right one.
my wrong, sorry.

i now manually pulled the important things from that - lets call it - bugfix branch.
now the mlv_snd module is being called right from the vsync cbr of mlv_lite, giving minimum chance for desynchronization.
(previously even the module load order could have introduced a frame delay)

@a1ex:
did i miss some important thing to graft?
as said, i want to drop mlv_rec with this branch as soon it is stable.
filmed a few TiB now with mlv_lite and close to zero with mlv_rec.

@all:
please feel free to test the latest build for audio sync issues.
the mlv_snd menu now is in the raw recording menu in case you miss it.
within that menu there is an option to configure how many frames the audio should be delayed.
if everything is as i expect it, it should be ideal at 1 - make sure you didnt accidentally set it to any other value.

[a1ex]

first of all, i missed the point that the branch "crop_rec_4k_mlv_snd" and "crop_rec_4k_mlv_lite_snd" are two different ones

Well, at that time, Erwin's version was simpler and worked reasonably well, but at the same time I didn't want to exclude yours by picking the other. So I tried to keep both branches in sync, to decide later.

filmed a few TiB now with mlv_lite and close to zero with mlv_rec.

I've filmed close to zero with both (actually one clip during the holidays -> used H.264, since it was half an hour long and I had a 32GB card).

If we drop mlv_rec, I'd suggest renaming mlv_lite back to mlv_rec (feature-wise, it's no longer lite anyway) and port the remaining important features (card spanning, maybe exfat, etc). I'm also thinking to undo the preallocation experiments, as I've got a lot of trouble from them anyways.

Will check later.

[MichaelVito]

Thanks for rounding up all those changes!

Just did a quick check with crop_rec_4k.2018Jun27.5D3123. Now mlv_lite 14-bit uncompressed plays nice with both mlv_dump and mlrawviewer. I'll do a deeper check for other modes and sync soon.

If the plan is to retire mlv_rec, I won't keep asking about it. But you may want to just remove it now, or at least note that sound no longer works.

I checked some of the older builds, and seems mlv_rec sound has been broken since the very first mlv_lite + mlv_snd build. With February 2, recording with mlv_rec I get the message "audio failed to stop, state 0" after hitting stop and no WAV is created with either mlv_dump or mlrawviewer. With March 10, the last ErwinH version, I get the error message as well as an assert referring to mlv_snd_vsync.

With the g3gg0 builds, the errors stopped, though sound is still broken. If sacrificing mlv_rec is the only way to get mlv_snd integration with mlv_lite, I'm ok with that. Maybe a memorial is in order? (^^)

[a1ex]

Back in February, mlv_rec worked fine with mlv_snd, but you do have to load only one module at a time (either mlv_rec or mlv_lite, not both). There's no check for that though, just "tribal knowledge".

[g3gg0]

With the g3gg0 builds, the errors stopped, though sound is still broken. If sacrificing mlv_rec is the only way to get mlv_snd integration with mlv_lite, I'm ok with that. Maybe a memorial is in order? (^^)

to be correct, crop_rec_4k_mlv_snd *is* the branch i started initially 17 months ago.
its goal was to make mlv_snd integrate nicely into mlv_lite, so mlv_rec can get dropped.
it just wasnt finished and certainly still is not.

so there are no "g3gg0 builds" or anything else.
the memorial is "MLV"

[Levas]

I'm working on Crop_rec presets for 6d and aiming for 25fps presets mostly, but how exact does this 25 fps frame rate have to be and does it have influence on audio ?
For instance I have presets with 24.995fps and 25.005fps, is this good enough or do I have to fiddle with A and B timers to get exactly 25.000fps ?
And is ML audio with MLV_lite linked to exact frame rate of the A and B timer settings, or is it linked to PAL or NTSC setting ?

[a1ex]

Usually it's easy to get exact values for 25p; you can use a small Python script to do the math. Something like this:

Code: [Select]

main_clock = 24e6
desired_fps = 25.0

for timer_a in range(500, 600):
    timer_b = int(main_clock / timer_a / desired_fps)
    actual_fps = main_clock / timer_a / timer_b
    print timer_a, timer_b, actual_fps - desired_fps

I'd like to automate this step in the source code, rather than hardcoding all these FPS timers, btw.

Audio sync shouldn't depend on FPS, at least in theory. Audio gets recorded at whatever rate is selected in the menu (44 kHz etc); furthermore, all frames are timestamped.

[Levas]

I see I have still not the right shutter time with the crop mode build I'm using.
It's probably going wrong in the calculations for shutter blanking, the piece of code for shutter blanking is probably optimized for 5d3 or 700d in the code I'm using.
1/50th on top display gives me 1/35th shutter time in ML.
1/100th on top display gives me 1/50th shutter time in ML.

Does anybody know where I have to fix this error in shutter blanking calculations ?

Code: [Select]

/* adapted from fps_override_shutter_blanking in fps-engio.c */
static int adjust_shutter_blanking(int old)
{
    /* sensor duty cycle: range 0 ... timer B */
    int current_blanking = nrzi_decode(old);

    int video_mode = get_video_mode_index();

    int fps_timer_b_orig = default_timerB[video_mode];

    int current_exposure = fps_timer_b_orig - current_blanking;
   
    /* wrong assumptions? */
    if (current_exposure < 0)
    {
        return old;
    }

    int default_fps = default_fps_1k[video_mode];
    int current_fps = fps_get_current_x1000();

    dbg_printf("FPS %d->%d\n", default_fps, current_fps);

    float frame_duration_orig = 1000.0 / default_fps;
    float frame_duration_current = 1000.0 / current_fps;

    float orig_shutter = frame_duration_orig * current_exposure / fps_timer_b_orig;

    float new_shutter =
        (shutter_range == 0) ?
        ({
            /* original shutter speed from the altered video mode */
            orig_shutter;
        }) :
        ({
            /* map the available range of 1/4000...1/30 (24-30p) or 1/4000...1/60 (50-60p)
             * from minimum allowed (1/15000 with full-res LV) to 1/fps */
            int max_fps_shutter = (video_mode_fps <= 30) ? 33333 : 64000;
            int default_fps_adj = 1e9 / (1e9 / max_fps_shutter - 250);
            (orig_shutter - 250e-6) * default_fps_adj / current_fps;
        });

    uint32_t (*reg_override_func)(uint32_t, uint32_t) =
        get_engio_reg_override_func();

    /* what value we are going to use for overriding timer B? */
    int fps_timer_b = (reg_override_func)
        ? (int) reg_override_func(0xC0F06014, fps_timer_b_orig - 1)
        : fps_timer_b_orig;

    /* will we actually override it? */
    fps_timer_b = fps_timer_b ? fps_timer_b + 1 : fps_timer_b_orig;

    dbg_printf("Timer B %d->%d\n", fps_timer_b_orig, fps_timer_b);

    int new_exposure = new_shutter * fps_timer_b / frame_duration_current;
    int new_blanking = COERCE(fps_timer_b - new_exposure, 10, fps_timer_b - 2);

    dbg_printf("Exposure %d->%d (timer B units)\n", current_exposure, new_exposure);

#ifdef CROP_DEBUG
    float chk_shutter = frame_duration_current * new_exposure / fps_timer_b;
    dbg_printf("Shutter %d->%d us\n", (int)(orig_shutter*1e6), (int)(chk_shutter*1e6));
#endif

    dbg_printf("Blanking %d->%d\n", current_blanking, new_blanking);

    return nrzi_encode(new_blanking);
}

extern WEAK_FUNC(ret_0) void fps_override_shutter_blanking();

static void FAST adtg_hook(uint32_t* regs, uint32_t* stack, uint32_t pc)
{
    /* This hook is called from the DebugMsg's in adtg_write,
     * so if we change the register list address, it won't be able to override them.
     * Workaround: let's call it here. */
    fps_override_shutter_blanking();

    uint32_t cs = regs[0];
    uint32_t *data_buf = (uint32_t *) regs[1];
    int dst = cs & 0xF;
   
    /* copy data into a buffer, to make the override temporary */
    /* that means: as soon as we stop executing the hooks, values are back to normal */
    static uint32_t copy[512];
    uint32_t* copy_end = &copy[COUNT(copy)];
    uint32_t* copy_ptr = copy;
   
    struct adtg_new
    {
        int dst;
        int reg;
        int val;
    };
   
    /* expand this as required */
    struct adtg_new adtg_new[10] = {{0}};

    /* scan for shutter blanking and make both zoom and non-zoom value equal */
    /* (the values are different when using FPS override with ADTG shutter override) */
    /* (fixme: might be better to handle this in ML core?) */
    int shutter_blanking = 0;
    int adtg_blanking_reg = (lv_dispsize == 1) ? 0x8061 : 0x805F;
    for (uint32_t * buf = data_buf; *buf != 0xFFFFFFFF; buf++)
    {
        int reg = (*buf) >> 16;
        if (reg == adtg_blanking_reg)
        {
            int val = (*buf) & 0xFFFF;
            shutter_blanking = val;
        }
    }

    /* some modes may need adjustments to maintain exposure */
    if (shutter_blanking)
    {
        shutter_blanking = adjust_shutter_blanking(shutter_blanking);
    }


[a1ex]

Guess: default_timerB needs to be adjusted (currently it's hardcoded for 5D3).

Or, the code needs to be updated to autodetect it (maybe save its value prior to overriding).

[Levas]

I did adjusted the sensor timing table, if that is what you mean ?
Just to be sure, that third column with 29.987fps, is 1080 3x3 binning in NTSC right , or is it 5x zoom ?

Code: [Select]

/* from SENSOR_TIMING_TABLE (fps-engio.c) */
/* hardcoded for 6d */
const int default_timerA[] = { 0x221, 0x27F, 0x221, 0x27F, 0x207 };
const int default_timerB[] = { 0x7a2, 0x63f, 0x61b, 0x31f, 0x334 };
const int default_fps_1k[] = { 23982, 25000, 29978, 50000, 59964 };

/* adapted from fps_override_shutter_blanking in fps-engio.c */
static int adjust_shutter_blanking(int old)
{
    /* sensor duty cycle: range 0 ... timer B */
    int current_blanking = nrzi_decode(old);

    int video_mode = get_video_mode_index();

    int fps_timer_b_orig = default_timerB[video_mode];


[Levas]

Or, the code needs to be updated to autodetect it (maybe save its value prior to overriding).

If that's the case, that means it's also not working properly in the Crop_rec_4k build, for the 5d3, on the download page, right ?

[a1ex]

Out of ideas then; you could try enabling the debug messages from that function.

It works fine on 5D3, but with all these hardcoded values, it's hard to port to other models (that's why I prefer changing it to something more generic).

[Levas]

Wait, the 5d3 has max shutter time of 1/8000th and 6d 'only' has max shutter time of 1/4000th, is that info used somewhere in that piece of code ?

[a1ex]

In movie mode it has 1/4000 in Canon UI, but the hardware lets you set the exposure time to 2 units (that is, blanking set to timer B - 2). In microseconds, that would be 1 / (24e6 / timerA / 2). That gives 1/25000 in 50p (timer A = 480), "overclockable" to 1/30000 (timer A = 400), or close to 1/15000 in full-res LV (timer A = 792).

Without crop_rec, you should be able to get these fast shutter speeds from the Movie menu, Image fine-tuning.

With crop_rec, there is an option to remap the shutter speed range (Canon UI lets you dial from 1/4000 to 1/30, but you can choose to remap this range to whatever the hardware allows). Caveat: if you enable shutter remapping ("Full range" in the submenu), you will no longer get "round" shutter speeds, like 1/50; the remapping formula doesn't take care of that.

And yes, 1/4000 is hardcoded in the code you have pasted (250e-6; don't remember why I wrote it like that; 1.0 / 4000 would have been the same).