Magic Lantern Forum

Using Magic Lantern => Raw Video => Topic started by: smasry on June 22, 2017, 03:25:59 PM

Title: Possible (erratic) bug in crop_rec_4k with HDMI monitor/recorder
Post by: smasry on June 22, 2017, 03:25:59 PM
Hi, I filmed a fulll 90-minute dress-rehearsal, using the crop_rec_4k branch 2017Jun03.5D3123 on my 5D III, in regular (not with 5x zoom or crop mode) HD, 1920x1080, 23.976 fps, in 12-bit lossless mode on a Computerbay 1066x CF, 256Gb.

My settings were, manual white balance, 1600 ISO, f5.6 (fully manual lens), 1/50th exposure. As a backup, I used an Atomos Ninja 2 monitor to record the feed in ProRes. To make sure the backup ProRes was usable, I set up 4 LV display presets, using the INFO button to toggle between them. Modes 0 to 2 were permutations of global draw with zebras, histogram, focus peaking and level indicator. Mode 3 I reserved for HDMI recording, completely turning off global draw, to send a clean, though small, image to the Atomos.

I tested the above setting in full, recording a digital clock with a full battery and empty CF card, until the card was full, ensuring that the signal remained clean even beyond that point, until the battery itself died, letting the Atomos keep on recording, again as a backup. To make sure the signal stayed clean beyond the stopping of the RAW recording, I had clear overlays set 'on idle', again to send out a clean feed. I tested this twice, and was impressed that I would get nearly 100 minutes RAW recording, with a clean (not corrupted) MLV file, even though the card ran out of space.

In all tests, everything worked perfectly, and as expected. On the day of recording, I configured the camera in the same way, but noticed one problem: INFO would flip through all the modes correctly, and would then 'freeze' when entering mode 3, the global draw off mode. After displaying a frozen clean image, the mirror would promptly drop, with the LCD and upper mode display screens frozen. Only removing the battery helped. The good thing, however, is that the camera stayed in the final mode (mode 3), so I could record a clean feed in the Atomos.

I tried this several times, always with exactly the same result. What I didn't [want to] notice, is that every time the camera went into mode 3, the displayed image was darker (less exposed) than in the other three modes, for all the same settings. So, I recorded the show, and was horrified that both the RAW data (DNGs from the MLV) and that in the Atomos were hugely underexposed. I would guess that the footage actually recorded at ISO 800, even though 1600 is reported. For example, I used the Footage app to get a preview of a usable exposure, by increasing the exposure by 6, lowering the blacks, increasing shadows and boost settings.

( (

Running the file through [the latest mlv_dump] using 'mlv_dump -v' gives:

 MLV Dumper v1.0

Mode of operation:
   - Input MLV file: '/Volumes/Virtualisation HD/Inbox/MDBA 2/M18-1555.MLV'
   - Verbose messages
   - Verify file structure
   - Dump all block information
File /Volumes/Virtualisation HD/Inbox/MDBA 2/M18-1555.MLV opened
File /Volumes/Virtualisation HD/Inbox/MDBA 2/M18-1555.M00 not existing.
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 10251550851593754641
    FPS         : 23.976000
    File        : 0 / 0
    Frames Video: 134740
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 0.777000 ms
    Res:  1920x1080
      api_version      0x00000001
      _int_height      1318
      _int_width       2080
      _int_pitch       3640
      _int_frame_size  0x00493450
      bits_per_pixel   14
      bit_packing      0
      black_level      2047
      white_level      5586
      active_area.y1   28
      active_area.x1   146
      active_area.y2   1318
      active_area.x2   2078
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: RAWC
  Offset: 0x000000e8
    Size: 32
    Time: 0.790000 ms
Unknown Block: RAWC, skipping
Block: IDNT
  Offset: 0x00000108
    Size: 84
    Time: 0.801000 ms
     Camera Name:   'Canon EOS 5D Mark III'
     Camera Serial: '468D84D62F'
     Camera Model:  0x80000285
Block: EXPO
  Offset: 0x0000015c
    Size: 40
    Time: 0.810000 ms
     ISO Mode:   0
     ISO:        1600
     ISO Analog: 104
     ISO DGain:  0/1024 EV
     Shutter:    19983 ┬Ás (1/50.04)
Block: LENS
  Offset: 0x00000184
    Size: 96
    Time: 0.837000 ms
     Name:        '28-28mm'
     Serial:      ''
     Focal Len:   28 mm
     Focus Dist:  0 mm
     Aperture:    f/8.00
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: RTCI
  Offset: 0x000001e4
    Size: 44
    Time: 0.850000 ms
     Date:        18.06.2017
     Time:        15:55:52 (GMT+0)
     Zone:        ''
     Day of week: 0
     Day of year: 168
     Daylight s.: 0
Block: WBAL
  Offset: 0x00000210
    Size: 44
    Time: 7.147000 ms
     Mode:   9
     Kelvin:   4700
     Gain R:   482
     Gain G:   1024
     Gain B:   635
     Shift GM:   0
     Shift BA:   0
Block: VERS
  Offset: 0x0000023c
    Size: 150
    Time: 38.384000 ms
Unknown Block: VERS, skipping
Block: VIDF
  Offset: 0x3733d4fc00
    Size: 1761792
    Time: 5621353.807000 ms
   Frame: #134739
    Crop: 152x132
     Pan: 146x133
   Space: 32
Reached end of chunk 1/1 after 134755 blocks
Processed 134740 video frames

The DNGs extracted report ISO 1600 and the same black and white levels, obviously.

As it's a 237Gb file, I can't upload it, so I'll try to attach a screenshot from Footage, as well as an equivalent DNG, if the forum software lets me.

I'm afraid the footage itself can't be repaired; I'd love to be proven wrong by one of you, though! If this helps remove a bug from this experimental branch, that at least is for everyone's benefit.

As a close to this report, I have used the camera as normal, without an HDMI screen connected, and it has returned to its usual behaviour, not exhibiting freezing upon entering mode 3, nor dimming once it does so. Testing it with the screen plugged in again still works as it should. On that day, I could easily replicated this problem a dozen times, now it is just behaving as normal, and no matter what I do, I have no idea how to replicate the problem. Until further notice, I'm holding off from updating to the latest in this branch: Latest Build (2017-06-19 20:27).

I know I've overloaded this report, but the only other strange thing I've experienced with this release, which I've never had before, is an overlaid thin and colourful stripe pattern when I enter x10 zoom mode, if this is of any help. I'll try to attach this image too. These stripes grow faint and disappear after half a minute.

( (

I hope there is some help for my footage. Thanks in advance.
Title: Re: Possible (erratic) bug in crop_rec_4k with HDMI monitor/recorder
Post by: smasry on June 28, 2017, 03:43:29 PM
Hi guys, anyone have any suggestions for this, or have I posted in the wrong board?

Title: Re: Possible (erratic) bug in crop_rec_4k with HDMI monitor/recorder
Post by: a1ex on June 28, 2017, 04:37:40 PM
      black_level      2047
      white_level      5586

That means, you have used 12-bit lossless. Bit depth reduction, in this case, works by... darkening the image (

The vertical stripes in 10x are also known (not sure how to detect the zoom mode switch at the right moment), but should be harmless (it's the vertical correction data messed up, and only in a mode where you can't record from). They should disappear after a couple of seconds, as the correction data is updated continuously. This bug is not erratic at all; it's deterministic here (happens every time I change zoom in the affected mode).

Will look at the crash.
Title: Re: Possible (erratic) bug in crop_rec_4k with HDMI monitor/recorder
Post by: smasry on July 01, 2017, 06:36:50 PM
Thanks for the reply @a1ex. Even though I've been following your 10/12-bit and compressed threads for the past 8 months, and I knew that two bits (in case of 12-bit recording) were sacrificed, I didn't realise it was the high-end bits...

I've now gone back and re-read all your posts, trying to understand this better. To be honest I chose to record in 12-bit for two reasons: the hint printed in the menu suggesting 14-bit for ISO 100, and 12-bit for 100-1600. That, and I was greedy, hoping for a few extra minutes of video on the card.

I've shot a colour chart (in ISO 800), at the same lens aperture, on the same lens, in both 14- and 12-bit modes, and it's obvious that the 12-bit is darker. In Resolve (and I know it's quirky with ML footage, especially exposure) the 12-bit only looks the same when the exposure is upped by 0.4. In this limited test, though, I can't see any information loss, of which I seem to have plenty in my footage.

With this information, I played a hunch and can see why I messed up so badly. With my phone, I recorded the back of the camera (particularly the LCD) showing what I saw when setting up for the shoot. In 14-bit mode, the live view behaves as expected in all 4 of my display profiles. In 12-bit mode, it seems to be somehow 'virtualised'; the display is just as bright in all of my display modes with global draw on, but with a lag which simply isn't there in 14-bit mode. As I flip through the display modes, the  display briefly dims to an underexposed image, quickly returning to the bright and laggy display. Finally, when flipping to the display preset with global draw off, the display dims, displaying an underexposed image, but there is no lag, just as in 14-bit mode.

14-bit mode:

( (
( (

12-bit mode:

( (
( (
( (
( (

My takeaway from this is that I trusted the output of the 12-bit mode to be the expected result, thinking that the darkening in my 'clean' global-draw off mode was an error, when it was the other way around the whole time!

Playing with this also (errratically, can't work out how to reliably replicate) crashed the camera in the way I described in the original post, with a frozen LCD image and a 'Err 80' flashing on the status screen on top.

( (

( (

@a1ex, if you want me to, I can keep trying different permutations of actions to work out what causes the crash, it may take quite a bit of fiddling, just let me know.

And finally, I wasn't worried about the stripes in 10x view, I just noticed them for the first time in this build, and wondered if this may have something to do with it.

I'll go and play with flat-frame and dark-frame combinations to see if I can't slightly improve my recorded footage.

Many thanks Alex.