crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[a1ex]

Maybe something interesting in the Debug menu: red dot in front of "memory patches"

That's unrelated; it's simply telling you about some module (here, crop_rec) using ROM patches aka cache hacks (which are a bit fragile in the backend).

You can use this build to get a startup log with all the debug messages from Canon. Before the log gets saved, you should try to perform the sensor cleaning, so messages about it will get included:

https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/

[BBA]

@A1ex
Thanks a lot !
I will give it a try and report.

[GutterPump]

And I'm getting vertical stripes on the .dng exports.

I also noticed this issue with the 14 bits lossless feature.

Even  if i change the code in the raw2dng.c file :

Code Select Expand


      if (stripes_correction_needed)
    {
        printf("\n\nVertical stripes correction:\n");
        for (j = 0; j < 8; j++)
        {
            if (stripes_coeffs[j])
                printf("  %.5f", (double)stripes_coeffs[j] / FIXP_ONE);
            else
                printf("    1  ");
        }
        printf("\n");
    }
}

to

Code Select Expand

     printf("\n\nVertical stripes correction:\n");
        for (j = 0; j < 8; j++)
        {
            if (stripes_coeffs[j])
                printf("  %.5f", (double)stripes_coeffs[j] / FIXP_ONE);
            else
                printf("    1  ");
        }
        printf("\n");
    }

For constant vertical stripe fixs with mlv_dump. (Thanks to Danne for this tip)
The fix don't appear in the command line.

[a1ex]

To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).

[GutterPump]

Sorry a1ex didnt saw this message ! Thanks for the remind.

[Hans_Punk]

Ah yes...my fault for not acknowledging that info. I should have noticed the custom mlv_dump disclaimer.

Cheers A1ex / Danne & GutterPump for reply x

[BBA]

@A1ex
I don't know how to save the debug msgs after a "manually started" cleaning process:

Before the log gets saved, you should try to perform the sensor cleaning, so messages about it will get included:

When I switch the camera on, the "auto sensor cleaning" is set to "disabled";
The bottom red led is lit a while and then a few times (blocks of data written to the card ?)
It ended at 14:42
I went to the Canon SETUP3 menu to start a manual immediate cleaning
The camera made some noise as if taking 2 pictures then this greyed menu item became yellow again.
I then went to "auto cleaning" item to set it to "enabled"
I then went in the ML DEBUG menu and pressed SET button with cursor on DebugMsgLog
The bottom red led was lit a very little while (but I saw no file on the card created at that time = 14:46).
Finally I switched the camera off.

The following file is generated just when the camera gets started (gets the creation time = 14:42):

https://www.dropbox.com/s/mjickmertwial3y/DM-0001.LOG?dl=0

[a1ex]

Diagnosed the bricking issue - it was caused by a null pointer bug in ML

Only Apr04 is affected - both older and newer builds are safe.

Fortunately, it's easy to recover (at least in the two previous cases). One uint32_t at an arbitrary address (between 0 and around 60MB, with a non-uniform probability distribution) might be overwritten with 0xA5A5A5A5 (and eventually end up saved into ROM at camera shutdown, when Canon code saves camera settings).

@BBA: if you believe your camera might be affected by this bug, please send me a recent (bad) copy of your ROM, alongside with an older (good) copy.

If you don't have any backup, copy the oldest files from ML/LOGS you can find on a ML card (hopefully these are "last known good" configuration).

To get a fresh copy, simply delete ML/LOGS from a ML card, and it will be re-created at next startup.

[GutterPump]

I went to the Canon SETUP3 menu to start a manual immediate cleaning
The camera made some noise as if taking 2 pictures then this greyed menu item became yellow again.

This is normal, it's the same for me and for all 5DIII users.

infos here

[BBA]

@ Gutterpump
Thanks for your feed !

[Mefist]

I get a black DNG (((((

CMD the code is:
@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "c:\RAW\mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )

The code is wrong?

I have Windows 10 x64. I drag MLV to CMD, it immediately closes, creates a folder and it has black DNG

[hjfilmspeed]

@A1ex I got the nasty error explosion today using the April 6 build.
I think it was probably because I was using an odd resolution
3008x1280 or something like that. Pulled battery everything seem fine with the camera.
Tried again with 3072x what ever 2.35:1 is and it seemed to work ok.
Would you like me to report the bug.
Here is a link to what was on my card. Error codes and all.
https://www.dropbox.com/sh/8lmopv8z7p2i91f/AAA4saYTmLE7rFQ3t7nSUBCaa?dl=0

[GutterPump]

I get a black DNG (((((

Can you share a small part of your MLV file ? Or record a very short part of movie for get arround 10 frames.
Be sure you're using the good version of mlv_dump too.

[Mefist]

That's what I used and settings.

@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "c:\RAW\mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )

[beauchampy]

Just testing April 6th build on 123 with HDMI.

First up, it's a bit sketchy getting signal from the HDMI - have to power on, go into Canon menu, back out again, screen flashes all kinds of weird, live view then appears. First time I tried the camera crashed.

Second up, only getting 7 seconds at 1920x800 60p, GD off, 14-bit lossless, ISO 100. Data rate is low - only about 57mb/s

Just pulling out the HDMI getting 15 seconds. Data rate is more like 67mb/s.

Strange how plugging in the HDMI seems to lower the data rate by 10mb/s?

[a1ex]

Successfully unbricked the two cameras that had problems earlier in this thread! (ju64 and filipe), and also solved BBA's issue.

More cameras might be affected (actually up to 817); will add an automatic check to identify them in the next build. The side effects range from nothing at all (overwriting an unused memory location) to soft-bricking (recoverable) and anything in-between (e.g. BBA's issue). The bug can overwrite any setting (aka property in Canon code), or the data structure itself. My own 5D3 was also affected, without any obvious side effects though (as I don't know where the affected setting is used in Canon code).

I'll probably stop adding features for a while and will look into ways for preventing this class of bugs.

Strange how plugging in the HDMI seems to lower the data rate by 10mb/s?

It's not strange at all, as the camera has to work 6 times as much (1920x1080 vs 720x480) to create the HDMI image (strictly talking about the LiveView buffer). Most of the work is done by the image processing chips (external to the main CPU) and the DMA (EDMAC), so most of this load is on the memory bus.

The "Frozen LV" (previously known as Hacked preview) works by stopping LiveView buffers from being updated (therefore freeing the memory bus and a bit of CPU time).

[pc_bel]

@beauchampy

First up, it's a bit sketchy getting signal from the HDMI - have to power on, go into Canon menu, back out again, screen flashes all kinds of weird, live view then appears

Same for me, but the same thing happens to me with older (and more stable) builds... To get hdmi signal I often go to play button and back out again. In my case using a Zacuto EVF.

I get continuous recording with april6th build with ISO100 1920x960 50p GD off 14bit losseless with or without hdmi in 5Dmk3 113.

[beauchampy]

@beauchampy

Same for me, but the same thing happens to me with older (and more stable) builds... To get hdmi signal I often go to play button and back out again. In my case using a Zacuto EVF.

I get continuous recording with april6th build with ISO100 1920x960 50p GD off 14bit losseless with or without hdmi in 5Dmk3 113.

I didn't have much problem with previous builds. Might be because right now I'm using a Leica prime with no electronics, so that might be throwing things off a bit (camera probably thinking a lens not attached).

Just tried 50p and yes, getting continuous at a steady 79MB/s @ ISO 100, 14-bit lossless with HDMI, GD off.

Bit weird how the 60p mode seems to have a much lower MB/s (57MB/s with HDMI, 67MB/s without), so I'm guessing its something a little more complex than just write speed which is the issue there. a1ex / devs  if I can help test in any way, please do let me know.

Off to Bermuda tomorrow, so I'll do some more testing of this mode there. Thanks for the suggestion

[pc_bel]

Fantastic!!!

Be aware with ISO. For me 800 is where problems start. I can get only few seconds with 800 ISO. And as I report in post #148, I get a "cutted" image in the top of frame with paning movements... So no usable for serious work...

[goldenchild9to5]

@a1ex I've used the April 4th build so does that mean my 5D Mark III is affected as well?  I remember when using it red numbers were flashing on the top of my screen about memory being low.   

[Mefist]

I still do not get a good DNG. If you shoot as before then the DNG is good.

[reddeercity]

I still do not get a good DNG. If you shoot as before then the DNG is good.

Are the frames black in any of the App's you use ? e.g. Resolve , A.E. ACR  ? Or what ever you use

[hjfilmspeed]

@A1ex is it only 5Diiis that used April 4 build or could it be in the April 3 build too?

[a1ex]

Here's a Python script to check for sure (run it and follow the instructions):

Code Select Expand


# checks whether your camera might have been affected
# by a null pointer bug from the 2017Apr04 ML build.
# updated Apr10 2017 (ignore some false warnings)

from __future__ import print_function
import os, sys
from struct import unpack

def getLongLE(d, a):
   return unpack('<L',d[a:a+4])[0]

try:
    ROM = open(sys.argv[1], "rb").read()
except IndexError:
    print("usage:")
    print("1) Backup the contents of ML/LOGS from your card")
    print("2) Debug -> Dump ROM and RAM (to get a new ROM copy)")
    print("3) Run this command on the latest copy of your ROM:")
    print("   python %s ROM1.BIN" % sys.argv[0])
    raise SystemExit

# scan for the magic number 0xA5A5A5A5 that might have been
# written into ROM as a result of the null pointer bug
found = 0
ignored = 0
can_ignore = False
ignore_msg = ""
gaonisoy = False
for a in range(0, len(ROM)-4, 4):
    v = getLongLE(ROM, a)
    n = getLongLE(ROM, a+4)
    block_name = ""
    can_ignore = False
    if a >= 0xF40000 and a < 0xF80000:      # RING - that's where most errors were
        a0 = a & ~0xFFF                     # header of this block
        if getLongLE(ROM, a0) == 0xFFFF:    # active block? only check this one
            block_name = "active RING"
            if a == a0:
                print("Active RING block: %x-%x" % (a0, a0+0xFFF))
        else:
            block_name = "inactive RING"
            can_ignore = True               # ignore inactive RING blocks
    if a >= 0xF80000 and a < 0xFA0000:      # last debug log - ignore
        block_name = "last debug log"
        can_ignore = True
    if v == 0xA5A5A5A5 and n != 0xE5E5E5E5:
        print(hex(a), hex(v), block_name)
        found   += (0 if can_ignore else 1)
        ignored += (1 if can_ignore else 0)
    if v == 0x6E6F6167 and n == 0x796F7369:
        assert a == 0xC0004
        gaonisoy = True

print("\nErrors found: %d, ignored: %d." % (found, ignored))

if not gaonisoy:
    print("Doesn't look like a valid ROM1 dump.")
elif found > 0:
    print("Your camera might have been affected by the Apr04 null pointer bug.")
    print("Please contact Magic Lantern developers on IRC (#magiclantern on freenode)")
    print("to take a closer look.")
else:
    print("Your camera appears OK, no need to worry.")


[etx]

Great work to all involved! Amazing!

I'm testing this out on my 5D Mk III on 1.2.3. I'm really missing the "centered 5x zoom" option from the older crop_rec module. I was previously running crop3x.2017Jan13.5D3123 version and using the centered 5x zoom was killer for avoiding the warping from the corners of the frame.

Am I missing something, or is this something that I just have to wait on future release?