Canon 6D

[a1ex]

Code: [Select]

mlv_dump M18-2132.MLV --dng --no-stripes --no-fixcp
exiftool -BlackLevelRepeatDim="2 2" -BlackLevel="1920 1792 1536 1024" -overwrite_original *.dng
dcraw -W *.dng
dcraw -a -W -b 4 *.dng

gives that fine maze pattern; however:

Code: [Select]

dng_validate.exe -tif out M18-2132_000000.dng

gives clean image to my eye, and so does darktable on the same DNG.

After swapping the black levels on the two greens, RawTherapee also gives clean image.

This one:

Code: [Select]

octave:1> a = read_raw('M18-2132_000000.dng');
octave:2> a(1:2:end,1:2:end) += 128;  # red
octave:3> a(1:2:end,2:2:end) += 256;  # green1
octave:4> a(2:2:end,1:2:end) += 512;  # green2
octave:5> a(2:2:end,2:2:end) += 1024; # blue
octave:6> imwrite(uint16(a), 'out.pgm');
octave:7> system('pgm2dng out.pgm');
octave:8> system('dcraw -a -W -b 4 out.DNG');

gives clean image in dcraw (no black level tweaking required, as pgm2dng hardcodes 2048), but also with different white balance. To run it, you need read_raw and pgm2dng.

So far, the above suggests a bug in dcraw when handling files with different per-channel black levels, but at least darktable and RawTherapee can give a clean image, and I'd expect Adobe programs to do the same (they wrote the DNG spec and dng_validate.exe, after all).

For the end user, given that per-channel black levels do not work as expected in all raw processing programs, I'd say the easiest workaround would be to do what the above octave script does, in mlv_dump (and other software that wants to handle lossless MLVs from 6D). Extremely easy coding task, so I'm going to leave it for novice programmers who may be looking for some low-hanging fruit to get started.

[8rnity]

found.
combination of both options --no-stripes --no-fixcp produces valid dng without pattern.
use just one or none of these two and have a pattern on the image.

what I wrote above is not true. it's mlv_dump version. both "mlv_dump on steroids" and the one used in "simple right clic menu mlv dump batch" produce the issue. used the one in download section and works perfect after exiftool!

[bouncyball]

In every proggie I tested there is a pattern in the output (--no-stripes --no-fixcp does not help for mlv_dump), The only correct way to get valid DNG with working good/old plain black level is the method a1ex showed above (octave example).

regards
bb

[Danne]

@8rnity
batch_mlv is using mlv_dump_on_steroids only it's called mlv_dump. There is some info about versions in the thread and probably also in the gpl license files.

[8rnity]

In every proggie I tested there is a pattern in the output (--no-stripes --no-fixcp does not help for mlv_dump), The only correct way to get valid DNG with working good/old plain black level is the method a1ex showed above (octave example).

regards
bb

yes, I edited my post. original mlv_dump version and exiftools give dng without pattern. white balance is not the same as not-lossless

[bouncyball]

Looks like a bug.

Here is the log from mlv_dump (6D MLV example from @8rnity):

Code: [Select]

Block: EXPO
  Offset: 0x0000015c
  Number: 4
    Size: 40
    Time: 1.751000 ms
     ISO Mode:   0
     ISO:        800
     ISO Analog: 96
     ISO DGain:  0/1024 EV
     Shutter:    0 microseconds (1/inf)

Shutter is not baked to MLV EXPO header. Hence:

$ ./dng_validate 6D_INVALID_000159.dng
Validating "6D_INVALID_000159.dng"...
*** Warning: The ExposureTime is <= 0 ***
*** Warning: Too little padding on left edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on top edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on right edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on bottom edge of CFA image (possible interpolation artifacts) ***
Raw image read time: 0.026 sec
Linearization time: 0.006 sec
Interpolate time: 0.081 sec
Validation complete

Ignore warnings about 'too little padding' they are same for all ML/MLtools produced DNGs.

[dfort]

There's been lots of changes on the crop_rec_4k and crop_rec_4k_mlv_lite_snd branches.

The 6D pull request hasn't been merged yet because there are some issues with lossless compression that still need to be worked out. If you want a preview of coming attractions I put up a test build on my downloads page. Who knows, maybe we got lucky?

[Levas]

Did some quick tests with your build Dfort.
And looks like sound recording works with mlv_lite.
I'v got working wav files with my 14bit lossless mlv files 

I assume crop mode didn't change for 6d meanwhile ?
Because I tried the crop mode, the crop mode only has the 3x3 720p option.
Tried it by setting the cam to 720p50fps, activated crop mode, but it still records squeezed frames, so it's still doing 3x5 binning.
The crop menu shows a menu within the 3x3 720p option, with:
Target YRES
Delta ADTG 0
etc...
All options are set to 0 by default
Tried setting the YRES to 960, but still got squeezed frames.

[dfort]

Screenshots? Sample MLVs? Is 10bit/12bit lossless working?

Only the 3x3 crop_rec option is coded but I don't have a 6D to check it out. If you do mlv_dump -v [your.MLV] you should be able to check out if it is 3x5 or 3x3. That crop_rec option might display a stretched LiveView but the DNG frames should look normal.

[Levas]

So took a while, but here's a 14 bit lossless MLV example with audio from the 6d:
https://drive.google.com/open?id=1idNTNn9lciEOd-CfUPrYaPXWOMOjlETp


Ofcourse, there are still the usual quirks the 6d has with lossless, so you need to fix the black levels of all 4 channels with exiftool as discussed earlier in this topic.
Also, the full wide 1824 pixels recording option still gives frames with the bottom half corrupted(as known and discussed before).
So this example is recorded in 1808 pixels wide resolution.
I'v also put the wav file and a dng frame with the right black levels in the same folder.

Other bitrates work, for audio.
But after fixing the black levels, there is still some minor pattern visible, probably due rounding errors by extracting lower bit depth files to standard 14 bit(as MLV_dump does) and then fixing black levels for 14 bitdepth values.
But this was for now always the case with lossless and lower than 14 bit lossless files on the 6d

Tried some more with the crop_rec, but I can't get unsqueezed footage, all settings I tried gave squeezed image, 3x5 pixel binning.

So the 6d still has some weird quirks with lossless
  -Needs black level fix, which doesn't seem to work perfect with lower than 14 bit (probably) due small rounding errors.
  -Widest resolution of 1824 pixels gives frames with bottom half corrupted.

But besides the quirks, it works for me, 14 bit lossless with audio  Thanks!

[dfort]

Tried some more with the crop_rec, but I can't get unsqueezed footage, all settings I tried gave squeezed image, 3x5 pixel binning.

Let's try changing the MEM_CMOS_WRITE and MEM_ADTG_WRITE addresses to the same as the 5D3. To be honest I'm not sure how to find those addresses. The same addresses worked on the EOSM/100D/650D/700D so I assumed it would work on the 6D. Not sure why the 5D3 is different or why the MEM_CMOS_WRITE and MEM_ADTG_WRITE share the same address on that camera.

I put up a crop_rec_4k test build on my downloads page. That build also has a working adtg_gui module so you should be able try this experiment:

Using adtg_gui I overrode the ADTG 800C register that controls the vertical line skipping to switch from 3x5 (0x4) to 3x3 (0x2) line skipping mode.
...
To get the camera to accept the overridden register one has to use the INFO button to switch into Canons menu and display modes and back into magic lantern. The preview image should be stretched when it worked.

[Levas]

Thanks Dfort,

Tried your build, right out of the box, crop record still gives squeezed frames in the MLV file.
Unfortunately the adtg_gui module won't load, OldApi it says, something with different version than expected, v6 vs v7.

[dfort]

Well it was worth a shot. Been learning about adtg_gui, looks like there is more than one version. You might try the one from the modules downloads page. With the camera in mv720 mode you should be able to adjust the ADTG 800C register so that it stretches the preview image vertically.

So, starting with the Canon menu set to 1280x720/60 (mv720) your LiveView should look normal:



Find the ADTG 800C register in adtg_gui. According to the tool tip (bottom line in green) it skips 4 lines in 720p mode:



Change that register so it skips 2 lines like in 1080p mode:



The LiveView should look like this:



That's what we're looking for. If you look at my pull request for the 6D, in crop_rec.c we need to find the correct values. Not sure if adtg_gui is showing them. You might need a build from the iso-research branch? IDK.

[Levas]

Ha ha nice coaster.
Had some good beers from that small brewery 

Will take a look at the adtg_gui module tomorrow.

[dfort]

So the 6d still has some weird quirks with lossless
  -Needs black level fix, which doesn't seem to work perfect with lower than 14 bit (probably) due small rounding errors.
  -Widest resolution of 1824 pixels gives frames with bottom half corrupted.

But besides the quirks, it works for me, 14 bit lossless with audio  Thanks!

So the first one needs to get fixed in mlv_dump and other MLV post applications:

For the end user, given that per-channel black levels do not work as expected in all raw processing programs, I'd say the easiest workaround would be to do what the above octave script does, in mlv_dump (and other software that wants to handle lossless MLVs from 6D). Extremely easy coding task, so I'm going to leave it for novice programmers who may be looking for some low-hanging fruit to get started.

The second one might need some adjustments in the skip offsets and/or the timer values. We did that recently for the EOSM/100D/650D/700D in this pull request. We could start by gathering some simple silent DNG frames with the camera set in the various video modes.

[Levas]

Downloaded the adtg_gui module from the module download page and played with it in combination with your(Dfort) 6d crop build.

When ADTG2[800c] is altered to value 2, (3x3 mode) the upper half of the live view is weird, looks like it scans more vertical lines of the sensor and overlays it on the top half of the image.

Now when I alter the value of CMOS[7] I can get normal framing.
CMOS[7] = 3c2 by default in 720p mode, When I change it to CMOS[7]=2a2 I get normal framing.
More options work for CMOS[7], which result in normal framing, but other (vertical) part of the sensor.
2a2 works, 308 and 3cb works and probably more..value 2a5 looks rather like the center.

I dived into the source code and it seems that what CMOS[7]on the 6d does what CMOS[2] CMOS[1] does on other cams ?

I couldn't get the resolution larger than 634 pixel vertical, not sure how to steer that.

Results for now are 1808 x 632 resolution in 3x3 mode and 50fps 

[dfort]

You got it working, fantastic!

Now all that is needed is to figure out how to interpret your findings to the correct MEM_CMOS_WRITE and MEM_ADTG_WRITE in this bit of code:

modules/crop_rec.c

Code: [Select]

    else if (is_camera("6D", "1.1.6"))
    {
        CMOS_WRITE = 0x2445C;
        MEM_CMOS_WRITE = 0x????????;
       
        ADTG_WRITE = 0x24108;
        MEM_ADTG_WRITE = 0x????????;
       
        is_basic = 1;
        crop_presets                = crop_presets_basic;
        crop_rec_menu[0].choices    = crop_choices_basic;
        crop_rec_menu[0].max        = COUNT(crop_choices_basic) - 1;
        crop_rec_menu[0].help       = crop_choices_help_basic;
        crop_rec_menu[0].help2      = crop_choices_help2_basic;
    }     


[a1ex]

These can be found by printing... MEM(CMOS_WRITE) and MEM(ADTG_WRITE). They are just some additional validation (e.g. if there's a typo in one of these addresses, or if somebody tries that code on a different firmware version). They are similar to the "hack error" message from mlv_lite - they simply point out that some stub might be incorrect.

[Levas]

Eh, still don't understand how to get those addresses 

Do we need some customised build that can print these addresses on screen ?
 

[dfort]

Yes. I just recycled what I did to get the SRM_BUFFER_SIZE printed on the screen:

QEMU to the rescue!



So maybe this will work?

Code: [Select]

    else if (is_camera("6D", "1.1.6"))
    {
        CMOS_WRITE = 0x2445C;
        MEM_CMOS_WRITE = 0xE92D47F0;
       
        ADTG_WRITE = 0x24108;
        MEM_ADTG_WRITE = 0xE92D41F0;
       
I'll update the pull request and put up anther test build to my downloads page.

[Levas]

Dfort, tested your newest build, but crop_rec still gives squeezed frames 

[dfort]

Ok--found what might be the problem. I took some of the values from here:

modules/adtg_gui.c

Code: [Select]

    else if (is_camera("6D", "1.1.6")) // from 1% (match 6D.113), JL, checked by Maqs
    {
        CMOS_WRITE_FUNC = 0x2445C; //"[REG] ############ Start CMOS OC_KICK"
        CMOS2_WRITE_FUNC = 0x2420C; //"[REG] ############ Start CMOS"
        ADTG_WRITE_FUNC = 0x24108; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS16_WRITE_FUNC = 0x24548; //"[REG] ############ Start CMOS16 OC_KICK"
    }

Other cameras just have CMOS_WRITE_FUNC but the 6D also has a CMOS2_WRITE_FUNC. I'm not sure if this will work because as you found out that you needed to adjust CMOS[7].

Let's give it another go--we might get lucky this time or at least we're getting closer to solving this.

Named this one crop_rec_4k_ CMOS_WRITE.2018Feb15.6D116 because that's what changed since the last test.

[dfort]

@Levas or any other 6D user - no pressure, just curious to find out if that last build finally worked with crop_rec. I think that's the last piece that is needed for the 6D to be merged into the official crop_rec_4k branch. The issues with lossless compression need to be solved with the MLV post processing apps.

[Levas]

Dfort, just checking the build. But I still get squeezed images.

[Levas]

When I activate crop mode, the screen goes black and applies the settings, but other than that, nothing changes.
Still 3x5 binning.
When I start adtg_gui, both 800C and CMOS[7] haven't changed, 800C is still on value 4 after enabling crop_rec