Canon 6D

[Levas]

Another problem is that for most users, compiling ML yourself, is the biggest hurdle.

I tried it last year on OSX. For testing purposes for 10/12 bit.
Followed a tutorial on the forum and after copy and paste some stuff in terminal app, I got ML source code on my computer.
After that I could made small changes to the code and compile ML for testing purposes.
But the compiling step was also difficult, never really understood how to use it properly.
Most off the times I ended up with builds which didn't work for the 6d, or the modules wouldn't work with the ML version.
Lot's of errors are reported during the compiling, which seems normal I learned.
But if you're doing stuff you don't fully understand, it scares the shit out of you
At last I ended up compiling only the module I altered and tested it in a ML build from someone else, which worked with the module I compiled.
So the whole compiling thing didn't feel comfortable to me.

I guess it seems really straightforward and simple if your used to programming or really savvy with computers.
But for most of us, using the terminal app (or for windows user, using the command prompt) is already a big deal.
It feels like besides risking to brick our cameras with ML, were also risking to brick our computers

The last week I could do tests with the help of Dfort, he compiled the builds needed for testing and shared it with PM.
Works for me, works for Dfort and the end result for ML community is the same.

If things needs to be tested for the 6d, I'm happy willing to help. But I prefer a compiled build which is ready for use.


Also wanna help with the pull request Alex mentioned:
http://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff
If someone can provide me two builds, or maybe one build with the alterations only is enough, I'll be able to post two screenshots of the Free Memory dialog

[dfort]

Yay! We've got a virtual 6D.



Well almost:

Code: [Select]

[EOS] loading './6D/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './6D/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[MPU] FIXME: no MPU spells for 6D.

So put the cork back on the bottle. There's more work to be done.

@Levas - Compiling has gotten a whole lot easier since you tried it a year ago. I'd suggest giving it another try with the latest scripts. In fact I just rebuilt my QEMU environment on my Mac and was amazed at how much it has improved since I first attempted it. Thanks to a1ex and kichetof!

[dfort]

Posted new-dryos-task-hooks and dm-spy-experiments 6D.116 builds for testers. The dm-spy-experiments was built with CONFIG_DEBUG_INTERCEPT_STARTUP and I added some modules that might help move this along. Is the mpu_dump module something that should be run or is just the startup log needed?

Note to testers, search for instructions because this has probably been done before on other cameras.

As always, the builds are on my Bitbucket downloads page and may be changed or removed without notice.

[Levas]

@Alex
Got two screenshots from Free Memory Dialog.
https://drive.google.com/drive/folders/0B1BxGc3dfMDaTHUwR2kzNjNsbms?usp=sharing

One is from the nightly build for 6d, 4 okt 2017 and one is from a build according to this pull request:
https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff

[dfort]

How about posting the screenshots to make it easier to compare yours to the ones Audionut posted on the pull request.

Nightly


New DryOS task hooks

[e2k2]

Hello!
I have tried crop_rec module to record video(crop_rec_4k.2017Oct17.6D116).
h.264 1080p and 720p.
Aliasing problem didn't changed. Picture has bad aliasing caused by 5x line skipping.

Lossless compression tests with all bit depths have the same problem:


 

Maybe I'm doing something wrong.

Thanks

[a1ex]

Looks good. Can you also check the CPU usage and get some screenshots of "Debug - Show tasks" ?

Also, it would be great if you can play a bit with it and look for crashes and other weird behaviors not present in the regular nightly (aka regressions). This backend has the potential to alter any ML tasks (ideally it shouldn't change anything, but who knows).

Committed another test I'd like you to run: selftest.mo - null pointer test. It's what caused trouble on 100D after merging this backend.

Also note the lua_fix build (Experiments page) has a few additional selftest's - can you also run the stubs API from there, and also api_test.lua?

[Levas]

EDIT:PROBABLY DID USE WRONG BUILD HERE FIRST, FIRST TESTRESULTS ARE USELESS, WILL DO TESTS AGAIN WITH THE RIGHT BUILD.
UPDATED, screenshots shown here are done with the right build 'new-dryos-task-hooks.2017Oct20.6D116'

@alex
When camera is idle, cpu usage floats around 3,5 %

Show tasks:

[Levas]

Posted new-dryos-task-hooks and dm-spy-experiments 6D.116 builds for testers. The dm-spy-experiments was built with CONFIG_DEBUG_INTERCEPT_STARTUP and I added some modules that might help move this along. Is the mpu_dump module something that should be run or is just the startup log needed?

Note to testers, search for instructions because this has probably been done before on other cameras.

As always, the builds are on my Bitbucket downloads page and may be changed or removed without notice.

Am I right in thinking we need to have lossless full res silent pictures working on the 6d before the dm-spy-experiments are of any use ?

That way we can compare Normal CR2 file taken by canon to a lossless full res silent picture

[Levas]

Committed another test I'd like you to run: selftest.mo - null pointer test. It's what caused trouble on 100D after merging this backend.

@Alex, can't find the null pointer test in the Selftest menu in the 'New DryOS task hooks' build Dfort made
I find the commit:
https://bitbucket.org/hudson/magic-lantern/commits/84dd3fdfe3c69ce111b7b8729739595e36bf3aa0?at=unified
Can't compile a build myself yet. Anybody else is willing to compile ?

Also note the lua_fix build (Experiments page) has a few additional selftest's - can you also run the stubs API from there, and also api_test.lua?

Have run the stubs api test, didn't know what to do with it, made a screenshot, seems to me like the test fails, it's hanging, doing nothing ?


Have run the api_test.lua, here's the log file:
https://drive.google.com/drive/folders/0B1BxGc3dfMDaWmhoMDEzbmpGRDg?usp=sharing

[dfort]

I find the commit:
https://bitbucket.org/hudson/magic-lantern/commits/84dd3fdfe3c69ce111b7b8729739595e36bf3aa0?at=unified
Can't compile a build myself yet. Anybody else is willing to compile ?

Looks like you only need the selftest module to run the null pointer test. Compiled and on my Bitbucket downloads page.

[Levas]

@Dfort

The selftest module you compiled won't link with the new-dryos-task-hooks.2017Oct20.6D116 build you made 

[dfort]

So much for taking short-cuts.

Uploaded a new complete new-dryos-task-hooks build for the 6D that includes the selftest module. Deleted the old build to avoid confusion.

[Levas]

@Alex and Dfort

Null-pointer test works, no crash and a crash log:

https://drive.google.com/drive/folders/0B1BxGc3dfMDaYmQ4TnRQZ2M4TkU?usp=sharing

[Levas]

This defect is likely predictable and can be easily worked around in post. Let's check the actual black levels (if you didn't do the math already):

Code: [Select]

octave:7> for i = 1:2, for j = 1:2,
>           obch = b(i:2:end,j:2:50);
>           disp(median(obch(:)))
>         end, end
1919
1791
1536
1024

Dial these values in darktable (the 4 black level sliders) and the image is back to normal

Homework for you: hardcode the above black levels in the DNG with exiftool / exiv2 / whatever you prefer.

I've managed to get the lossless 14 bit files right in RawTherapee, look at the raw black point sliders at the right upper corner.
These work with values which are extracted from the baked in black level.
Now look at those numbers, -128, -512, -256 and -1024
Just what Alex explains in his post each offset differs by the power of two.



This trick doesn't work with 12 bit or 11 bit lossless files, there's still a little pattern to be seen, probably due to some dividing data and rounding numbers during recording.

[Levas]

Did some more testing with the mlv_lite and lossless 14bit.
All showed frames are corrected for black levels

In full width sensor mode, it always gives half corrupted frames (resolution 1824x1026):


But 1808 x 1016 resolution is normal:


And 1792 x 1008 has none corrupted frames but, looks like I can see a little pattern showing up:


Original dng's for download:
https://drive.google.com/drive/folders/0B1BxGc3dfMDabVdqUEdUUGMzckk?usp=sharing

[a1ex]

Can you also check 1824x1000, 1002, 1004 ... 1032? Being able to compile should help with this one.

Otherwise, just check (whatever you can select from menu) x (the above heights).

[dfort]

Yes, being able to compile helps. Give the Mac tutorial another shot.

In the meantime, do what a1ex says.

I'm thinking that maybe trying a different PREFERRED_RAW_TYPE might be interesting? Search the forum for clues on how to use it.

src/raw.c

Code: [Select]

#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
I prepared a build for testers that allows changing the PREFERRED_RAW_TYPE from the Debug menu. You know where to find it. 

[Levas]

Tested some different resolutions

1776 x 1000 = normal
1504 x 1002 = Corrupted as the 1824 x 1026 in the post above
1680 x 1008 = normal
1520 x 1012 = (almost) normal - looks a bit like the chroma isn't aligned
1360 x 1020 = (almost) normal - looks a bit like the chroma isn't aligned
1536 x 1024 = gives MLV's which can't be processed to dng's, normal file size  (tried multiple files, record a clip twice)
1552 x 1032 = normal

Corrected jpegs and original dng's are here:
https://drive.google.com/drive/folders/0B1BxGc3dfMDaTkpKQ0FpSFMyVmc?usp=sharing

[Levas]

Also tried 1824 x different resolutions:

1824 x 986 = corrupted frame
1824 x 1026 = corrupted frame
1824 x 1094 = corrupted frame

[Levas]

Trying different settings for PREFERRED_RAW_TYPE, but so far no luck 

Gone from 0 to 10 (forget the letters, hexadecimal  )
And tested 11 all the way to 29, so far no luck 
Now I'm out of batteries 

Most RAW_TYPES, give the half corrupted, are you sure the solution is in the preferred raw type ?
It almost looks like some dimension settings are wrong 

[Levas]

Could the problem be in raw.c in the height settings for 6d:

There's this part in raw.c :

*width  = ((bot_right & 0xFFFF) - (top_left & 0xFFFF)) * column_factor;
    *height = (bot_right >> 16)     - (top_left >> 16);

    /* height may be a little different; 5D3 needs to subtract 1,
     * EOS M needs to add 1, 100D usually gives exact value
     * is it really important to have exact height?
     * for some raw types, yes! */

The reason I'm thinking about this is because 10/12 bit had some problems with frames too and it was in the exact frame dimension settings I believe:

@eNnvi
I've tried the following dma flag as used for 700d
uint32_t dmaFlags = 0x20000000

And it causes the 'RAW DETECT ERROR' and when it will record it gives the scrambled frames.

So the only settings that were wrong in your build are the height value settings for the different video modes (width values were all good)
#ifdef CONFIG_6D
    *width  = zoom ? 2768 : mv720 ? 1920 : 1920;
    *height = zoom ? 988 : mv720 ?  662 : 1252;    /* find correct mv720 height -- must be exact! */   
    return 1;
#endif

And the dma flag setting, which must be either
uint32_t dmaFlags = 0x40000000 or uint32_t dmaFlags = 0x60000000

[a1ex]

That could be relevant for raw type 0x12 (used with 8..12-bit lossless, but not with 14-bit and other uncompressed modes). To see whether this is the case, try raw type 0x12 for regular 14-bit video. This raw type should be affected by digital ISO (e.g. ISO 320, 400 and 500 should look identical with default settings, but different with raw type 0x12). If height is not exact, only every other frame will be valid (easy to notice on the non-realtime preview).

Frame corruption: the encoder appears to require image height to be modulo 8. Try commenting out cases 0 and 4 (only the lines with return) in calc_res_y (mlv_lite.c) and see if you can still get invalid frames afterwards.

[dfort]

mlv_lite-edited.2017Oct23.6D116.zip posted. Have fun testing!

[ilguercio]

mlv_lite-edited.2017Oct23.6D116.zip posted. Have fun testing!

Where do we find it?