12-bit (and 10-bit) RAW video development discussion

[Danne]

Great man!
Confirmed fixed. Tried with 10-bit and with raw_rec on 5D mark III. raw_twk works and no green messed up files. Will put up new builds here again.
https://bitbucket.org/Dannephoto/cr2hdr/downloads

[dfort]

False alarm on the focus pixel elimination. a1ex pointed out that this has been tried in the past:

https://www.magiclantern.fm/forum/index.php?topic=6658.msg69400#msg69400

and things haven't changed with 10bit 12bit:



Bummer.

[Danne]

And cold pixel fix on this?

[dfort]

And cold pixel fix on this?

Doesn't fix it.

I'm running at test now to see if maybe there is some sort of a pattern to this. In any case this doesn't seem to be a 10bit/12bit issue so we're off topic.

[baccinoluigi]

@Danne
Tested on my 5dMkIII 10 and 12 bit
it works great
Thanks

[Surin Dmitriy]

Great man!
Confirmed fixed. Tried with 10-bit and with raw_rec on 5D mark III. raw_twk works and no green messed up files. Will put up new builds here again.
https://bitbucket.org/Dannephoto/cr2hdr/downloads

Can you add crop rec module to your build?

[a1ex]

@eNnvi: 1736x976 is divisible by 16 bytes at all bit depths. For 700D, the horizontal resolution is 1808. Therefore, EDMAC configuration must be, with the notation from here:

10bpp: (2170, skip 90) x 975, 2170
12bpp: (2604, skip 108) x 975, 2604
14bpp: (3038, skip 126) x 975, 3038

All these configurations work fine on 5D3, so I'm not sure what the issue is.

[eNnvi]

Ok still have to rightly understand the code (expecially Emacs, my arm knowledge is stuck on Cortex m0 and m4 from stm)

I'll open another thread to understand the whole firmware (i know it's hard) so i Can begin writing a wiki for developers and later for users.

So if i understood it right your calc is (W*H*bpp)%16==0 ?

[reddeercity]

Doing some more testing with 5D2 on 10-12bit raw with updated resolutions , A very strange thing is happen when I use 1872x#### in 10 or 14bit I get corruption but 12bit is almost good just some tearing on top 50 or so line no color noise the image is good . When I select 1856x#### 14bit is fine. It seems that 1872 only work with 12bit without crashing (early stop with all corrupted Cdng's, color bars and about 100 no more then that ) with Canon preview . Anything with 1880 in all 3 bit ranges are totally corrupted and crashes the recording session , ( early stop etc..) but not the camera . You would think in 14bit the higher resolutions above 1856 should not be corrupted , even if it's too much data bandwidth  it just should drop a frame and stop recording .

[teatotalTED]

The corruption seen in the 10bit and 12bit dforts 543 post looks like its to do with bit planes https://en.m.wikipedia.org/wiki/Bit_plane, guess this is obvious to those understand math and code, not me.

[Levas]

Trying to fix live view during recording for 6d and found some more pieces of the CONFIG_EDMAC_RAW_SLURP puzzle.

I found some references in edmac-memcpy.c and raw.c
Already added some extra code for the 6d to the emac-memcpy.c file.(see end of this post)
Now I found this piece of code in raw.c, I need to find DEFAULT_RAW_BUFFER MEM address for the 6d.
It says on top in the descriptions how to get that address(see bold text), but I'm too stupid too understand it
Can anyone help me here and tell me how to find this memory address?

#ifdef CONFIG_EDMAC_RAW_SLURP
/* undefine so we don't use it by mistake */
#undef RAW_LV_EDMAC

/* hardcode Canon's raw buffer directly */
/* you can find it from lv_raw_dump, arg1 passed to dump_file:
 *
 * raw_buffer = get_raw_buffer()
 * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
 * ...
 * dump_file(filename, raw_buffer, 7*something...)
 */

#ifdef CONFIG_60D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5028))
#endif

#ifdef CONFIG_600D
#define DEFAULT_RAW_BUFFER MEM(MEM(0x51FC))
#endif

#ifdef CONFIG_5D3
#define DEFAULT_RAW_BUFFER MEM(0x2600C + 0x2c)  /* 113 */
//~ #define DEFAULT_RAW_BUFFER MEM(0x25f1c + 0x34)  /* 123 */
#endif

#ifdef CONFIG_EOSM
#define DEFAULT_RAW_BUFFER MEM(0x404E4 + 0x44)
#endif

#else

/* with Canon lv_save_raw, just read it from EDMAC */
#define DEFAULT_RAW_BUFFER shamem_read(RAW_LV_EDMAC)

#endif


In edmac-memcpy.c I already added the 6d to this piece of code
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP

/* for other cameras, find a free channel with find_free_edmac_channels  */
#ifdef CONFIG_5D3
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_60D
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_600D
// write-index 1, 4, 6, 8, 10, 11, 13
uint32_t raw_write_chan = 4;
#endif

#ifdef CONFIG_EOSM
uint32_t raw_write_chan = 0x12;
#endif

#ifdef CONFIG_6D
uint32_t raw_write_chan = 8;
#endif

[dmilligan]

a1ex gave me some tips a couple pages back. Start by looking up the function lv_raw_dump in Canon firmware (pretty easy to find, it's an eventproc). It should be calling a function that returns a hardcoded pointer (or in 60D's case pointer to a pointer) to the raw buffer address.

[Levas]

I read the tips Alex gave you and I'm even more confused now, and that stuff was in raw_rec module, although the code should be much the same.
How do I look up the LV_raw_dump function in Canon firmware ?
I've made a rom dump, load it in a texteditor and found two times the text 'lv_raw_dump', but around it was a bunch of unreadable rom code.

Maybe you or Alex can give some more fool proof help ?

[dmilligan]

https://www.magiclantern.fm/forum/index.php?topic=12177.0

I like to use the HTML pages generated by ARM Console. Getting it setup is no easy task and it make take several days or more to run.

[Levas]

Getting it setup is no easy task and it make take several days or more to run.

Real motivating words

I was hoping for something more simple, but hey, if this is the way how to get live view back, I'll shall take a look at it.

[budafilms]

BUILD magiclantern-Nightly.2016Nov26.5D3113_raw_twk

When I press Zoom for focus, absolutly brick the camera.
Battery out resolve the problem.

10 Bits has the pink problem on the top  and 12 bits image on the higlights.
Not problem at all in 14 bits

Changing the aspect to 4x3 (1920 x 1280) has pink color on the higlights as the 12 bits.

[Levas]

I'm first trying to go this route to find the 6d address for lv_raw_dump
https://www.magiclantern.fm/forum/index.php?topic=12177.0

But I'm a little stuck, I do have 'disassemble.pl' file half working.
I do get the ROM1.BIN.strings file, but the other two, .dis and .labels are empty.
Terminal output:
------------------------------------------------------------------
string dump

create elf file
sh: arm-elf-objcopy: command not found
sh: arm-elf-objcopy: command not found
label scan
sh: arm-elf-objdump: command not found

disassemble and string lookup
sh: arm-elf-objdump: command not found

job complete!
--------------------------------------------------------------------
I think I don't have the path locations right.
The 'gcc-arm-none-eabi-4_8-2013q4' folder is in my user root directory.
And the original disassemble.pl file also asks for '/gcc-4.1-arm/bin/' directory, which I don't have (but that is probably old version of the 'gcc-arm-none-eabi-4_8-2013q4'folder' ?

This is now in my disassemble.pl file
----------------------------------------------------
# adjust these for your needs (note final slash):
#$path = "$ENV{'HOME'}/gcc-arm-none-eabi-4_8-2013q4/bin/";
$path = "";
----------------------------------------------------

Anybody now how to setup the path to the /gcc-arm-none-eabi-4_8-2013q4/bin/ directory which is in the root of my user directory on mac osx

[Oswald]

I tried the @nikfreak :s 7d build and it didn't work at all. 10 and also 12 bit neither of these worked (i tried mlv_rec). I tried crop mode and regular mode.  Same thing also showed on the camera screen when playing back.

[Danne]

Well it works with a little corruption here and there for the 7D(Used Nikfreak,s build). Raw_rec both 10bit and 12bit. SOme frames are perfect so there is potential. Tried with raw_twk added as well but symbols_engio error.

[nikfreak]

@Oswald: you better use MLVFS and check the DNG's instead of using MLRAWVIEWER. Otherwise you won't have luck.

...10 and also 12 bit neither of these worked (i tried mlv_rec).

Use raw_rec!!

[Levas]

@Oswald: Is that screenshot from MLrawviewer ?
MLrawviewer doesn't work with 10bit files, at least not the version I have with Canon 6d files.
I use MLV_dump to get dng out of the MLV files and can view them in RawTherapee.

[dmilligan]

Anybody now how to setup the path to the /gcc-arm-none-eabi-4_8-2013q4/bin/ directory which is in the root of my user directory on mac osx

It should be as easy as changing the line: $path=""; to simply point at the path of your arm gcc bin folder (any easy way to get that is to drag and drop that folder on a terminal window.

Code: [Select]

$path="~/gcc-arm-none-eabi-4_8-2013q4/bin/";


[Oswald]

I got it working with MLVFS, sorry about it. I tested again and 12bit and 10bit crop mode litlle bit working, i see few totally corrupted frames and other good ones. I see huge potential with filming with continues 10bit raw with resolution of 2520 x 1052.

http://imgur.com/a/icoiU

Second is 10bit and first one is 12bit.

[Levas]

@dmilligan
Thanks!

I was confused by this piece of code in the sample path.
"$ENV{'HOME'}

After the path fix I needed to duplicate 'arm-none-eabi-objcopy' and 'arm-none-eabi-objdump' and renamed them to 'arm-elf-objcopy' and 'arm-elf-objdump'
And now it works!

[Levas]

First of all, I have no idea what I'm doing 

Found this piece of code in the ROM1.BIN.dis file.

ff35a8d8:    ebf7afc0    bl   loc_ff1467e0
ff35a8dc:    e59f10a8    ldr   r1, [pc, #168]   ; ff35a98c: (ff359efc)
ff35a8e0:    e28f00a8    add   r0, pc, #168   ; ff35a990: (725f766c)  *"lv_raw_dump"

So am I right saying function lv_raw_buffer is called with address ff1467e0 ?
But now what to do, because this isn't the memory address I'm looking for ?
Skim the ROM1.BIN.dis file for ff1467e0 ?