Author Topic: ProcessTwoInTwoOutLosslessPath  (Read 125704 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3718
Re: ProcessTwoInTwoOutLosslessPath
« Reply #375 on: March 14, 2018, 09:53:03 PM »
Let's try just this today:

Code: [Select]
    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
+       EngDrvOut(0xC0F26908, 0x12369168); /* 0x1235FFAA on 70D */
    }

On my downloads page. You can compare your results with the crop_rec_4k.57614b3 changest build that was extensively tested.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #376 on: March 15, 2018, 01:11:35 AM »
Do not try that!

edit: whew, it's a read channel. The address 0x12345678 was something that should only be tried in QEMU, never on real hardware (who knows what actually is at that address). If that was a write channel, you could have written into some memory region managed by other code, with completely unpredictable side effects (ranging from ¯\_(ツ)_/¯ to anything else).

edit: ok, it's overwritten later, so luckily the above change is a NOP. Will do the math later to explain what that does.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3718
Re: ProcessTwoInTwoOutLosslessPath
« Reply #377 on: March 15, 2018, 03:43:36 AM »
So nobody was brave enough to try it? There was one download.

Took it from the diff log:

       6D                                                  70D
153   [0xC0F26908] <- 0x12369168: RAM address   n   153   [0xC0F26908] <- 0x1235FFAA: RAM address
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2054
Re: ProcessTwoInTwoOutLosslessPath
« Reply #378 on: March 15, 2018, 06:02:18 AM »
A little more process , I can almost save a simple lossless dng -- much further then the frsp
I actually say saving 1 of 1 then freezes , battery pull needed still no dng save to card .



I added
Code: [Select]
if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1)); 
        EngDrvOut(0xC0F12020, PACK32(width    - 1,  height/10 - 1));  /* 0x18A0127 */
    }
not sure if this making any difference with simple , but no difference for sure with FRSP
I also added this to raw.c as per dfort suggestion
Code: [Select]
#if defined(LV_STATE)  /* older Digic 4 */
#define BLACK_LEVEL 1024
#endif
mlv_Lite still is not being made , need to look further in to it
"EngDrvOut" was a shoot in the dark , need to do more investigation

Edit: Notice in photo mode that the simple dng always is set to 470 ISO even if I set frsp to 100 ISO , strange

David_Hugh

  • Freshman
  • **
  • Posts: 75
Re: ProcessTwoInTwoOutLosslessPath
« Reply #379 on: March 15, 2018, 10:09:11 AM »
@dfort and a1ex

So is the build safe to test? Then I will do so as soon as I have time!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #380 on: March 15, 2018, 11:13:14 AM »
Notice in photo mode that the simple dng always is set to 470 ISO even if I set frsp to 100 ISO , strange

That sounds about right. Try setting the exposure time in LiveView to 1 second or longer, then compare with the exif info from the simple silent DNG.

Quote
Took it from the diff log

The diff log was made with this code, in particular:
Code: [Select]
       .buffer = 0x12345678,

That was there because:
1) it's not trivial to get a valid address (you'd have to capture an image, this is a bit of hit and miss, depending on camera model)
2) I wanted to see only how the JPEG encoder is configured (the emulation won't go further anyway)
3) 0x12345678 is easily recognizable, so you can see what registers depend on the image buffer
4) it was the simplest way that worked on any camera model

But that 0x12345678 was not meant to be used on real hardware! There you'd have to allocate RAM for both input and output, fill the buffer with valid image data, fill the raw_info structure with correct parameters and so on.

Quote
So is the build safe to test?

This particular build is safe, but it will not make any difference (that change is overwritten later, so it's not worth testing it). Generally, with this kind of changes you should be very careful, because in our case, any task (or DMA channel) can write to any part of the memory, and Canon code reflashing things back into the ROM at shutdown is what makes this operation possibly dangerous.

Overwriting memory that you don't know what it does is a risky thing to do.
- if that memory is unused (not read after you have overwritten it), there will be no side effects at all (you may even think it's all OK, when it isn't)
- if that memory (which may contain data or code) is read after you have overwritten it, the side effects can range from:
  - temporary (e.g. some parameter changes for a split-second and is set back to the old value at the next iteration
  - noticeable (something changes on the screen, prints error, crashes etc)
  - insidious (the side effect is visible long after the overwriting, possibly on a totally unrelated feature)
  - domino (the code using the modified data gets confused and starts overwriting other stuff)
  - extreme (it's not impossible to eventually reach EraseSectorOfROM)
  - extremely insidious (modified data gets written into ROM and is noticed days/weeks/months later - this actually happened!)

In this case, that register was the address of an EDMAC channel (which can read from or write to memory). Luckily, it was a read channel (something that reads from memory), so it's likely safe (not 100% safe - the image processor is a black box - but extremely unlikely to cause issues).

In our particular case:
Code: [Select]
[0xC0F26908] -> 0x1235FFAA: RAM address    ; somewhere in Canon code before our overrides
; our overrides (that big red block on 6D with CLOCK_ENABLE between the lines)
[0xC0F26908] <- 0x12345678: RAM address     ; after our overrides, value changed back by SetEDmac

Given that last line, any attempt to override this register has zero effects.

=> it's safe to try this particular build, but you will not notice any difference
=> it's generally not safe to try other builds with similar issues

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3718
Re: ProcessTwoInTwoOutLosslessPath
« Reply #381 on: March 15, 2018, 06:02:48 PM »
I thought I was on to something because:

try overriding the highlighted registers and see if it helps.

       6D                                                  70D
153   [0xC0F26908] <- 0x12369168: RAM address   n   153   [0xC0F26908] <- 0x1235FFAA: RAM address

and:

0x12369168 != 0x12345678

but now that I take a closer look at the diff log I see that 0xC0F26908 comes up multiple times and is quite often 0x12345678. So it might be one of these cases:

  - temporary (e.g. some parameter changes for a split-second and is set back to the old value at the next iteration

This is the code that runs on the camera:

modules/silent/silent.c
Code: [Select]
static unsigned int silent_init()
{
    /* fixme in core: prop handlers should trigger when initializing, but they do not */
    prop_iso = lens_info.raw_iso;
    prop_shutter = lens_info.raw_shutter;

    silent_pic_mlv_available = (mlv_generate_guid() != 0ULL);

    if (is_camera("500D", "*") || is_camera("550D", "*") || is_camera("600D", "*"))
    {
        /* see http://www.magiclantern.fm/forum/index.php?topic=12523.msg129874#msg129874 */
        long_exposure_fix_enabled = 1;
    }

    menu_add("Shoot", silent_menu, COUNT(silent_menu));
    return 0;
}

but in order to run in QEMU it was changed to this:

Code: [Select]
static unsigned int silent_init()
{
    /* keep the old stuff */

    /* code for running only in QEMU - no error checking done, no valid input, no valid results expected */
    struct memSuite * out_suite = shoot_malloc_suite_contig(16 * 1024 * 1024);
    struct raw_info out_raw_info = {
       .buffer = 0x12345678,
       .width = 1920,
       .height = 1080,
    };
    lossless_compress_raw(&out_raw_info, out_suite);

    return 0;
}

My mistake is clearly evident so I took down yesterday's test build but maybe there is something here? If all cameras were tested with ".buffer = 0x12345678" what accounts for these differences in 0xC0F26908?

5D3
0x12347F28 - 0x12345678 = 0x028B0

100D/650D/700D/EOSM/EOSM2
0x12349EA8 - 0x12345678 = 0x04830

6D
0x12369168 - 0x12345678 = 0x23AF0

70D
0x1235FFAA - 0x12345678 = 0x1A932

Perhaps the only thing this exercise proves is that there is a group of cameras with something in common.

Back to trial and error--hopefully with a little less error. Anything we haven't tried on the 70D?
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #382 on: March 15, 2018, 07:57:11 PM »
By default, a CR2 is smaller than a full-res DNG, so Canon code must be skipping some lines and columns. From that offset, we can figure out how many. Refer to this post for EDMAC configurations.

6D: 0x12369168 - 0x12345678 = 146160 bytes. Full buffer width: 9744 bytes = 5568 pixels. CR2 width (dcraw -i -v): 5568. 146160 / 9744 = 15 lines skipped.

In other words, to match a full-res silent DNG with a CR2 from 6D, one has to crop 15 lines at the top.

70D: 0x1235FFAA - 0x12345678 = 108850 bytes. Full buffer width: 9884 bytes = 5648 pixels. CR2 width: 5568 pixels. 108850 / 9884 = 11, 108850 % 9884 = 126 = 72px.

In other words, to match a full-res silent DNG with a CR2 from 70D, one has to crop 11 lines at the top, 72 columns at the left side and 8 columns at the right side.

Anyone has the patience to confirm this theory by pixel peeping? If you can get two images with absolutely no movement (e.g. with a Lua script or remote trigger), that's great; otherwise, just compare the active areas and ignore the image contents.

edit: found a CR2 from 70D on some camera review site and looked at active areas.
First active pixel in CR2: 72, 38.
Last active pixel in CR2: 5567, 3707 (with some doubts about the last line).
First active pixel in DNG: 144, 48 (delta = 72, 10 - where did one line go?)
Last active pixel in DNG: 5639, 3717 (delta = 72, 10, 8 black pixels at the right)

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1474
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #383 on: March 15, 2018, 08:57:25 PM »
Sounds like a fun project for tomorrow, will check it for the 6d.

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1474
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #384 on: March 16, 2018, 08:30:39 PM »
On 6d:

Full resolution silent picture = 5496 x 3670 pixels
CR2 = 5472 x 3648 pixels

For as far as I can tell, the difference is as follows:
The FRSP has 12 pixels extra on the left side and 12 pixels extra on the right side
The FRSP has 9 pixels extra at the top and 13 extra pixels on the bottom


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #385 on: March 16, 2018, 08:39:28 PM »
That's likely caused by difference in (declared) optical black areas (ActiveArea tag). My theory does not consider these at all (therefore, the check should be done with dcraw -E, or the equivalent setting from other software).

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1474
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #386 on: March 16, 2018, 08:54:55 PM »
Did the 'dcraw -E' command on both pictures and got two grayscale images including black borders:

FRSP = 5568 x 3722 pixels
CR2 = 5568 x 3708 pixels

Is that what you meant ?

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1474
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #387 on: March 16, 2018, 09:00:37 PM »
The FRSP has some data in the top 'black border' across the whole width of the image and about 4 pixels in size vertical, do you know where that is used for by Canon ?


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #388 on: March 16, 2018, 09:23:23 PM »
No idea where it's used. I might find out after emulating the photo capture process (including lossless encoding and CR2 creation).

Remember those huge logs full of registers? Managed to replay 7 of them :D

That was one tiny step towards emulating complex stuff like LiveView or photo capture.

lorenzo353

  • New to the forum
  • *
  • Posts: 15
Re: ProcessTwoInTwoOutLosslessPath
« Reply #389 on: March 29, 2018, 09:25:17 PM »
Hi,

you've got all info here, per camera
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_database.txt



By default, a CR2 is smaller than a full-res DNG, so Canon code must be skipping some lines and columns. From that offset, we can figure out how many. Refer to this post for EDMAC configurations.

6D: 0x12369168 - 0x12345678 = 146160 bytes. Full buffer width: 9744 bytes = 5568 pixels. CR2 width (dcraw -i -v): 5568. 146160 / 9744 = 15 lines skipped.

In other words, to match a full-res silent DNG with a CR2 from 6D, one has to crop 15 lines at the top.

70D: 0x1235FFAA - 0x12345678 = 108850 bytes. Full buffer width: 9884 bytes = 5648 pixels. CR2 width: 5568 pixels. 108850 / 9884 = 11, 108850 % 9884 = 126 = 72px.

In other words, to match a full-res silent DNG with a CR2 from 70D, one has to crop 11 lines at the top, 72 columns at the left side and 8 columns at the right side.

Anyone has the patience to confirm this theory by pixel peeping? If you can get two images with absolutely no movement (e.g. with a Lua script or remote trigger), that's great; otherwise, just compare the active areas and ignore the image contents.

edit: found a CR2 from 70D on some camera review site and looked at active areas.
First active pixel in CR2: 72, 38.
Last active pixel in CR2: 5567, 3707 (with some doubts about the last line).
First active pixel in DNG: 144, 48 (delta = 72, 10 - where did one line go?)
Last active pixel in DNG: 5639, 3717 (delta = 72, 10, 8 black pixels at the right)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3718
Re: ProcessTwoInTwoOutLosslessPath
« Reply #390 on: March 30, 2018, 05:44:23 PM »
you've got all info here, per camera
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_database.txt

I made a spreadsheet out of it, took out the PowerShot models, sorted by Model ID, added the name variations and highlighted the cameras supported by Magic Lantern.

https://docs.google.com/spreadsheets/d/1vMcOUh13TVOV3GY2576ke8rxiLn8qiMN9q57BEAXk-E/edit?usp=sharing

Interesting that some cameras are listed multiple times with different values. Why???
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

lorenzo353

  • New to the forum
  • *
  • Posts: 15
Re: ProcessTwoInTwoOutLosslessPath
« Reply #391 on: March 30, 2018, 07:39:48 PM »
hi,
there are lines for sraw flavors
http://lclevy.free.fr/cr2/#sraw

I made a spreadsheet out of it, took out the PowerShot models, sorted by Model ID, added the name variations and highlighted the cameras supported by Magic Lantern.

https://docs.google.com/spreadsheets/d/1vMcOUh13TVOV3GY2576ke8rxiLn8qiMN9q57BEAXk-E/edit?usp=sharing

Interesting that some cameras are listed multiple times with different values. Why???

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2054
Re: ProcessTwoInTwoOutLosslessPath
« Reply #392 on: April 03, 2018, 06:49:08 AM »
5d2 disassembly , found some path stuff:
there's a lot of this "OneOut"  then go to "TwoOut" etc....  just a jpeg where as the TwoOut  raw+jpeg+thumb image ?
Code: [Select]
*"PathTwoInOneOut"
*"DevelopPath\TwoInOneOutDevelopPath.c"


Code: [Select]
*"[DEV] JpegEncodePath_ChangeCBR(ID=%#lx)"
*"[DPATH] JPG_EN START J:%#lx Y:%#lx C:%#lx"
 *"[DPATH] JPG_EN PONY Enable(%d)"
*"[DPATH] JPG_EN LUCKY Enable(%d)(SEL:%d)"
Start , Pony , Lucky ?
Ok , some strange names for sure.

Code: [Select]
"***** DPU_GetJpegOrRawFullPath arraySize(%d)":
*"***** DPU_GetJpegOrRawFullPath FM_GetFilePath JPEG"
 *"***** DPU_GetJpegOrRawFullPath FM_GetFilePath RAW"
kind of fits in to the "TwoInOneOut" or is that "TwoOut"

Code: [Select]
*"===sCroppedImagePath(%s)"
*"***** DPU_CalcTrimImageHeightForBorderless input data = 0"
not sure if this has anything with lossless , look interesting thou.

Code: [Select]
*"[RD ERROR] ProcessRawDecodePath failed [%#x]"
*"[RD ERROR] ProcessJpegEncodePath failed [%#x]"
*" ClearDecodeJpegPath %d"
 *"TWIN StopDecWinPath Error:%d"

could this be the lossless decode path ?

Time to fire up QEMU and do some investigating

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2054
Re: ProcessTwoInTwoOutLosslessPath
« Reply #393 on: April 25, 2018, 12:16:12 AM »
Had a look at the cr2 & frsp .dng with dcraw -i -v had some confusing info
Code: [Select]
C:\dcraw>dcraw -i -v _MG_8665.CR2

Filename: _MG_8665.CR2
Timestamp: Sun Oct 29 19:01:35 2017
Camera: Canon EOS 5D Mark II
Owner:
ISO speed: 800
Shutter: 1/49.4 sec
Aperture: f/4.0
Focal length: 24.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  5616 x 3744
Full size:   5792 x 3804
Image size:  5634 x 3753
Output size: 5634 x 3753
Raw colors: 3
Filter pattern: RG/GB
Daylight multipliers: 2.391381 0.929156 1.289254
Camera multipliers: 2153.000000 1024.000000 1817.000000 1024.000000

C:\dcraw>dcraw -i -v 86630000.DNG

Filename: 86630000.DNG
Timestamp: Thu Jan 01 00:00:00 1970
Camera: Canon EOS 5D Mark II
DNG Version: 1.3.0.0
ISO speed: 0
Shutter: 1.0 sec
Aperture: f/1.0
Focal length: 0.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:   128 x 84
Full size:   5792 x 3804
Image size:  5632 x 3752
Output size: 5632 x 3752
Raw colors: 3
Filter pattern: RG/GB
Daylight multipliers: 2.391381 0.929156 1.289254
Camera multipliers: 2.111331 1.000000 1.602564 0.000000
One of the biggest difference is the thumb image , cr2 full res (5616 x 3744) & ML writes a small thumb (128 x 84)
but that's not what confuses me , it's the "Camera multipliers" there very different
Code: [Select]
CR2
Camera multipliers: 2153.000000 1024.000000 1817.000000 1024.000000
Code: [Select]
FRSP DNG
Camera multipliers: 2.111331 1.000000 1.602564 0.000000
thou the "Daylight multipliers" are the same
Code: [Select]
2.391381 0.929156 1.289254Is this the same on all FRSP ? are with d4 cam's only  ?
Could lossless be looking for the  CR2  "Camera multipliers" and crashes on 5d2/D4 ? or is this a result of FRSP for the gradation from top to bottom ?

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2054
Re: ProcessTwoInTwoOutLosslessPath
« Reply #394 on: August 04, 2018, 08:53:28 AM »
I may have found  the "resources" for lossless , read this post for the LOGs
Code: [Select]
76146> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942d70) x3 from ffa0b7b0:
76188> LiveViewMg:00096678:00:00:      1)    6000f (?)
761B9> LiveViewMg:00096678:00:00:      2)    20003 (write connection 0x3)
761E7> LiveViewMg:00096678:00:00:      3)        1 (write channel 0x1)
76234> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76272> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
762B1> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76325> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942db8) x7 from ffa0b7d4:
76366> LiveViewMg:00096678:00:00:      1)    60005 (?)
7638D> LiveViewMg:00096678:00:00:      2)    6000c (?)
763AF> LiveViewMg:00096678:00:00:      3)   1d0000 (?)
763D5> LiveViewMg:00096678:00:00:      4)   220016 (?)
763FB> LiveViewMg:00096678:00:00:      5)   220017 (?)
7641E> LiveViewMg:00096678:00:00:      6)   220027 (?)
76444> LiveViewMg:00096678:00:00:      7)   220028 (?)
76488> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
764C6> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
764F9> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76579> LiveViewMg:ffa0b3e0:99:02: ClearWbPass
765BC> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(92b440) x7 from ff8e79e0:
765FA> LiveViewMg:00096678:00:00:      1)   130009 (?)
76622> LiveViewMg:00096678:00:00:      2)   13000a (?)
76646> LiveViewMg:00096678:00:00:      3)   13000b (?)
76670> LiveViewMg:00096678:00:00:      4)    20002 (write connection 0x2)
7669A> LiveViewMg:00096678:00:00:      5)        2 (write channel 0x2)
766BA> LiveViewMg:00096678:00:00:      6)    70000 (?)
766DC> LiveViewMg:00096678:00:00:      7)    f0000 (?)
7671C> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76760> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76792> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
767BF> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76823> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(92b498) x1 from ff8e79e0:
76864> LiveViewMg:00096678:00:00:      1)    b0000 (?)
7689C> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76901> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(92b4d8) x2 from ff8e79e0:
76945> LiveViewMg:00096678:00:00:      1)    c0000 (?)
76969> LiveViewMg:00096678:00:00:      2)   120000 (?)
769A1> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
769DD> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76A3C> **INT-50h*:ff871ad4:00:01: [PM] DisablePowerSave (Counter = 4)
76A76> **INT-50h*:ff871b44:00:01: [PM] EnablePowerSave (Counter = 3)
76AA3> **INT-50h*:FF99F344:MMIO:  [0xC022009C] -> 0x00000047
76ACB> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76AFC> LiveViewMg:ff9a4810:00:02: [ENG] UnregisterEDmacAbortCBR(8)
76B38> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76B4F> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76B87> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942bf0) x54 from ff8e79e0:
76BCC> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76BE1> LiveViewMg:00096678:00:00:      1)   100000 (?)
76C2B> **INT-36h*:000969b4:00:00: *** mpu_recv(0a 08 09 05 00 80 00 00 00), from ff99f6cc
76C6B> **INT-36h*:00096840:00:00: *** TryPostEvent(PropMgr, 0xa, 0x51a59c, 0x14), from ff85ec40
76CD2> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76CF0> LiveViewMg:00096678:00:00:      2)    50013 (?)
76D25> LiveViewMg:00096678:00:00:      3)    50001 (?)
76D4F> LiveViewMg:00096678:00:00:      4)    5000f (?)
76D74> LiveViewMg:00096678:00:00:      5)    5001a (?)
76D96> LiveViewMg:00096678:00:00:      6)    5000e (?)
76DB8> LiveViewMg:00096678:00:00:      7)    50019 (?)
76DDB> LiveViewMg:00096678:00:00:      8)    50007 (?)
76DFE> LiveViewMg:00096678:00:00:      9)    50009 (?)
76E20> LiveViewMg:00096678:00:00:     10)    50006 (?)
76E42> LiveViewMg:00096678:00:00:     11)    5000d (?)
76E64> LiveViewMg:00096678:00:00:     12)    50005 (?)
76E87> LiveViewMg:00096678:00:00:     13)    80000 (?)
76EAD> LiveViewMg:00096678:00:00:     14)    d0000 (?)
76ECF> LiveViewMg:00096678:00:00:     15)    e0000 (?)
76EF4> LiveViewMg:00096678:00:00:     16)   150000 (?)
76F17> LiveViewMg:00096678:00:00:     17)    60003 (?)
76F3A> LiveViewMg:00096678:00:00:     18)    6000b (?)
76F5F> LiveViewMg:00096678:00:00:     19)    60007 (?)
76F81> LiveViewMg:00096678:00:00:     20)    60009 (?)
76FA2> LiveViewMg:00096678:00:00:     21)    60010 (?)
76FC5> LiveViewMg:00096678:00:00:     22)    60014 (?)
76FEA> LiveViewMg:00096678:00:00:     23)    60013 (?)
7700D> LiveViewMg:00096678:00:00:     24)   130000 (?)
77034> LiveViewMg:00096678:00:00:     25)   130001 (?)
77059> LiveViewMg:00096678:00:00:     26)   130006 (?)
7707E> LiveViewMg:00096678:00:00:     27)   130007 (?)
770A2> LiveViewMg:00096678:00:00:     28)   130008 (?)
770C6> LiveViewMg:00096678:00:00:     29)   13000c (?)
770EC> LiveViewMg:00096678:00:00:     30)   13000d (?)
77112> LiveViewMg:00096678:00:00:     31)   13000e (?)
77136> LiveViewMg:00096678:00:00:     32)   13000f (?)
7715D> LiveViewMg:00096678:00:00:     33)   130010 (?)
77183> LiveViewMg:00096678:00:00:     34)   130016 (?)
771A6> LiveViewMg:00096678:00:00:     35)   130017 (?)
771CA> LiveViewMg:00096678:00:00:     36)   130019 (?)
771F0> LiveViewMg:00096678:00:00:     37)   13001a (?)
77216> LiveViewMg:00096678:00:00:     38)   13001b (?)
7723A> LiveViewMg:00096678:00:00:     39)   220000 (?)
7725E> LiveViewMg:00096678:00:00:     40)   220001 (?)
77281> LiveViewMg:00096678:00:00:     41)   220012 (?)
772A5> LiveViewMg:00096678:00:00:     42)   220013 (?)
772CA> LiveViewMg:00096678:00:00:     43)   220014 (?)
772F2> LiveViewMg:00096678:00:00:     44)   220015 (?)
7731D> LiveViewMg:00096678:00:00:     45)    20004 (write connection 0x4)
77349> LiveViewMg:00096678:00:00:     46)    20000 (write connection 0x0)
77372> LiveViewMg:00096678:00:00:     47)    30008 (read connection 0x8)
7739E> LiveViewMg:00096678:00:00:     48)    3000f (read connection 0xf)
773C9> LiveViewMg:00096678:00:00:     49)        9 (write channel 0x12)
773F1> LiveViewMg:00096678:00:00:     50)    10001 (read channel 0x9)
7741E> LiveViewMg:00096678:00:00:     51)    10000 (read channel 0x8)
77447> LiveViewMg:00096678:00:00:     52)    50011 (?)
7746A> LiveViewMg:00096678:00:00:     53)    5001b (?)
7748C> LiveViewMg:00096678:00:00:     54)    5001e (?)
774DA> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
  ............
77B25> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
77B55> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
77BDB> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942d04) x12 from ff8e79e0:
77C22> LiveViewMg:00096678:00:00:      1)    6000e (?)
77C4A> LiveViewMg:00096678:00:00:      2)    6000d (?)
77C6D> LiveViewMg:00096678:00:00:      3)    60015 (?)
77C90> LiveViewMg:00096678:00:00:      4)   180000 (?)
77CB3> LiveViewMg:00096678:00:00:      5)   130002 (?)
77CD6> LiveViewMg:00096678:00:00:      6)   130005 (?)
77CF9> LiveViewMg:00096678:00:00:      7)   130012 (?)
77D1B> LiveViewMg:00096678:00:00:      8)   130014 (?)
77D3E> LiveViewMg:00096678:00:00:      9)   130018 (?)
77D62> LiveViewMg:00096678:00:00:     10)   13001c (?)
77D90> LiveViewMg:00096678:00:00:     11)    20015 (write connection 0x15)
77DBE> LiveViewMg:00096678:00:00:     12)        0 (write channel 0x0)
77DF9> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
...........
77F24> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��

Also this
Code: [Select]
822AC> ShootCaptu:ff87ffec:93:03: GetCaptureResources start.
822D4> ShootCaptu:000965bc:00:00: *** LockEngineResources(680404) x38 from ffa384f8:
8230D> ShootCaptu:00096678:00:00:      1)    50001 (?)
82332> ShootCaptu:00096678:00:00:      2)    50005 (?)
82353> ShootCaptu:00096678:00:00:      3)    50007 (?)
82374> ShootCaptu:00096678:00:00:      4)    50009 (?)
82397> ShootCaptu:00096678:00:00:      5)    5000e (?)
823B7> ShootCaptu:00096678:00:00:      6)    50011 (?)
823D7> ShootCaptu:00096678:00:00:      7)    50019 (?)
823F7> ShootCaptu:00096678:00:00:      8)    5001b (?)
82419> ShootCaptu:00096678:00:00:      9)    5001e (?)
8243B> ShootCaptu:00096678:00:00:     10)    50013 (?)
8245C> ShootCaptu:00096678:00:00:     11)    50020 (?)
8247D> ShootCaptu:00096678:00:00:     12)    50021 (?)
8249F> ShootCaptu:00096678:00:00:     13)    70000 (?)
824C1> ShootCaptu:00096678:00:00:     14)    f0000 (?)
824E2> ShootCaptu:00096678:00:00:     15)    b0000 (?)
82505> ShootCaptu:00096678:00:00:     16)   200002 (?)
82528> ShootCaptu:00096678:00:00:     17)   140000 (?)
8254C> ShootCaptu:00096678:00:00:     18)   120000 (?)
82570> ShootCaptu:00096678:00:00:     19)   130009 (?)
82593> ShootCaptu:00096678:00:00:     20)   13000a (?)
825B9> ShootCaptu:00096678:00:00:     21)   13000b (?)
825DB> ShootCaptu:00096678:00:00:     22)   13000d (?)
825FE> ShootCaptu:00096678:00:00:     23)   13000e (?)
82621> ShootCaptu:00096678:00:00:     24)   13000f (?)
82645> ShootCaptu:00096678:00:00:     25)   130016 (?)
82669> ShootCaptu:00096678:00:00:     26)   22001b (?)
82695> ShootCaptu:00096678:00:00:     27)        0 (write channel 0x0)
826BC> ShootCaptu:00096678:00:00:     28)        1 (write channel 0x1)
826E0> ShootCaptu:00096678:00:00:     29)        2 (write channel 0x2)
82704> ShootCaptu:00096678:00:00:     30)        6 (write channel 0x6)
82728> ShootCaptu:00096678:00:00:     31)    10000 (read channel 0x8)
82750> ShootCaptu:00096678:00:00:     32)    10001 (read channel 0x9)
82779> ShootCaptu:00096678:00:00:     33)    20000 (write connection 0x0)
827A4> ShootCaptu:00096678:00:00:     34)    20002 (write connection 0x2)
827CD> ShootCaptu:00096678:00:00:     35)    20012 (write connection 0x12)
827F9> ShootCaptu:00096678:00:00:     36)    2001c (write connection 0x1c)
82822> ShootCaptu:00096678:00:00:     37)    30008 (read connection 0x8)
8284B> ShootCaptu:00096678:00:00:     38)    3000f (read connection 0xf)
828C1> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
82908> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
................
82DDA> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
82E08> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
82E36> ShootCaptu:ff880004:93:03: GetCaptureResources end.

Cr2 lossless ?
Code: [Select]
EEAF8>     RscMgr:ff8b65c4:80:01: Allocate MEM3 0 0
EEB1F>     RscMgr:ff8b6670:80:01: OK AllocateMEM3 0xBAA4C0(876544)(1)
EEBD1> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5742)
EEC5E> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5742)��(SemOK)
EEC9A> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5742)
EECE9> FrontShtDe:000965bc:00:00: *** LockEngineResources(72bed4) x17 from ffa59b1c:
EED33> FrontShtDe:00096678:00:00:      1)    10002 (read channel 0xa)
EED62> FrontShtDe:00096678:00:00:      2)        3 (write channel 0x3)
EED88> FrontShtDe:00096678:00:00:      3)        4 (write channel 0x4)
EEDAE> FrontShtDe:00096678:00:00:      4)    30000 (read connection 0x0)
EEDD8> FrontShtDe:00096678:00:00:      5)    20005 (write connection 0x5)
EEE03> FrontShtDe:00096678:00:00:      6)    20016 (write connection 0x16)
EEE26> FrontShtDe:00096678:00:00:      7)    50003 (?)
EEE48> FrontShtDe:00096678:00:00:      8)    5000d (?)
EEE69> FrontShtDe:00096678:00:00:      9)    5000f (?)
EEE8B> FrontShtDe:00096678:00:00:     10)    5001a (?)
EEEAC> FrontShtDe:00096678:00:00:     11)    80000 (?)
EEECE> FrontShtDe:00096678:00:00:     12)    90000 (?)
EEEEF> FrontShtDe:00096678:00:00:     13)    a0000 (?)
EEF10> FrontShtDe:00096678:00:00:     14)   160000 (?)
EEF33> FrontShtDe:00096678:00:00:     15)   130003 (?)
EEF58> FrontShtDe:00096678:00:00:     16)   130004 (?)
EEF7C> FrontShtDe:00096678:00:00:     17)   130005 (?)
EF007> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF048> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF07C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF0AE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF0DD> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF10C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF140> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF173> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF1A1> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF1CF> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF1FD> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF227> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF250> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF27E> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF2C1> FrontShtDe:ffa59694:16:03: [TTJ][150,5742,0] RAW(5792,3804,0,14)
EF366> FrontShtDe:00096224:00:00: *** ConnectReadEDmac(0xa, 0x0), from ffa59770
EF3C5> FrontShtDe:00096224:00:00: *** RegisterEDmacCompleteCBR(0xa, 0xffa593f8 "[TTJ][%d,%d,%d] Read1CompleteCBR", 0x0), from ffa59780
EF415> FrontShtDe:00096224:00:00: *** ConnectWriteEDmac(0x3, 0x5), from ffa597f8
EF4F6> FrontShtDe:00096224:00:00: *** RegisterEDmacCompleteCBR(0x3, 0xffa9285c "[PackMem] CompleteInterrupt In %d", 0x3), from ffa92be0
EF522> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
EF572> FrontShtDe:00096224:00:00: *** ConnectWriteEDmac(0x4, 0x16), from ffa59830
EF5D1> FrontShtDe:00096224:00:00: *** RegisterEDmacCompleteCBR(0x4, 0xffa59340 "[TTJ][%d,%d,%d] Write2CompleteCBR", 0x0), from ffa59840
EF600> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1acac0f0
EF639> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864

Code: [Select]
DBAF3> ShootPreDe:ffa5a5ac:16:03: [FB] START WR7:0x1f50000 RD4:0x124e9de4
DC2DA> ShootPreDe:ffa5a158:16:03: [FB] STOP WR7:0x1f5ccc0 RD4:0x1f5c8a4
DFBC5> ShootPreDe:ffa5a270:16:03: [FC] STOP RD4:0x1f54b00
DFC9E> ShootPreDe:ff9a4810:00:02: [ENG] UnregisterEDmacAbortCBR(12)
DFCE8> ShootPreDe:000965bc:00:00: *** UnLockEngineResources(72a764) x5 from ffa5a4bc:
DFD27> ShootPreDe:00096678:00:00:      1)   140001 (?)
DFD55> ShootPreDe:00096678:00:00:      2)        7 (write channel 0x10)
DFD7B> ShootPreDe:00096678:00:00:      3)    10004 (read channel 0xc)
DFDA6> ShootPreDe:00096678:00:00:      4)    20017 (write connection 0x17)
DFDD2> ShootPreDe:00096678:00:00:      5)    3000e (read connection 0xe)
DFE20> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFE6C> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFEA3> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFED4> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFF0F> ShootPreDe:ff8852f8:95:05: Face Num : 0, Face All Num : 0
DFFA7> ShootPreDe:00096098:00:00: *** SPSState: (3) --4--> (2)          ff885408 (x=68187c z=0 t=0)
DFFFB> ShootPreDe:ff885448:95:05: spsStartProcess(5742)
E8E49> ShootPreDe:ff884154:95:05: WB RectH:(2960, 2992)
E8E97> ShootPreDe:ff884174:95:05: WB RectV:(1911, 1943)
E8EC3> ShootPreDe:ff884190:95:05: WB Limit:(H:11464) (L:1072) )
E90B5> ShootPreDe:ff884564:95:05: WB Num=2
E90E1> ShootPreDe:ff88458c:95:05: WB R =3831(1915)
E9109> ShootPreDe:ff8845b8:95:05: WB G1=2060(1030)
E912E> ShootPreDe:ff8845e4:95:05: WB G2=1919(959)
E9154> ShootPreDe:ff884610:95:05: WB B =4039(2019)
E919D> ShootPreDe:ff885aa4:95:05: ISO_DIGITAL_GAIN_1

I havn't tired to implement any of the above info in to the lossless code yet , need to exam the log's closer , anyone that's want to help
get 5D2 lossless working feel free to jump in any time .



reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2054
Re: ProcessTwoInTwoOutLosslessPath
« Reply #395 on: August 05, 2018, 04:24:49 AM »
For 5d2 , looks like I found the same resources as in this code from dfort , which if I'm not mistaken came from the 7D #lossless.c-274
Code: [Select]
EED33> FrontShtDe:00096678:00:00:      1)    10002 (read channel 0xa)
EED62> FrontShtDe:00096678:00:00:      2)        3 (write channel 0x3)
EED88> FrontShtDe:00096678:00:00:      3)        4 (write channel 0x4)
EEDAE> FrontShtDe:00096678:00:00:      4)    30000 (read connection 0x0)
EEDD8> FrontShtDe:00096678:00:00:      5)    20005 (write connection 0x5)
EEE03> FrontShtDe:00096678:00:00:      6)    20016 (write connection 0x16)
EEE26> FrontShtDe:00096678:00:00:      7)    50003 (?)
EEE48> FrontShtDe:00096678:00:00:      8)    5000d (?)
EEE69> FrontShtDe:00096678:00:00:      9)    5000f (?)
EEE8B> FrontShtDe:00096678:00:00:     10)    5001a (?)
EEEAC> FrontShtDe:00096678:00:00:     11)    80000 (?)
EEECE> FrontShtDe:00096678:00:00:     12)    90000 (?)
EEEEF> FrontShtDe:00096678:00:00:     13)    a0000 (?)
EEF10> FrontShtDe:00096678:00:00:     14)   160000 (?)
EEF33> FrontShtDe:00096678:00:00:     15)   130003 (?)
EEF58> FrontShtDe:00096678:00:00:     16)   130004 (?)
EEF7C> FrontShtDe:00096678:00:00:     17)   130005 (?)

Below I have a strong believe that this could be the missing info , I was using the 7D value that did not work even on the 7D in lossless . lossless.c-253
from the 5d3 code I know I need , 1 "read connection" & 2 "write connections"
Code: [Select]
1CB6C> FrontShtDe:00096678:00:00:      5)    30000 (read connection 0x0)
1CB95> FrontShtDe:00096678:00:00:      6)    30021 (read connection 0x21) **** I think this one could be removed totally****
1CBBD> FrontShtDe:00096678:00:00:      7)    20005 (write connection 0x5)
1CBE5> FrontShtDe:00096678:00:00:      8)    20003 (write connection 0x3)
so I'm thinking read 0x0 , write on 0x5 & 0x3 , haven't try it yet , I want to do same some investigation with  io_trace_full branch first to get more info on this .

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2054
Re: ProcessTwoInTwoOutLosslessPath
« Reply #396 on: August 13, 2018, 06:58:03 AM »
Volunteers needed  :) 
I'm looking for some users with digic4 cams to take (capture) a cr2 photo outside of Liveview with a special "io-trace-full" ml build for log info in to the "TwoInTwoOutLosslessPath"
and also outside of Liveview review a cr2 photo but not on the same Log as the captured cr2 . To run test , load ml io-trace-full build (link for cams are below)
you only need the "5D2_212.sym" file (your cam will be different e.g. 7d_703)  exit liveview go to the debug menu activate the Logging feature
-- exit menu so there no liveview take a cr2 photo return to debug menu and hi-light logging again to save the log . Do the same for review cr2 photo , this will help understand the Jpeg raw compression process as it's not the same as digic5 so it's a bit of a mystery and hopefully we can finally get d4 's to be supported in lossless .
io-trace-full_magiclantern-Nightly.2018Aug11.7D203.zip
io-trace-full_magiclantern-Nightly.2018Aug11.50D109.zip 
io-trace-full_magiclantern-Nightly.2018Aug11.550D109.zip
io-trace-full_magiclantern-Nightly.2018Aug11.500D111.zip
As you may have guessed I've already completed the Log's for 5D2
but for anyone that what to reproduce my results here the build for the 5D2 io_trace_full_magiclantern-Nightly.2018Aug04.5D2212.zip
if there any problems with the builds please let me know
Thanks to all who wish to push digic4 into Lossless  :D

aprofiti

  • Contributor
  • Member
  • *****
  • Posts: 177
Re: ProcessTwoInTwoOutLosslessPath
« Reply #397 on: August 13, 2018, 02:41:30 PM »
Here are logs from 50D: Cr2 capture and Cr2 Review

They were taken without entering LiveView.
Hope they are good, let me know if you need others.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #398 on: August 13, 2018, 06:27:02 PM »
They are very good, thanks a lot.

Nitpicks:
- figure out why there's a bunch of zeros at the beginning of the log (likely some bug)
- load edmac.mo during the experiment (the log contains some lines telling you to do so). It's a nitpick because the information printed is a bit redundant (I can figure it out from MMIO lines), but it's still nice to have.

aprofiti

  • Contributor
  • Member
  • *****
  • Posts: 177
Re: ProcessTwoInTwoOutLosslessPath
« Reply #399 on: August 18, 2018, 02:52:26 AM »
- load edmac.mo during the experiment (the log contains some lines telling you to do so). It's a nitpick because the information printed is a bit redundant (I can figure it out from MMIO lines), but it's still nice to have.

Here is the same procedure but with edmac.mo copied from nightly (as the io-trace-full build doesn't have it):
Cr2TakeEdmac and Cr2ReviewEdmac

Also no bunch of zeroes int these logs... Don't know why it happened previously

Had a look to find read and write channels and the appears a bit different from reddeercity's logs