Author Topic: ProcessTwoInTwoOutLosslessPath  (Read 129877 times)

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #225 on: February 18, 2018, 07:32:49 AM »
BTW -- Never heard of MS Visual Studio Code so I downloaded it and took a look at it. I might be missing something but I don't see how it can be useful for working with disassembled ARM code. There are some powerful apps out there but I'm just using the disassembly.pl script as recommended in Tutorial: finding stubs by a1ex and a text editor.
No , I use it to search the disassemble .dis and strings.htm file for address & as a code editor , it's very fast in searching .
   


I was going to ask you the same question, how are you looking these up? Are you checking the addresses you found against disassemblies for the cameras that already have lossless compression working?

No , I just looked at the code and started to search by "key word statements" e.g. mem1toraw as you can see in the images & it searches all files loaded (5d2.212.dis , lossless.c , strings.htm)
Then I would read the disassemble , and see what was going , which gives me a very,  very basic understanding , the more I read the more I discovery  :D

 


reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #226 on: February 18, 2018, 10:19:27 PM »
Thanks for the help @dfort , sent you a pm

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 697
Re: ProcessTwoInTwoOutLosslessPath
« Reply #227 on: February 18, 2018, 11:18:43 PM »
Can confirm VSCode is only good thing Microsoft ever made.

I'm a bit confused as to how dfort found the addresses it seems more complex than just finding strings in the disassembly :( Can't imagine I'll ever figure this out.
5D2, MLV App

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #228 on: February 19, 2018, 07:14:44 AM »
I must have a bad 5d2 disassembly or bad rom1, I can't find
Code: [Select]
loc_ff1beb18: ; 5 refsI have not problem finding
Code: [Select]
loc_ff32330c: ; 10 refsand all the other stuff make sense now , I can't even find any phase/statement , etc. ... about "loc_" or "5refs"
in my 5d2 disassembly , but I did find
Code: [Select]
ff1beb18: e92d41f0 push {r4, r5, r6, r7, r8, lr}which is very close to 5d3-1.1.3
Code: [Select]
ff1beb18: e92d4070 push {r4, r5, r6, lr}
Quote
This won't make any sense unless you can flip between the 5D2 and the 5D3 disassemblies.
It is almost a perfect match. Keep scrolling through that function and just before you get to the next function (2 refs on both 5D2 and 5D3) you should have an "ah ha" moment.
I wish could say that , but sadly no there must be a problem with my disassembly , or is there a state that the camera needs to be in .e.g. "Photo Mode" or "Video Mode"
for compressed raw ? to save to the ROM ,  does that make a different ? I think I was always in video mode so maybe those routines/are not present .
 

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #229 on: February 19, 2018, 09:03:32 AM »
You might have used a different disassembler. I used the perl script from the Tutorial: finding stubs. Hint, just search for the address, not the entire line. Yours might be formatted differently.

I think I was always in video mode so maybe those routines/are not present .

I doubt that. The firmware should have everything you need no matter what camera settings were active when the dump was saved. It does make a difference in QEMU. Just checked out the 5D2 in QEMU and it looks like the emulation has gotten a lot better:



Though I can't seem to be able to get to the ML menu and this probably won't help you figure out lossless compression so -- never mind!
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

theBilalFakhouri

  • Contributor
  • Senior
  • *****
  • Posts: 356
Re: ProcessTwoInTwoOutLosslessPath
« Reply #230 on: February 19, 2018, 12:48:18 PM »
Hello a1ex

at 23.976fps 1736x976 resolution - ISO 100:

14-bit uncompressed = 14-bit lossless (with compression ratio of ~57%).
12-bit uncompressed ≠ 12-bit lossless (negative gain fake lossless with enhanced compression ratio of ~50% compared to native 14 bit ).
10-bit uncompressed ≠ 11-8 bit lossless (negative gain fake lossless with enhanced compression ratio of ~45% compared to native 14 bit).

So the 67.7mb/s became ~38.4mb/s (in 14-bit).

Now the 10-bit uncompressed has write speed of 48.4mb/s ,if it was native 10bit lossless, Will we have same as 57% compression ratio
of 14-bit/14-bit lossless compared to 10-bit/10-bit native lossless?

The 48.4mb/s (10-bit) will become ~27mb/s (10-bit native lossless) instead of the current 11-bit lossless with write speed of ~32.4mb/s ?

Or it's just the same ?

I hope I talked well  :P
700D 1.1.5 | no more ISOless LV err 8

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #231 on: February 20, 2018, 01:33:34 AM »
Well guess what
Code: [Select]
******5D2.2.1.2******
loc_ff1beb18: ; 5 refs
found it now , thanks @dfort  ;D


Code: [Select]
*****5D3.1.1.3******
loc_ff32330c: ; 10 refs


AH Ha !! 
This should help me advance this faster for D4

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #232 on: February 20, 2018, 08:42:09 AM »
Code: [Select]
****5d2.2.1.2****from dfort****
TTL_RegisterCBR = (void *) 0xFF2DDA1C;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
Checking address FF2DDA1C  in 5d2 and does not really add up like the other one from 5d3



This the way it is for digic5
Code: [Select]
****5d3.1.1.3****
TTL_RegisterCBR = (void *) 0xFF3D3774;  /* RegisterTwoInTwoOutLosslessPathCompleteCBR */



After some time searching etc. .... I think I found the pattern for 5d2 but it has a different name/operation  then digic 5/5d3
I believe  FF023288 is the one , but not 100% sure
Code: [Select]
TTL_RegisterCBR = (void *) 0xFF023288;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */

Looks the same (almost) expect for the address , how I'm I convinced ? I searched "e92d401c" (there was 150  :o , lucky guess)
and looked for something to do with  "TwoInTwoOutJpeg" I didn't find it but I found something that's close (I hope)
Code: [Select]
*"PreDevelop\PreDevelop.c"the digic5 has
Code: [Select]
./DevelopPath/TwoInTwoOutLosslessPath.cI will not know for sure 100% until I test but looks like maybe .

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #233 on: February 20, 2018, 02:07:55 PM »
After some time searching etc. .... I think I found the pattern for 5d2 but it has a different name/operation  then digic 5/5d3
I believe  FF023288 is the one , but not 100% sure
Code: [Select]
TTL_RegisterCBR = (void *) 0xFF023288;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */

Good you're checking my work. I was way off but take a look at this one, it seems to be a better match than what you found:

Code: [Select]
TTL_RegisterCBR = (void *) 0xFF258E70;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
Now this is where we're at:

Code: [Select]
    if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutJpegPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF1BEB18;  /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF259B58;  /* called right after ProcessTwoInTwoOutJpegPath */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF258E70;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF0AA9D4;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF259E3C;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF259E74;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF259EAC;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

When I first looked at this I thought maybe I made a mistake with TTL_SetArgs and TTL_SetFlags because those addresses are quite a distance from the rest of the addresses and the comments seem to indicate that this is a block of code with the functions living very close to each other. Let's do a different kind of pattern matching and look at some other cameras. Do their TTL_SetArgs and TTL_SetFlags functions also come quite a bit earlier in the disassembly?

Code: [Select]
    if (is_camera("EOSM", "2.0.2"))
    {
        /* ProcessTwoInTwoOutJpegath, EOSM 2.0.2 */
        TTL_SetArgs     = (void *) 0xFF361498;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF429210;      /* called right after ProcessTwoInTwoOutJpegath(R) Start(%d); */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF4281F4;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF36D124;      /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF4292B8;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF428440;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF429328;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

Yes. Note that like the 5D2 the EOSM also uses ProcessTwoInTwoOutJpegath while the 5D3 uses ProcessTwoInTwoOutLosslessPath. Probably a brother from a different mother but they are still related.

When looking for stubs I often look for the pattern on a few different cameras. You might stumble onto a solution for an issue you weren't even looking for. That's what happened when I was hunting for the EOSM2 stubs. Start with this post then jump to this one and what a1ex posted right after it.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 697
Re: ProcessTwoInTwoOutLosslessPath
« Reply #234 on: February 20, 2018, 08:30:15 PM »
How necessary is DecodeLosslessPath for lossless recording? Also would it be dangerous to compile and run it "where we're at" right now?

Is the pattern matching just finding really similar blocks of assembly instructions in the disassembled code of different cameras?
5D2, MLV App

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #235 on: February 20, 2018, 10:49:58 PM »
There are three sections in lossless.c that need to be figured out. My suggestion would be to look at a recent pull request that got this working on another camera, for example:

https://bitbucket.org/hudson/magic-lantern/pull-requests/864/crop-rec-4k-eos100d-lossless/diff

Is DecodeLosslessPath necessary? Would it be dangerous to compile and run this? Try it and let us know if you and your camera survive.  ;D

The pattern matching I've been talking about is pretty much as you described it. Sometimes it is an obvious match that is easy to find, other times it is like looking for a needle in a haystack.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #236 on: February 21, 2018, 06:12:39 AM »

Code: [Select]
FF258E70Yes , that much better , I see I didn't look anymore after I found that one .
Code: [Select]
e92d401c
this seem to be a very important , as all stubs so far has it in the chain .
Did a quick check and there 750 references with e92d401c , that address was about half way thought.

Yea ,  as I search for the two in two out stuff I keep getting side tracked -- I see stuff that I can use in 3.5k/UHD
and compressed Full Lv , 4K etc. ... (mjpeg) and the best so far I see the hdmi stuff I want to change (hopefully loss the pillarbox at least 5d2)

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #237 on: February 22, 2018, 09:52:49 AM »
Working on

Code: [Select]
if (is_camera("5D2", "2.1.2"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xff******;
        Start_DecodeLosslessPath    = (void *) 0xff******;
        Cleanup_DecodeLosslessPath  = (void *) 0xff******;
    }
I can't seem to find DecodeLosslessRawPath yet but coming close
Code: [Select]
ff253b88: e28f0f75 add r0, pc, #468 ; ff253d64: (44776152)  *"RawDecodePath"
ffa48448: e28f2f69 add r2, pc, #420 ; ffa485f4: (4e495754)  *"TWIN DecodeMode == DECODE_TWIN"
ffa48d68: e28f2fb5 add r2, pc, #724 ; ffa49044: (4e495754)  *"TWIN CompJpCoreDecode"
ff1889e0: e28f2f7e add r2, pc, #504 ; ff188be0: (61745320)  *" StartJpegTwinDecoding %d"
Nothing about LossLess , wondering if TWIN DecodeMode may be related ?

the search continues

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #238 on: February 23, 2018, 03:20:03 AM »
@dfort how did you find "DecodeLosslessRawPath" on 7D
The decode, "RequestDecodeLosslessRawPath" on the 7D, seems pretty straightforward:

Code: [Select]
    if (is_camera("7D", "2.0.3"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xFF2E8EAC;
        Start_DecodeLosslessPath    = (void *) 0xFF2E8F5C;
        Cleanup_DecodeLosslessPath  = (void *) 0xFF2E906C;
    }
I have trouble finding any good reference  to it on 5d2 , or is it called something else I'm not a ware of

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #239 on: February 23, 2018, 05:19:36 AM »
Found it in the 7D ,
Code: [Select]
  *"RequestDecodeLosslessRawPath"but not in 5D2 , must be called something else

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 697
Re: ProcessTwoInTwoOutLosslessPath
« Reply #240 on: February 24, 2018, 02:00:03 AM »
I feel we're getting close to lossless on the 5D2.

Is DecodeLosslessPath necessary? Would it be dangerous to compile and run this? Try it and let us know if you and your camera survive.  ;D
:D
I tried, but there seems to be a function that is missing from the 5D2 codebase that won't let it link. I'll try to get rid of that error tomorrow.

@reddeercity have you had any luck compiling, is it even time to try it yet?

I've also been thinking about a pattern matching script...
5D2, MLV App

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #241 on: February 27, 2018, 08:46:25 AM »
@Ilia3101 it's not ready yet , it's missing stubs for
Code: [Select]
*"RequestDecodeLosslessRawPath

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #242 on: February 27, 2018, 09:00:21 AM »
@a1ex It seem D4 (5d2) doesn't use
Code: [Select]
*"RequestDecodeLosslessRawPathor I can't find it like the 7d but I did find something very close it maybe the same not sure .
Code: [Select]
ff24a178: e28f2f4d add r2, pc, #308 ; ff24a2b4: (4e495754)  *"TWIN StartResWinResizeDecode"
ff24a1d0: e28f2c01 add r2, pc, #256 ; ff24a2d8: (4e495754)  *"TWIN StartTwinDecode "
ff24a1f8: e28f20f0 add r2, pc, #240 ; ff24a2f0: (4e495754)  *"TWIN DECODE_WINRES "
ff24a218: e28f20e4 add r2, pc, #228 ; ff24a304: (4e495754)  *"TWIN DECODE_TWIN "
ff24a260: e28f20d4 add r2, pc, #212 ; ff24a33c: (4e495754)  *"TWIN StartTwinDecode->DecodeEngineStart"
ff24a374: e28f2e13 add r2, pc, #304 ; ff24a4ac: (4e495754)  *"TWIN StopDecWinPath Error:%d"
and this also
Code: [Select]
ff248240: e28f2e2e add r2, pc, #736 ; ff248528: (4e495754)  *"TWIN DecodeEngineStart EngineCB:0x%lX"
ff248270: e28f2fb6 add r2, pc, #728 ; ff248550: (4e495754)  *"TWIN 1_WrCH:%#lx, 2_WrCH:%#lx"
ff248330: e28f2f8e add r2, pc, #568 ; ff248570: (4e495754)  *"TWIN EngineStop EngineCB: 0x%lX"
ff248368: e28f2e22 add r2, pc, #544 ; ff248590: (4e495754)  *"TWIN EngineCB = READ_DMAC_BIT"
ff24840c: e28f2f6f add r2, pc, #444 ; ff2485d0: (4e495754)  *"TWIN EngineState == STATE_DECODE"
ff248448: e28f2f69 add r2, pc, #420 ; ff2485f4: (4e495754)  *"TWIN DecodeMode == DECODE_TWIN"
Do you think these could be what we are after on digic 4 models ?
there is no lossless decode I can see it all this *"TWIN stuff , hope you can shed light on this  :D

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #243 on: February 28, 2018, 02:11:32 AM »
...there is no lossless decode I can see it all this *"TWIN stuff...

That *"TWIN stuff is also in the 7D so it doesn't really bring you any closer to finding DecodeLossless in the 5D2. Maybe a1ex can verify this idea, use the dm-spy-experiments branch to log what happens viewing a CR2 still. CR2 files use lossless compression so viewing them is probably decoding the raw file. Then again maybe the LCD is just displaying the jpeg that's embedded in the CR2 file.

As dfort guessed, all of this is found by following Canon's CR2 compression routines (codenamed Mem1ToRaw). Unfortunately, there are differences between models, so it's not always just a pattern matching.

Mem1ToRaw is in the 5D2 but I keep getting lost trying to figure out where the decoding functions are located.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #244 on: February 28, 2018, 02:47:39 AM »
Maybe a1ex can verify this idea, use the dm-spy-experiments branch to log what happens viewing a CR2 still. CR2 files use lossless compression so viewing them is probably decoding the raw file.
I did this a couple of weeks ago but I wasn't sure how to decipher until I start to read the disassembly
it make more sense to me now .
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0001_cr2_2018_2_12.log
found this  so far  , after I finish analyzing the dm-cr2 log I'll perform another test .
Code: [Select]
50EA1> ImgPlayDrv:ff98b9e8:19:03:  ClearDecodeJpegPath 414
50EDF> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
50F0C> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
50F35> ImgPlayDrv:ffa3b688:1a:02: DEC CancelDecodeJpeg Rdmac:1 Wdmac:0 Jp:0 Er:0
50F76> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
50F9D> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
52061>    CtrlSrv:ff98acd0:19:03:  (PUB) ExecuteJpegDecoding 2057
52330> ImgPlayDrv:ff9886a8:19:03:  StartJpegDecoding 614

Code: [Select]
524D9> ImgPlayDrv:ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:1) 193
527CB> ImgPlayDrv:ffa3b324:1a:04: DEC RequestDecodeJpeg Before Lock
52803> ImgPlayDrv:ffa3b340:1a:04: DEC RequestDecodeJpeg After Lock

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #245 on: February 28, 2018, 04:27:57 AM »
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0003_Photo_Liveview_reviewing_cr2.log
much more here now , dm-log in Photo Liveview reviewing cr2  , Found this , it not in the other log
Code: [Select]
717A3> FrontShtDe:ff88ad78:96:05: sdsMem1ToRawCompressionthis also
Code: [Select]
72185> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5702)
72207> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5702)��(SemOK)
7224B> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5702)

Code: [Select]
725FC> FrontShtDe:ffa59694:16:03: [TTJ][150,5702,0] RAW(5792,3804,0,14)
726FC> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1ae2c0f0
72733> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864
72998> RearShtDev:ff887524:97:05: sdfGetDevelopDevType(5702)(0x7000)
96EBC>    ml_init:ff9b1dc8:91:01: SendDataToDfe[0x180e0000]
9C5F3> **INT-64h*:ffa59470:16:03: [TTJ][150,5702,0] JpCoreCompleteCBR( 0x14469ba )
9C719> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x344ea36 WR2:0x1aea8248
9C763> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x124cf564 RD2:0x124d1864
9CAA5> FrontShtDe:ff888f10:96:05: ProcessTwoInTwoOutJpegPath(R) End(5702)
9DB0B> FrontShtDe:ff88b008:96:05: sdsMem1ToJpegDevelop(5702)
9DB53> FrontShtDe:ff8890a0:96:05: sdfExecuteMem1ToJpegPath(5702)
9DBD5> FrontShtDe:ff889290:96:05: ProcessTwoInTwoOutJpegPath(J) Start(5702)

This looks very interesting
Code: [Select]
C8C90> FrontShtDe:ff887068:97:05: (ISR)sdcCompleteFrontDevelop::DisplayJpegComplete
2AC> FrontShtDe:ff888418:96:05: sdfCompleteFrontDevelop(5702)
C92F9> FrontShtDe:ff888444:96:05: ->spsComplete ExtractStageJob(5702)
C93BB> RearShtDev:ff887f40:97:05: sdsYuvToDcfCompression(5702)(0,0)��(0x80000006)
C9435> RearShtDev:ff887f40:97:05: sdsYuvToDcfCompression(5702)(0,1)��(0x80000003)
C947E> RearShtDev:ff8869a8:97:05: sdsExecuteYuvToDcfPath(5702)
C954A> RearShtDev:ff886a4c:97:05: ProcessYuvToDcfPath(5702)
:o :o :o something close
Code: [Select]
DEEE5> ImgPlayDrv:ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:0) 193
DF1D2> ImgPlayDrv:ffa3b324:1a:04: DEC RequestDecodeJpeg Before Lock
DF206> ImgPlayDrv:ffa3b340:1a:04: DEC RequestDecodeJpeg After Lock
getting warmer
Code: [Select]
E05F4> ImgPlayDrv:ff98b9e8:19:03:  ClearDecodeJpegPath 414
E0635> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
E0660> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
E068B> ImgPlayDrv:ffa3b688:1a:02: DEC CancelDecodeJpeg Rdmac:1 Wdmac:0 Jp:0 Er:0
E06C9> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
E06EF> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #246 on: February 28, 2018, 04:58:38 AM »
this seems to be the one closest 
Code: [Select]
E1C28> ImgPlayDrv:ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:1) 193

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #247 on: March 01, 2018, 01:16:40 AM »
@reddeercity -- Keep at it and you'll have the entire log posted.  :P

There is a branched called compressed_raw where the experiments were done on the 5D3. I tried making changes on it and compiling for the 7D but it doesn't work at all. Then I tried getting the EOSM working on that branch. Should be easy, right? After all the EOSM is already has lossless compression working--yeah, not so easy. The compressed DNG's are coming out all white or all black. Can't seem to figure it out. I'll keep trying because this branch is all about getting just lossless compression working. The crop_rec_4k branch has lossless but it also includes 8...12 bit reduced bit depth and the crop_rec module so you're going to have to juggle several babies to get it working.

I looked for *"RequestDecodeLosslessRawPath" and it isn't in any of these Digic 4 cameras:

Code: [Select]
5D2
50D
500D
550D

However, it is in these:

Code: [Select]
7D
60D
600D

The "interesting" stuff you are posting are functions that are also in the 7D firmware and these routines aren't being used for decoding lossless.

My suggestion is not to worry about the decoding stuff for now. Check the commits made on lossless.c and you'll find out that it only appears to be useful in mlv_play so don't worry about it until you can successfully save a lossless compressed MLV clip.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2095
Re: ProcessTwoInTwoOutLosslessPath
« Reply #248 on: March 01, 2018, 05:41:21 AM »
compiled the "compressed_raw" branch and it locks up the camera (5d2) when trying to load either mlv_lite.mo or silent.mo
Locks up at the sense cleaning , need to pull battery , but full mlv loads find and works records 10, 12 & 14 uncompressed raw etc.
Needs more work to find what's locking it up. I'll try the "crop_rec_4k branch" with lossless and see if it makes a difference

Edit: "crop_rec_4k branch" with lossless will not compile (raw.mo errors)
keeps messing up on  raw.c "digital gain" stuff line 2336  :-X

Edit:#2 Got it to compile finally , but mlv_lite didn't make , all the rest did .
loaded silent.mo , still lockups the camera - a little good new 10 ,12bit full mlv  works better in 1:1 but that's off topic .

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3739
Re: ProcessTwoInTwoOutLosslessPath
« Reply #249 on: March 02, 2018, 03:06:02 AM »
@reddeercity -- Getting it to compile is rather easy, getting it to work is much harder. Did you put in the 5D2 specific changes in lossless.c? One thing that is still missing for the 5D2 is the "resources[]" which should be in your dm logs but they aren't. Check out Reply #148 for an example and Reply #130 for a deeper dive into how this code works.

Today I thought I'd see what changes that need to be made so that the 7D will compile on the crop_rec_4k branch with a working silent module. Turned out there were only a few things to change:

src/gdb.c
Code: [Select]
-    unpatch_memory(bkpt->address);
+//    unpatch_memory(bkpt->address); // patch and unpatch not working on 7D?
Code: [Select]
-    patch_instruction(bkpt->address, MEM(bkpt->address), GDB_BKPT_OPCODE, "GDB hook");
+//    patch_instruction(bkpt->address, MEM(bkpt->address), GDB_BKPT_OPCODE, "GDB hook");  // patch and unpatch not working on 7D?

Not that I know what this code does so I might have short circuited the whole thing here?

In addition:

src/raw.c
Code: [Select]
+#if defined(LV_STATE)  /* older Digic 4 */
+#define BLACK_LEVEL 2047
+#endif

This is copied from the EVF_STATE code -- all cameras that work on crop_rec_4k at this point are EVF_STATE, right? Again, I'm not sure what I'm doing but it won't compile without defining BLACK_LEVEL. I made it a separate block of code in case LV_STATE cameras need a different value though I have a feeling that it will probably be the same.

Never mind that many modules fail to build, all I'm looking for right now is a silent module that can save a simple lossless compressed DNG. It will save an uncompressed DNG but hangs when saving a lossless. No crash logs.

I made a fake pull request to make it easy to see what changes I made. Note that I also copied in sections of the dm log files to show where I got the "resources[]" addresses.

This feels so close, yet so far.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102