Author Topic: ProcessTwoInTwoOutLosslessPath  (Read 130313 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12324
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #50 on: December 30, 2016, 09:12:14 AM »
Hypothesis: 4C XX is a relative jump of XX*2 bytes. It's used to guard the strings in the firmware (appears to jump over them).

Example (400D):
Code: [Select]
0000027e  4c 1a 6c 61 75 6e 63 68  65 72 2e 73 20 72 65 76  |L.launcher.s rev|
0000028e  2d 31 2e 30 20 32 30 30  31 2f 30 37 2f 31 37 20  |-1.0 2001/07/17 |
0000029e  61 75 74 68 6f 72 20 53  61 74 6f 73 68 69 20 4e  |author Satoshi N|
000002ae  61 69 74 6f eb 01 6a 17  ae 21 ba 07 a2 2f 8b 20  |aito..j..!.../. |

0x27e + 2 * 0x1A = 0x2b2 (right after the string, which is not null-terminated)

This matches most of the strings, with one exception (not sure how to interpret it):
Code: [Select]
00000400  4c c2 20 65 6e 63 64 65  63 5f 6a 70 65 67 20 72  |L. encdec_jpeg r|
00000410  65 76 32 2e 31 31 20 32  30 30 34 2f 30 32 2f 31  |ev2.11 2004/02/1|
00000420  32 20 59 75 6b 69 6f 20  43 68 69 62 61 20 26 20  |2 Yukio Chiba & |
00000430  53 61 74 6f 73 68 69 20  4e 61 69 74 6f 20 8a 00  |Satoshi Naito ..|

Side note: last night, jpaana suggested it might be a TMS320 DSP.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #51 on: December 30, 2016, 11:55:52 AM »
already tried TM320 but didnt work.
what i might try is to swap the endianess before decoding, but that would be kind of odd.

about string in the firmware it also could be some "store x following words somewhere" instruction
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #52 on: December 30, 2016, 12:24:43 PM »
or better test all variants of the TMS320.. hold on..
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Ant123

  • Contributor
  • Member
  • *****
  • Posts: 121
Re: ProcessTwoInTwoOutLosslessPath
« Reply #53 on: December 30, 2016, 02:04:21 PM »
List of cameras based on TI's DSP...

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #54 on: December 30, 2016, 03:26:50 PM »
the TMS320 processor family varies a lot with their opcode layout.
so code compiled for TMS320C54 will look totally different on TMS320C55.
got in touch with these when hacking nokia phones ages ago.

although when using TMS32054 for decompilation, the code looks senseful in terms of instruction alignment.
a1ex, you remember our chat about opcode length - mixed 16 and 32 bit instructions.

according to http://magiclantern.wikia.com/wiki/Datasheets we have tracks of a TMS320DM36x in a 5DII

so lets try this one.. unfortunately my IDA pro doesn't have this processor :(
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #55 on: December 30, 2016, 04:03:40 PM »
this might also be a candidate
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

stevefal

  • Contributor
  • Hero Member
  • *****
  • Posts: 528
Re: ProcessTwoInTwoOutLosslessPath
« Reply #56 on: December 31, 2016, 02:51:26 AM »
Hey guys, I noticed this news on the interwebs. Exciting. I'll surely be back to give video a try if it leads to that.
Steve Falcon

71m363nd3r

  • New to the forum
  • *
  • Posts: 17
Re: ProcessTwoInTwoOutLosslessPath
« Reply #57 on: December 31, 2016, 04:57:14 AM »
@a1ex
@g3gg0

I can understand that we go with DNG, but what do you think to push it farther with OpenEXR https://en.wikipedia.org/wiki/OpenEXR it's full opensource standard and better compresion.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12324
  • Emergencies only
Re: ProcessTwoInTwoOutLosslessPath
« Reply #58 on: December 31, 2016, 11:01:17 AM »
Right now, we are still limited by the compression engine implemented by Canon; its output simply happened to be valid for both CR2 and DNG, so we just changed the header and did some minor layout tweaks. Not sure what's the point of using other formats in this case.

If we manage to identify the DSP, that might be a different story.

Some other notes:



I've attempted to verify the hypothesis about 4C XX (relative jumps). On 60D, I did the following changes (works with either Debug -> Memory Browser or as C code):

Code: [Select]
/* hardcoded for 60D */
uint8_t * jpcore = (void *) ((MEM(0x350A0) + 31) & ~32);

/* original: jpcore[4:5] = 4C 7E => probably jumps to 0x100;
 * jpcore[0x30] is in the middle of a string (400D) or spaces (60D)
 */

jpcore[0x5]  = (0x30 - 0x4) / 2;       /* jump to 0x30 instead */
jpcore[0x30] = 0x4C;                   /* from 0x30, jump to 0x100 */
jpcore[0x31] = (0x100 - 0x30) / 2;

Result: success (can take photos, can review them).

Next change (on top of the others):
Code: [Select]
jpcore[0x31] = 0;                       /* infinite loop */

Result: crash when taking photos (JPCORE interrupt not triggered).



On 60D, I've found the JP62 (H.264) firmware at 0xFF647D70 (initialization visible in QEMU, unlike 5D3). The code frequencies don't seem to match the regular JPCORE at all, so I guess it may be a different architecture.

16-bit codes with 16-bit alignment:
Code: [Select]
Code        60DJP62
    0000:     5288
    0ca8:      306
    0010:      301
    0020:      255
    0001:      238
    20c0:      149
    0c0c:      124
    5600:      123
    f01d:      113
    1ca8:      108

garry23

  • Contributor
  • Hero Member
  • *****
  • Posts: 1857
Re: ProcessTwoInTwoOutLosslessPath
« Reply #59 on: December 31, 2016, 11:17:05 AM »
@Danne (@A1ex)

Thanks for sharing the test version for the 5D3-113.

I've just tried it out and managed a single lossless DNG, but I had,'t appreciated that this this meant much less pixels! That is my lossless DNGs are 1932 x 1290.

Also, I tried using my FRSP LE script, but the Lua call (camera.shoot(64, false) or camera.shoot(false)) triggers the mechanical shutter.

Cheers

Garry

garry23

  • Contributor
  • Hero Member
  • *****
  • Posts: 1857
Re: ProcessTwoInTwoOutLosslessPath: UPDATE
« Reply #60 on: December 31, 2016, 12:15:38 PM »
OK, I think I worked out 'my mistakes'.

I don't think the lossless test build included the Lua fix.

Plus I hadn't switched to Full Res !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I'll wait a little :-)

calypsob

  • Freshman
  • **
  • Posts: 57
Re: ProcessTwoInTwoOutLosslessPath
« Reply #61 on: December 31, 2016, 11:47:14 PM »
 Will this feature only be available on 5D Mark III?
Full spectrum T2i
T3i

goldenchild9to5

  • Hero Member
  • *****
  • Posts: 534
Re: ProcessTwoInTwoOutLosslessPath
« Reply #62 on: January 02, 2017, 03:28:18 AM »
@calypsob Yeah it will be available for the 5D III when they figure it all out  8)

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: ProcessTwoInTwoOutLosslessPath
« Reply #63 on: January 02, 2017, 03:58:25 AM »
"My kingdom for A1ex's brain"
 ;)

gamedesign

  • New to the forum
  • *
  • Posts: 11
Re: ProcessTwoInTwoOutLosslessPath
« Reply #64 on: January 02, 2017, 05:18:42 PM »
Will this feature bring any benefits to RAW video on the 7D (mark 1)?
I currently use MLV all the time, its brilliant.

Licaon_Kter

  • Hero Member
  • *****
  • Posts: 519
  • M
Re: ProcessTwoInTwoOutLosslessPath
« Reply #65 on: January 03, 2017, 09:25:06 PM »
Forum overload in 3...2...1... https://www.dpreview.com/news/2451079037/magic-lantern-proof-of-concept-shows-in-camera-dng-shooting

That being said, looks like I need to raise the issue or DNG lossless compression support in Darktable again :(

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6905
Re: ProcessTwoInTwoOutLosslessPath
« Reply #66 on: January 03, 2017, 09:45:22 PM »
At least there might be a chance to comment on Petapixel not getting the point and put it right.
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #67 on: January 03, 2017, 10:25:54 PM »
noone realized that it is mainly a saving format meant for silent pictures
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Licaon_Kter

  • Hero Member
  • *****
  • Posts: 519
  • M
Re: ProcessTwoInTwoOutLosslessPath
« Reply #68 on: January 03, 2017, 11:05:56 PM »
noone realized that it is mainly a saving format meant for silent pictures
Details... details... that would need someone to actually read the thread, not only to grab the DNGs...  research in journalism, pffft

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6905
Re: ProcessTwoInTwoOutLosslessPath
« Reply #69 on: January 03, 2017, 11:15:11 PM »
Hey, no more ranting or "the company" may get sued ...
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3154
Re: ProcessTwoInTwoOutLosslessPath
« Reply #70 on: January 03, 2017, 11:32:22 PM »
this wasn't meant as negative comment.

well, it might be interesting for raw video too, but tests till now didn't show any way to make the JPCORE work in LV synchronized to frame display.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6905
Re: ProcessTwoInTwoOutLosslessPath
« Reply #71 on: January 04, 2017, 12:12:03 AM »
I haven't archived PetaPixel's post. There was some makeover, I think. Some corrections + introducing some weird concept (I take the liberty to say so).
Quote
If and when this feature arrives, you won’t have to worry about quality loss from converting your Canon RAW files into DNGs ...

Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

Licaon_Kter

  • Hero Member
  • *****
  • Posts: 519
  • M
Re: ProcessTwoInTwoOutLosslessPath
« Reply #72 on: January 04, 2017, 12:57:31 AM »
I haven't archived PetaPixel's post. There was some makeover, I think. Some corrections + introducing some weird concept (I take the liberty to say so).
And a lot of comments discuss that (quality change on conversion), not sure why would anyone would think that.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: ProcessTwoInTwoOutLosslessPath
« Reply #73 on: January 04, 2017, 07:23:05 AM »
not sure why would anyone would think that.

The world is flat and there's a magical fairy that sends you to hell if you misbehave.

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6905
Re: ProcessTwoInTwoOutLosslessPath
« Reply #74 on: January 04, 2017, 08:42:12 AM »
And a lot of comments discuss that

Just found out our proxy blocked PetaPixel's comments. Thanks for the info!
Got it wrong about makeover, too.
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!