Magic Lantern Forum

Developing Magic Lantern => Reverse Engineering => Topic started by: a1ex on December 18, 2016, 09:06:41 PM

Title: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 18, 2016, 09:06:41 PM
Managed to call ProcessTwoInTwoOutLosslessPath, which appears to perform the compression for RAW, MRAW and SRAW formats. The output looks like some sort of lossless JPEG, but we don't know how to decode it yet (this should help (http://lclevy.free.fr/cr2/#lossless)).

Proof of concept code (photo mode only for now): http://bitbucket.org/hudson/magic-lantern/commits/0e87b86
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 18, 2016, 10:33:23 PM
Proof of concept code works! Managed to decode a valid image with a tiny patch to dcraw.c.

https://bitbucket.org/hudson/magic-lantern/commits/06aedb0

Processing time: 120ms for 5920x3950 (14bpp), 11ms for 1920x1080 (not decoded).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on December 18, 2016, 11:06:57 PM
What do you mean with the 1920 x 1080 not decoded ?
Do you mean you can make the file but can't decode it ?
Isn't this standard canon cr2 files what you're making here, which should be read by almost all raw editors out there.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 18, 2016, 11:32:22 PM
I simply changed the buffer size to 1920x1080 to check the processing speed, and I did not attempt to decode it. That test was done outside LiveView.

Seconds ago, I've also managed to run the test in LiveView, and the output appears valid (though distorted - wrong pitch). Should be fixable.

It is standard CR2, without header, embedded jpeg and whatever else it might contain.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on December 19, 2016, 02:29:20 AM
Great news!
From your link a1ex
Quote(5D Mark II sraw1) 
JPEG_SOF3: length=17  bits=15, wide=3872, high=2574
(5D Mark II sraw2) 
JPEG_SOF3: length=17 bits=15, wide=2808, high=1872
So does that mean 15bit Jpeg compression in raw(cr2) ? or is that referring to something else

Edit: after reading more it appears sraw is YCbCr 4:2:2.
interesting !
QuotesRaw and sRaw2 (and surely sRaw1) are encoded in YCbCr format, and not as CFA RGB data like full RAW
•sRaw and sRaw2 use a YCbCr 4:2:2 chroma subsampling encoding : 2 Chroma values on the 1st Row of 4 pixels and 2 Chroma values on the 2nd row of 4 pixels,
•and sRaw1 use a 4:2:0 encoding : 2 Chroma values on the 1st Row of 4 pixels and 0 Chroma value on the 2nd row of 4 pixels
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 19, 2016, 08:43:30 AM
I've only tested regular RAW lossless compression, but other methods are available.


run_test:ff3d3f34:16:03: [TTL][0,0,0] RAW(5920,3950,0,14)
run_test:ff28df90:15:02: [JPCORE] SetJp57EncodeLosslessType 1
run_test:ff28f7d4:15:03: [JPCORE] SetJp57EncodeLosslessParam 2960,3950           <- note: 2 slices

run_test:ff3d3fe0:16:03: [TTL][0,0,0] SRAW(2880,1920,0,15)
run_test:ff28df90:15:02: [JPCORE] SetJp57EncodeLosslessType 2
run_test:ff28f8a8:15:02: [JPCORE] SetJp57EncodeYuv422LosslessParam 2880,1920

run_test:ff3d3f88:16:03: [TTL][0,0,0] MRAW(3960,2640,0,16)
run_test:ff28df90:15:02: [JPCORE] SetJp57EncodeLosslessType 3
run_test:ff28f97c:15:02: [JPCORE] SetJp57EncodeYuv420LosslessParam 7920,1320


The SRAW format appears to use 15-bit YUV422 internally.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on December 19, 2016, 08:55:41 AM
Now the question most of us are thinking off   ???
Does get some chip(probably the digic) fried inside when this functions is used 25 frames per second  :P
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: chmee on December 19, 2016, 10:22:07 AM
imho: i didnt like the mraw/sraw-Files because of the poor quality. Its using some kind of demosaicing inside the body to render these files. a lot of antialiasing and a noticeable hue-shift.

but in terms of video it should get a better output than the original video-processing-chain.

benefit: using the inbody demosaicing/rendering to yuv-(video)files
drawback: quality

regards chmee
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 19, 2016, 01:08:45 PM
i played a bit with it yesterday.
in theory it would mean that we can compress frames to 50%-60% of the original size.
for raw video this would give us a higher video resolution for the same card speed.
compression speed is - at least in photo mode - good enough. LV measurement while card is writing etc has to be done.

*if* we can somewhen manage to run it in our raw recording environment.
the JPEG core is a bit picky with accesses and can even lock up the whole camera just by reading or writing its registers at the wrong point in time..

@chmee: we are not talking about mRaw/sRaw but the original cr2 raw (cRaw). this engine handles all three of them and a1ex' numbers were the cRaw ones.
we didn't try the other ones properly yet. maybe its good for raw video. maybe not. all to be tested.


theres a good chance that it wont work nicely all together (LV, CFDMA, JPEG) and causes hickups, crashes etc...
a1ex opened the door to in-camera compression. the next step is to check if that door is usable for e.g. raw video.
and if not, for what else we could use it :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on December 19, 2016, 02:02:52 PM
Can't resist too ask  ;D
Does it handle 10 bit files ?




Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 19, 2016, 09:17:31 PM
I've just got a valid LiveView image (decoded successfully). 2080x1080 14bpp, processing time 23ms, file size about 1.9MB (uncompressed 3.75MB).

You can feed 10-bit data by faking the input dimensions, but the predictors won't do a good job in this case. A quick test resulted in a compressed size of... 4.2MB.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 19, 2016, 09:39:30 PM
instead of 3.75MiB?
a really good compression ;D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 19, 2016, 09:42:03 PM
... instead of 2080x1080x10bpp = 2.68MB :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dmilligan on December 19, 2016, 09:53:33 PM
 haha
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 19, 2016, 10:13:31 PM
haha overwhelming :)

but i am sure this is configurable. somehow.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on December 19, 2016, 10:26:24 PM
Wow, nice compression -> 3.75MB to 1.9 MB  :o
Didn't expect that much, most of CR2 files from the 6d are 25MB in size, uncompressed should be about 35MB...


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 20, 2016, 10:11:19 PM
0xC0F371FC is the unpacking mode (bit depth) register for the input module that transfers the image data from memory to the lossless encoder. Same values as PACK(16|32)_MODE. Other input modules with similar functionality are DSUNPACK, ADUNPACK, UNPACK24.

However, the encoder still treats the input data as 14-bit, as the input module pads the data with zeros when unpacking. The decompressed output has 14-bit metadata and levels, but the lower 2 or 4 bits are zero if the input data is 12- or 10-bit. Unfortunately, it doesn't help improving the compression ratio (output is still around 2M).




0xC0E20084 appears to configure the output bit depth and the number of slices. For bit depth, the only valid values appear to be 14 and 15. For 15-bit output, JPEGsnoop reports 15-bit data, but I didn't get numbers higher than 16383 when decoding (higher bits from a 16-bit input stream were trimmed). Changing the number of slices doesn't seem to work, probably some more registers have to be changed.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ilia3101 on December 21, 2016, 12:09:32 AM
I think noobs don't belong here, but is it possible to try this compression on other cameras? Or do they have different registers that need finding? I managed to compile the dm_whateveritscalled branch but don't know what to do with it. Could I be any help testing on the 5D2?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 21, 2016, 10:57:04 AM
@Ilia: will reply as soon as I'll have a complete answer.

Meanwhile, I've tried to slap a DNG header - it almost gets decoded by both dcraw and dng_validate.exe.

There is an issue: Canon divides the image in two slices (link from first post). In DNG, that would require either two strips or two tiles, but the DNG spec requires that metadata should contain individual offset for each strip or tile. Canon output renders cleanly in dcraw as a single image with half width and double height.

So, I've reconfigured the encoder to use two vertical slices (top half and bottom half), instead of default configuration (left half and right half); this happens to be considered as a single logical slice, so it gives valid lossless DNG! Tested with vanilla dcraw and dng_validate.

All that's needed to change in the DNG header is:
- Compression (tag 0x103) set to 7 (JPEG)
- StripByteCounts set to compressed size (otherwise dng_validate will complain)

Therefore, we have all the info required to create valid lossless DNGs directly from camera!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on December 21, 2016, 11:02:58 AM
QuoteTherefore, we have all the info required to create valid lossless DNGs directly from camera!
Ingenious.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: chmee on December 21, 2016, 01:28:23 PM
QuoteTherefore, we have all the info required to create valid lossless DNGs directly from camera!
marvelous.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on December 21, 2016, 04:10:22 PM
  :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 21, 2016, 10:27:04 PM
Proof of concept code available in the compressed_raw branch (silent pictures with lossless DNG output).

Some more details about how I've got valid DNGs:

Take a look at this picture (http://www.magiclantern.fm/forum/index.php?topic=18315.msg176378#msg176378) and look for TTL (TwoInTwoOutLossless) channels. The green one with two ramps (channel #8) has a pretty strange configuration:


(5180, skip 5208) x 3949, 5180, skip -41022212, (5180, skip 5208) x 3950


5180 + 5208 = 10388 = 5936 * 14/8 (horizontal raw resolution, including black bars and a few extra columns present in the raw buffer, but - for some unknown reason - not saved in the CR2)
5180 * 2 = 10360 = 5920 (horizontal CR2 resolution, with dcraw -4 -E)

(5180 + 5208) * 3949 + 5180 - 41022212 = 5180.

Now, the meaning of the above code is clear: read the left half of the image (5180 bytes = 2960 pixels), rewind and read the right half (also 2960). These are the two cr2_slice's (see dcraw code at lossless_jpeg_load_raw and lclevy's docs from first post).

A lossless DNG is decoded with a somewhat similar loop, in lossless_dng_load_raw, which treats the compressed payload as a single slice. With default encoder configuration, dcraw would render a 2960x7900 image (left and right halves stacked vertically).

Now, my encoder mod becomes obvious: double the slice width (so it becomes equal to image width) and halve the height. The encoder doesn't really care what the image layout is - it knows there are two slices (with given dimensions). It uses the data coming from the read EDMAC to fill the first slice, then the second one; therefore, the EDMAC transfer can be configured to just read the image as usual (the first half will be the top of the image, the second half will be the bottom).

In the DNG, the decoder sees a lossless jpg payload; it doesn't really care about the dimensions, as long as the total pixel count matches.

Result: the trick may be a little hackish, but Adobe's dng_validate accepts it :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on December 22, 2016, 12:05:02 AM
My 5D mark III just gave birth to some lossless DNG files. Here,s a compiled version for testing from a1ex compressed_raw branch. Awesome stuff.

5D mark III
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2016Dec21.5D3113_lossless_DNG.zip

Two lossless DNG files
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260001.DNG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260002.DNG
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on December 23, 2016, 04:19:43 PM
@a1ex Are you serious, that's awesome man Team Lantern is pushing things to the limit WOW..  :D  @Danne Images looks great. 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on December 23, 2016, 04:39:30 PM
@Danne Were you able to shoot continuously?  if not how many seconds or frames were you able to capture? 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on December 23, 2016, 04:59:55 PM
It,s coded into silent dng module for now as a1ex is stating. Only shot a few stills. Files about half size from a regular still dng file.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on December 23, 2016, 06:49:03 PM
@Danne That's awesome.. do you know the Bit depth 10Bit, 12, or 14.  When it's fully implemented it's going to be awesome. 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ricbi1 on December 23, 2016, 08:11:40 PM
Unbelievable! You guys are squeezing every inch of performance out of these cameras. Thanks all!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on December 23, 2016, 08:16:56 PM
@goldenchild9to5
QuoteThat's awesome.. do you know the Bit depth 10Bit, 12, or 14
Nice and beefy 14-bit lossless dng files.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ricbi1 on December 23, 2016, 08:28:05 PM
Donwloaded Dec 21 build to test lossless DNG on 5d3, but I can't figure out the workflow to make it work. Anyone has any hints?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on December 23, 2016, 09:26:10 PM
@Danne "Nice and Beefy 14 Bit" lol.. but WOW.. + 1/2 the file size that's just incredible.  Magic Lantern is making it harder and harder to wanna buy another Camera after the 5D3
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Markus on December 23, 2016, 11:02:39 PM
Crazy stuff! Freaking awsome!  :o :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: RenatoPhoto on December 24, 2016, 03:49:06 PM
Ohh my!  This is great.. A1lex said the processing time for a dng was 23 ms..  So at 30 fps each dng would require 33ms..  This means the hardware may be able to compress the dng during video!  :o
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Teamsleepkid on December 24, 2016, 08:44:18 PM
sounds amazing. think this will eventually work at 24fps? would we be able to still have crop mode for less aliasing? could we use more of the sensor in crop mode and get higher resolutions? sorry just wondering.. merry christmas
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: DavidSh on December 24, 2016, 10:35:16 PM
Wow! you guys...
This is really something.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on December 25, 2016, 02:00:28 AM
This DNG development got me too excited.  Happy Holidays everybody..
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: togg on December 25, 2016, 10:15:10 PM
QuoteMy 5D mark III just gave birth to some lossless DNG files. Here,s a compiled version for testing from a1ex compressed_raw branch. Awesome stuff.

5D mark III
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2016Dec21.5D3113_lossless_DNG.zip

Two lossless DNG files
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260001.DNG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260002.DNG


Wait a moment here, does all this means that MAYBE we can get this beautiful cat (nice pic!) on full hd raw video with half the GB/speed needed?
Same process for the 5DII ? (cross finger...) edit: I now understand that probably the CPU isn't up to the task... I guess I'll buy a 5DIII if things work out :D

This is incredible!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 25, 2016, 10:28:49 PM
-> MAYBE on april 1st ;)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 27, 2016, 12:05:30 PM
updated the register map.

houston, it really seems to run its custom firmware. couldnt identify the instruction set type yet.

___
The JPCore seems to an extra processor based engine that exists at three base addresses with equivalent layouts. JPCORE is e.g. used in YUV2JPEG while JP57 in RAW2JPEG. Those seem both to support lossless compression. It also handles MPEG compression, only seen in JP62 yet.

It uses its own firmware in main memory, the address (!) where it was copied into RAM is written to +0x928 before init. Its size is approx 16-18KiB, depending on its exact instance. The address must be aligned to 0x80 bytes.

There is also a buffer address written to 0x92C which is 0x5000 bytes big and starts with 0x500000 and the character sequence "xbuf". Seems not to be used in main firmware, perhaps it is the JPCORE's internal scratchpad memory?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 27, 2016, 12:24:42 PM
i found the "xbuf" reference in the firmware.
tried to understand which kind of CPU we have by looking how the string is used.


Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
000017F0  EC 20 93 4F 43 02 4C 86 BA 66 3E 75 3E 62 3E 78  ì "OC.L†ºf>u>b>x
00001800  6A 12 8B 1F 43 02 4C 81 6A 12 FA 02 52 17 AE 51  j.‹.C.L.j.ú.R.®Q
00001810  C6 18 97 41 41 7D BA 78 3E 64 3E 69 3E 78 6A 12  Æ.—AA}ºx>d>i>xj.
00001820  8B 1F 41 79 AE 13 82 11 82 11 EC 10 82 1F 6E 11  ‹.Ay®.,.,.ì.,.n.


i don't know which kind of CPU it is...

a few bytes later it also loads XIDX
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 27, 2016, 03:05:01 PM
Very interesting.

Some excerpts from QEMU logs, showing where these memory areas are initialized (5D3 1.1.3):

memcpy(0x40649080, 0xff9a0580, 18560) from ff570270
...
[JPCORE] at ShootCa:FF28D28C:00001D1C [0xC0E00928] <- 0x649080  : ???
[JPCORE] at ShootCa:FF28D28C:00001D1C [0xC0E0092C] <- 0x64D9BC  : ???



memcpy(0x40653080, 0xff9a4e00, 16000) from ff5702e8
...
[JP57] at ShootCapt:FF28DDA4:00001D1C [0xC0E20928] <- 0x652F80  : ???
[JP57] at ShootCapt:FF28DDA4:00001D1C [0xC0E2092C] <- 0x656EF0  : ???

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on December 27, 2016, 07:58:54 PM
@a1ex @g3gg0 Man you guy's are genius, don't know how you do it.  Those codes are like Matrix codes to me. 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: RenatoPhoto on December 27, 2016, 09:34:31 PM
The Magic at Magic Lantern!

(http://www.reactiongifs.com/r/shia.gif)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 28, 2016, 02:49:33 PM
Please find a small script that extracts all JPCORE instances from a ROM dump:


import os, sys
from struct import unpack

def getLongLE(d, a):
   return unpack('<L',(d)[a:a+4])[0]

def getShortBE(d, a):
   return unpack('>H',(d)[a:a+2])[0]

d = open(sys.argv[1]).read()

blob_id = 0
for a in range(0, len(d) & ~3, 4):
    v = getLongLE(d, a)
    if v == 0xF4B2:
        v2 = getLongLE(d, a+4)
        if v2 == 0x7E4C:
            sz = getShortBE(d, a + 0x84) * 2
            print hex(a), hex(sz)
            blob_id += 1
            blob = d[a : a+sz]
            fout = os.path.join(os.path.dirname(sys.argv[1]), "JPCORE%d.BIN" % blob_id)
            print " -> %s" % fout
            out = open(fout, "w")
            out.write(blob)
            out.close()
        else:
            print "false alarm at", hex(a), hex(v2)


Very old models (such as 5D and 400D) have some strings in the JPCORE firmware. For example, on 400D:


diagnostics() rev-1.0 2001/07/17 author  Satoshi Naito CM51
launcher.s rev-1.0 2001/07/17 author Satoshi Naito
bypass() rev-1.0 author S.Naito 2001/07/17
encdec_jpeg rev2.11 2004/02/12 Yukio Chiba & Satoshi Naito
predecode_mpeg2() rev-0.10 2001/4/23 author  Akihiro Uehara BH51
decode_mpeg1() rev-0.18 2001/07/09 author  Akihiro Uehara BH51
decode_mpeg2() rev-0.18 2001/07/09 author  Akihiro Uehara BH51
encode_mpeg1() rev-0.20 2001/5/13 author  Satoshi Naito and Akihiro Uehara BH51
encode_mpeg2() rev-0.19 2002/5/13 author  Satoshi Naito and Akihiro Uehara BH51


The JPCORE firmware is also present in PowerShots, and appears to be shared across many cameras. For example, 60D and A1100 (EOS and PowerShot, both DIGIC 4) contain identical JPCORE firmwares. I've tried to group both DSLR and PowerShot models that use identical JPCORE blobs:


(D2) 5D IXUS50 IXUS700 S2IS
(D2) 30D 400D IXUS30 IXUS40
(D3) 1000D 40D 450D A470 A580 A590 A650 A720 G9 IXUS80 IXUS82 IXUS85 IXUS860 IXUS90 IXUS960 IXUS970 SX100IS
(D4) 1100D 1200D 1300D 500D 50D 550D 5D2 600D 60D 7D A1100 A1100 A1200 A2100 A2200 A3200 A3300 D10 G10 G11 G12 IXUS1100 IXUS115 IXUS117 IXUS220 IXUS230 IXUS310 IXUS1000 IXUS100 IXUS105 IXUS110 IXUS120 IXUS130 IXUS200 IXUS210 IXUS300 IXUS870 IXUS95 IXUS980 IXUS990 S90 S95 SX10 SX1 SX120 SX130 SX150 SX200 SX200IS SX20 SX210IS SX220 SX220HS SX230 SX30
(D5) 100D 5D3 650D 6D 70D EOSM G1X S100 SX40
(D5) 700D
(D6) 5D4 750D 760D 7D2M 7D2S 80D EOSM10 EOSM3 G16 G3X G5X G7X IXUS145 IXUS150 IXUS155 IXUS157 IXUS160 IXUS170 IXUS175 IXUS180 IXUS265 IXUS275 S120 SX270 SX280 SX400 SX410IS SX520 SX530 SX600 SX60 SX610 SX700 SX710 SX720

(D2) A410 A420 A430 A450 A460 A530 A540 A550 A610 A620 A700 IXUS55 IXUS60 IXUS65 IXUS750 IXUS800 IXUSIZOOM IXUSW S3IS S80
(D2) A630 A640 A710
(D3) A1000 A2000 A3000 A3100 A3150 A480 A490 A495 A800 E1 SX110
(D3) A560 A570 G7 IXUS70 IXUS75 IXUS850 IXUS900 IXUS950 IXUSI7ZOOM S5IS TX1
(D4) A1300 A1400 A2300 A2400 A2500 A2600 A3400 A3500 A4000 A810 D20 D30 G15 IXUS125 IXUS132 IXUS140 IXUS240 IXUS255 IXUS500 IXUS510 N N-FB S110 SX160 SX170 SX240 SX260 SX260HS SX500 SX50 SX510


Some interesting ideas about how to figure out the meaning of these codes:
https://recon.cx/2012/schedule/attachments/40_Chernov-Troshina.pdf

Frequency of 16-bit codes, 16-bit alignment:

Code         5D3JP  5D3JP57    60DJP    80DJP   400DJP    Total
    0000:      153      250      151      228      314     1096
    f37e:      196      172      196      196      289     1049
    1762:      168      140      168      172      172      820
    02fa:      166      144      166      166      157      799
    7f1a:       97      140       97       97      186      617
    dfae:      109      108      109      109      121      556
    f76e:       82       70       82       82      204      520
    2762:       99       84       99       99      105      486
    41e8:       97       77       97       97       93      461
    04ba:       66       83       66       66      176      457


Frequency of 32-bit codes, 16-bit alignment:

Code         5D3JP  5D3JP57    60DJP    80DJP   400DJP    Total
00000000:      103      196      103      173      254      829
02faf37e:      102       90      102       98       96      488
20202020:      142        0      142      142        0      426
f37e00ec:       76       36       76       76       73      337
f37e02ec:       49       58       49       49       48      253
11cb1762:       45       39       45       45       45      219
176200e8:       43       39       43       43       43      211
304a1fae:        0        0        0        0      178      178
276241e8:       32       27       32       32       28      151
003e04ba:       24       21       24       24       44      137


Frequency of 32-bit codes, 32-bit alignment:

Code         5D3JP  5D3JP57    60DJP    80DJP   400DJP    Total
00000000:       52       99       52       90      131      424
02faf37e:       45       51       45       44       46      231
20202020:       72        0       72       72        0      216
f37e00ec:       33       16       33       35       34      151
f37e02ec:       27       25       27       27       23      129
11cb1762:       23       21       23       24       23      114
176200e8:       22       21       22       21       24      110
304a1fae:        0        0        0        0       76       76
f76a7f1a:       13       10       13       11       26       73
003e04ba:       13       11       13       13       22       72


Frequency of 64-bit codes, 32-bit alignment:

Code                 5D3JP  5D3JP57    60DJP    80DJP   400DJP    Total
0000000000000000:       43       88       43       79      118      371
2020202020202020:       67        0       67       67        0      201
f76a7f1a003e04ba:       10        4       10        9       12       45
5d5202faf37e02ec:       13        0       13       13        0       39
27660f1af162121a:        5        4        5        7        6       27
815202faf37e02ec:        9        0        9        9        0       27
f162121a21c62762:        5        4        5        7        6       27
0aeb008a10fc1fb2:        6        2        6        6        6       26
41e8004a27660f1a:        5        4        5        6        6       26
10fc1fb2017e067e:        6        2        6        6        6       26
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 28, 2016, 04:30:15 PM
Quote from: goldenchild9to5 on December 27, 2016, 07:58:54 PM
Those codes are like Matrix codes to me. 

Good idea. Here's the similarity matrix (Levenshtein ratio) between the JPCORE firmware groups:


         5D   30D 1000D  A560 A1000  A410  A630 1100D A1300  100D  700D   5D4
   5D 1.000 0.962 0.593 0.594 0.587 0.696 0.692 0.669 0.669 0.669 0.669 0.667
  30D 0.962 1.000 0.582 0.583 0.576 0.679 0.676 0.658 0.658 0.658 0.658 0.655
1000D 0.593 0.582 1.000 0.987 0.986 0.815 0.815 0.804 0.804 0.804 0.804 0.803
A560 0.594 0.583 0.987 1.000 0.973 0.816 0.814 0.804 0.804 0.804 0.804 0.803
A1000 0.587 0.576 0.986 0.973 1.000 0.806 0.806 0.814 0.814 0.814 0.813 0.812
A410 0.696 0.679 0.815 0.816 0.806 1.000 0.994 0.963 0.962 0.963 0.962 0.958
A630 0.692 0.676 0.815 0.814 0.806 0.994 1.000 0.966 0.966 0.966 0.966 0.958
1100D 0.669 0.658 0.804 0.804 0.814 0.963 0.966 1.000 0.999 1.000 0.999 0.984
A1300 0.669 0.658 0.804 0.804 0.814 0.962 0.966 0.999 1.000 1.000 0.999 0.984
100D 0.669 0.658 0.804 0.804 0.814 0.963 0.966 1.000 1.000 1.000 1.000 0.984
700D 0.669 0.658 0.804 0.804 0.813 0.962 0.966 0.999 0.999 1.000 1.000 0.983
  5D4 0.667 0.655 0.803 0.803 0.812 0.958 0.958 0.984 0.984 0.984 0.983 1.000


(http://a1ex.magiclantern.fm/bleeding-edge/jpcore/matrix.png)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 28, 2016, 05:03:17 PM
cool way of analyzing
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: DeafEyeJedi on December 28, 2016, 07:39:05 PM
Incredible work @a1ex!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Ant123 on December 28, 2016, 07:50:58 PM
a1ex
Definitely, that guy (https://jp.linkedin.com/in/satoshi-naito-931605112) should know something  :)
Look at his patents...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 28, 2016, 08:50:53 PM
yeah, for sure he knows a lot.
but i also bet he wont tell anything :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 30, 2016, 09:12:14 AM
Hypothesis: 4C XX is a relative jump of XX*2 bytes. It's used to guard the strings in the firmware (appears to jump over them).

Example (400D):

0000027e  4c 1a 6c 61 75 6e 63 68  65 72 2e 73 20 72 65 76  |L.launcher.s rev|
0000028e  2d 31 2e 30 20 32 30 30  31 2f 30 37 2f 31 37 20  |-1.0 2001/07/17 |
0000029e  61 75 74 68 6f 72 20 53  61 74 6f 73 68 69 20 4e  |author Satoshi N|
000002ae  61 69 74 6f eb 01 6a 17  ae 21 ba 07 a2 2f 8b 20  |aito..j..!.../. |


0x27e + 2 * 0x1A = 0x2b2 (right after the string, which is not null-terminated)

This matches most of the strings, with one exception (not sure how to interpret it):

00000400  4c c2 20 65 6e 63 64 65  63 5f 6a 70 65 67 20 72  |L. encdec_jpeg r|
00000410  65 76 32 2e 31 31 20 32  30 30 34 2f 30 32 2f 31  |ev2.11 2004/02/1|
00000420  32 20 59 75 6b 69 6f 20  43 68 69 62 61 20 26 20  |2 Yukio Chiba & |
00000430  53 61 74 6f 73 68 69 20  4e 61 69 74 6f 20 8a 00  |Satoshi Naito ..|


Side note: last night, jpaana suggested it might be a TMS320 DSP.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 30, 2016, 11:55:52 AM
already tried TM320 but didnt work.
what i might try is to swap the endianess before decoding, but that would be kind of odd.

about string in the firmware it also could be some "store x following words somewhere" instruction
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 30, 2016, 12:24:43 PM
or better test all variants of the TMS320.. hold on..
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Ant123 on December 30, 2016, 02:04:21 PM
List (http://www.photo-parts.com.ua/parts/dbphp_ru.php?part=320D) of cameras based on TI's DSP...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 30, 2016, 03:26:50 PM
the TMS320 processor family varies a lot with their opcode layout.
so code compiled for TMS320C54 will look totally different on TMS320C55.
got in touch with these when hacking nokia phones ages ago.

although when using TMS32054 for decompilation, the code looks senseful in terms of instruction alignment.
a1ex, you remember our chat about opcode length - mixed 16 and 32 bit instructions.

according to http://magiclantern.wikia.com/wiki/Datasheets we have tracks of a TMS320DM36x in a 5DII

so lets try this one.. unfortunately my IDA pro doesn't have this processor :(
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on December 30, 2016, 04:03:40 PM
this (http://www.ti.com/lit/ug/sprufe8b/sprufe8b.pdf) might also be a candidate
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: stevefal on December 31, 2016, 02:51:26 AM
Hey guys, I noticed this news on the interwebs. Exciting. I'll surely be back to give video a try if it leads to that.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: 71m363nd3r on December 31, 2016, 04:57:14 AM
@a1ex
@g3gg0

I can understand that we go with DNG, but what do you think to push it farther with OpenEXR https://en.wikipedia.org/wiki/OpenEXR (https://en.wikipedia.org/wiki/OpenEXR) it's full opensource standard and better compresion.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on December 31, 2016, 11:01:17 AM
Right now, we are still limited by the compression engine implemented by Canon; its output simply happened to be valid for both CR2 and DNG, so we just changed the header and did some minor layout tweaks. Not sure what's the point of using other formats in this case.

If we manage to identify the DSP, that might be a different story.

Some other notes:




I've attempted to verify the hypothesis about 4C XX (relative jumps). On 60D, I did the following changes (works with either Debug -> Memory Browser or as C code):


/* hardcoded for 60D */
uint8_t * jpcore = (void *) ((MEM(0x350A0) + 31) & ~32);

/* original: jpcore[4:5] = 4C 7E => probably jumps to 0x100;
* jpcore[0x30] is in the middle of a string (400D) or spaces (60D)
*/

jpcore[0x5]  = (0x30 - 0x4) / 2;       /* jump to 0x30 instead */
jpcore[0x30] = 0x4C;                   /* from 0x30, jump to 0x100 */
jpcore[0x31] = (0x100 - 0x30) / 2;


Result: success (can take photos, can review them).

Next change (on top of the others):

jpcore[0x31] = 0;                       /* infinite loop */


Result: crash when taking photos (JPCORE interrupt not triggered).




On 60D, I've found the JP62 (H.264) firmware at 0xFF647D70 (initialization visible in QEMU, unlike 5D3). The code frequencies don't seem to match the regular JPCORE at all, so I guess it may be a different architecture.

16-bit codes with 16-bit alignment:

Code        60DJP62
    0000:     5288
    0ca8:      306
    0010:      301
    0020:      255
    0001:      238
    20c0:      149
    0c0c:      124
    5600:      123
    f01d:      113
    1ca8:      108
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: garry23 on December 31, 2016, 11:17:05 AM
@Danne (@A1ex)

Thanks for sharing the test version for the 5D3-113.

I've just tried it out and managed a single lossless DNG, but I had,'t appreciated that this this meant much less pixels! That is my lossless DNGs are 1932 x 1290.

Also, I tried using my FRSP LE script, but the Lua call (camera.shoot(64, false) or camera.shoot(false)) triggers the mechanical shutter.

Cheers

Garry
Title: Re: ProcessTwoInTwoOutLosslessPath: UPDATE
Post by: garry23 on December 31, 2016, 12:15:38 PM
OK, I think I worked out 'my mistakes'.

I don't think the lossless test build included the Lua fix.

Plus I hadn't switched to Full Res !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I'll wait a little :-)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: calypsob on December 31, 2016, 11:47:14 PM
 Will this feature only be available on 5D Mark III?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on January 02, 2017, 03:28:18 AM
@calypsob Yeah it will be available for the 5D III when they figure it all out  8)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: budafilms on January 02, 2017, 03:58:25 AM
"My kingdom for A1ex's brain"
;)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: gamedesign on January 02, 2017, 05:18:42 PM
Will this feature bring any benefits to RAW video on the 7D (mark 1)?
I currently use MLV all the time, its brilliant.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Licaon_Kter on January 03, 2017, 09:25:06 PM
Forum overload in 3...2...1... https://www.dpreview.com/news/2451079037/magic-lantern-proof-of-concept-shows-in-camera-dng-shooting

That being said, looks like I need to raise the issue or DNG lossless compression support in Darktable again :(
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 03, 2017, 09:45:22 PM
At least there might be a chance to comment on Petapixel not getting the point and put it right.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on January 03, 2017, 10:25:54 PM
noone realized that it is mainly a saving format meant for silent pictures
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Licaon_Kter on January 03, 2017, 11:05:56 PM
Quote from: g3gg0 on January 03, 2017, 10:25:54 PM
noone realized that it is mainly a saving format meant for silent pictures
Details... details... that would need someone to actually read the thread, not only to grab the DNGs...  research in journalism, pffft
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 03, 2017, 11:15:11 PM
Hey, no more ranting or "the company" may get sued ...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on January 03, 2017, 11:32:22 PM
this wasn't meant as negative comment.

well, it might be interesting for raw video too, but tests till now didn't show any way to make the JPCORE work in LV synchronized to frame display.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 04, 2017, 12:12:03 AM
I haven't archived PetaPixel's post. There was some makeover, I think. Some corrections + introducing some weird concept (I take the liberty to say so).
QuoteIf and when this feature arrives, you won't have to worry about quality loss from converting your Canon RAW files into DNGs ...

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Licaon_Kter on January 04, 2017, 12:57:31 AM
Quote from: Walter Schulz on January 04, 2017, 12:12:03 AM
I haven't archived PetaPixel's post. There was some makeover, I think. Some corrections + introducing some weird concept (I take the liberty to say so).
And a lot of comments discuss that (quality change on conversion), not sure why would anyone would think that.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Audionut on January 04, 2017, 07:23:05 AM
Quote from: Licaon_Kter on January 04, 2017, 12:57:31 AM
not sure why would anyone would think that.

The world is flat and there's a magical fairy that sends you to hell if you misbehave.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 04, 2017, 08:42:12 AM
Quote from: Licaon_Kter on January 04, 2017, 12:57:31 AM
And a lot of comments discuss that

Just found out our proxy blocked PetaPixel's comments. Thanks for the info!
Got it wrong about makeover, too.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: vorob on January 10, 2017, 09:09:08 AM
Hi, can anyone explain what's going on here? Why do we need this? CR2 files are compressed? Lost smth? Not 14bit? Thank you and sorry for my lack of knowledge.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 10, 2017, 09:16:40 AM
CR2 out of the cam (with and without ML): Lossless compressed
DNG/MLV/RAW out of ML (produced by some ML features, not all of them): Uncompressed.

This thread is about enabling ML to produce lossless compressed output, too.
And compression means less bandwidth required writing to card.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: vorob on January 10, 2017, 09:20:19 AM
QuoteCR2 out of the cam (with and without ML): Lossless compressed
DNG/MLV/RAW out of ML (produced by some ML features, not all of them): Uncompressed.

This thread is about enabling ML to produce lossless compressed output, too.
And compression means less bandwidth required writing to card.

QuoteFiles are 15-20% smaller and image previews load faster in lightroom. The disadvantage is some metadata is discarded and dng files don't play well with some non-adobe software. There's also an argument that dng files are more future proof, but since old dng files are different to new dng files, I don't buy that argument; I'm sure I'll still be able to access/convert cr2 files in 20 years if it becomes and issue.

Basically if you never intend to move outside of Adobe's ecosystem then it's a nice choice for saving some hard drive space. With modern hardware the speed advantage will be minimal for most people, and there's the risk if you move to capture one or dxo (say) in the future, you'll have issues with camera profiles and lens correction not working accurately (despite the format being semi-supported).

Well, i still don't get it... :(
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on January 10, 2017, 10:12:20 AM
Ignore the second quote, as it does not apply to this thread at all.

What's researched here does not provide any reason to replace the CR2 format for regular still pictures.

- Better compression? In our DNGs, compression is not any better or worse than CR2; it is identical.
- Saving a conversion step (if you convert to DNG anyway)? Not a good reason for justifying the development effort IMO.
- Additional EXIF info? Maybe, but XMP (http://www.magiclantern.fm/forum/index.php?topic=18083) works (https://www.magiclantern.fm/forum/index.php?topic=10496.0) and is minimally invasive to Canon code.
- "Quality loss from converting your Canon RAW files into DNGs"? There's none. Once uncompressed, raw data is 100% identical (check with e.g. dcraw -4 -E after converting a CR2 with Adobe DNG Converter).

It's all about the files saved by ML features (silent pictures, possibly raw video). Maybe also opening the door to MJPEG.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 10, 2017, 10:37:29 AM
@vorob: Your second quote comes from Canonrumours.
Lesson to be learnt: Don't trust anything written about ML outside ML forum. They tend to mix up things badly. And almost never you will find a corrections section.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on January 10, 2017, 01:03:46 PM
well, one could ask before posting news.
and the best was to do that is... asking on the forum, either as thread or if necessary via PM to us core devs.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 10, 2017, 01:31:18 PM
May I ask if such an incident ever happened? I'm willing to bet it hasn't in the past and won't happen in the future.
If I'm loosing the bet I will also win because it will tell me about a reliable news site.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Maximo on January 27, 2017, 02:54:38 PM
Hi guys,
How likely will we have 4K lossless video DNG files on the 5D mark IV?  Many thanks.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on January 27, 2017, 02:58:59 PM
58.12699378 %
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Licaon_Kter on January 27, 2017, 03:24:36 PM
@Maximo: Each new 5DIV camera that @a1ex receives for free in order to develop ML for, will increase chances by 5%!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Wayne H on January 27, 2017, 04:41:29 PM
Really excited by this find and development and where it could potentially lead.

Motion JPeg, or dng at 50-60% compressed of the original file size and higher resolutions would be just unbelievable... maybe a april 1st announcement :D, fingers crossed.

Massive thank you to the whole magic lantern community for this amazing software and a special thanks to the core developers.

Any news on how things are progressing?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 20, 2017, 05:44:26 PM
After fiddling a bit with these routines, I have both good and bad news:

- good: 700D has routines very similar to 5D3 (named TwoInTwoOutJpegPath; not tested)
- good: 5D3 has two sets of routines for lossless compression (TwoInTwoOutLosslessPath and TwoInTwoOutJpegPath); both are working (second one requires some fiddling)
- bad: the routines are a bit different on 60D (not very LiveView-friendly)
- ugly: I wasn't able to decode the compressed images from 60D (just corrupted images), but didn't try too hard

- good: output bit depth (0xC0E00084) can be overriden from 14 to 15 bits
- info: a file with 15 bits is just 3 bytes (!) higher than a 14-bits one (headers show 15 bits and contents is totally different for same input image)
- bad: other bit depths (8, 10, 12) did not work at all
- bad: bit depth overriding did not work on 5D3 at all (maybe I'm missing something)

- good: the routines can accept raw image streams of arbitrary bit depths (10/12/14/16, configured with 0xC0F371FC)
- bad: the above doesn't appear to improve compression ratio at all
- ugly: compressing a zeroed out image gives a ratio of... 35.7%! (very large encoder overhead?)

What happens:
- output levels do not depend on input bit depth
- instead, 10-bit streams are padded with zero LSBs; the end result is the same as zeroing the lower bits.

From the 14/15-bit experiment, dividing the input data by some constant value (e.g. dividing by 16 for 14->10 bit conversion) probably has the same impact on file size as if we were able to reconfigure the encoder for a lower bit depth (which we can't). That's my current understanding, but I have no hard proof on that (as I don't even know how the encoder works).

Some numbers:
(http://a1ex.magiclantern.fm/bleeding-edge/jpcore/lossless.png)

Note: the identical checksum confirms that compressing a 10-bit raw image gives the same result as converting a 14-bit image with 4 LSBs zeroed out (therefore, the unpacker module probably normalizes the values internally by adding LSBs).

Bottom line: pushing the lossless compression ratio below 50% (relative to 14-bit uncompressed raw size) is hard. Of course, it depends on noise levels and scene contents, unlike uncompressed raw, but the variations appear to be small.




(M)JPEG is, indeed, done with the same routines. The main difference is the type of data accepted as input (8-bit debayered, probably hardcoded to YUV422).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on March 20, 2017, 10:01:00 PM
really good work :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: DeafEyeJedi on March 20, 2017, 10:17:31 PM
This is all mostly good news to me and excellent work as always @a1ex!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 01, 2017, 02:06:31 PM
Quote from: g3gg0 on December 25, 2016, 10:28:49 PM
-> MAYBE on april 1st ;)

Good prediction. It's working - experimental build posted.

(http://a1ex.magiclantern.fm/2017Apr01/lossless.png)

To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).

Baldand's LJ92 decoder from mlv_rec_lj92 branch almost decodes the payload, but the image data is rendered incorrectly for some reason. Didn't try to debug it yet, as I'm far from understanding how lossless JPEG compression works.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on April 01, 2017, 02:14:44 PM
finally! in-camera compressed MLV files. yay!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Licaon_Kter on April 01, 2017, 02:37:50 PM
WoW  :o
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on April 01, 2017, 02:43:56 PM
great job
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 01, 2017, 04:18:31 PM
Amazing! Had my fingers quietly crossed for this.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: janvkem on April 01, 2017, 04:38:58 PM
Great to see this getting along! Just tested and the DNG files are ~2.3 MB each whereas they normally are ~3.5 MB (1920x1080), so ~65% compression! It seems that there is still some bug inside however: When I record it crashes after a couple of seconds. There is an MLV file created on the card. Inspecting it on my pc showed roughly a double size of the MLV file compared to the extracted DNG files (819 MB for the MLV file and 387 MB for the extracted DNG files).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 01, 2017, 05:02:05 PM
Can you share your settings directory, so I can try to reproduce?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 01, 2017, 05:04:10 PM
Ran camera for 11 minutes at 3168 x 1320 24p. First frame of the recording looks good to me:

https://www.dropbox.com/s/8trg0exly76nx9a/m01-1531_000000.dng?dl=0

However mlv dump gave up after 101 frames with this message:

Unknown Block: )Nh,, skipping
[ERROR] Invalid block size at position 0x1981e600
Processed 101 video frames

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 01, 2017, 05:12:24 PM
Output of "mlv_dump file.mlv -v" ?

The longest clip I've tried was a 3GB 1920x1080 - 1867 frames.

edit: 3168 x 1320 24p (24.006 in x5 zoom), with grayscale preview, ran for 2 minutes, 3 chunks (MLV, M00, M01), 2908 frames. All seems OK. I've used the build from the experimental page for 1.2.3 (not a local compilation).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: MitchLally on April 01, 2017, 05:18:22 PM
Wow. This is incredible.

Will there be a custom MLV_DUMP for mac? :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: MCAN on April 01, 2017, 05:25:37 PM
Quote from: MitchLally on April 01, 2017, 05:18:22 PM
Will there be a custom MLV_DUMP for mac? :)

+1 Please!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: janvkem on April 01, 2017, 05:28:25 PM
Quote from: a1ex on April 01, 2017, 05:02:05 PM
Can you share your settings directory, so I can try to reproduce?

Sorry, my bad. Checked again with a completely fresh card (other test was just with deleting the ML folder and autoexec.bin file) and issue appears to be gone. I had 2 additional files on the card but cannot recall the names (removed them before I saw your post unfortunately)... My apologies for the incorrect information.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 01, 2017, 05:39:11 PM
Quote from: a1ex on April 01, 2017, 05:12:24 PM
Output of "mlv_dump file.mlv -v" ?

Is this what you need?

https://www.dropbox.com/s/d5op627w2seulc2/output.txt?dl=0

This is 1.1.3
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 01, 2017, 05:43:31 PM
Yes; unfortunately, it doesn't give me a clue about what happened...

Would be best if you can find a way to reproduce.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 01, 2017, 06:08:56 PM
I should note that on my first recording I had modules mlv_rec, _play and _lite loaded: I got the grayscale preview for a very brief time and then the LCD switched to some kind of report log that I haven't seen before.

2nd recording I switched off all modules except mlv_lite: I got greyscale preview continuously but recording stopped automagically after around 4 minutes.

3rd recording lowered resolution to 2880 x 1320 and very quickly got the onscreen log again:

https://s17.postimg.org/6fokginq7/20170401_170314.jpg

Here are my original settings (first recording) if it helps:

https://www.dropbox.com/s/d07hnipovcsrpf4/SETTINGS.zip?dl=0

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: marekk on April 01, 2017, 07:14:16 PM
Quote from: MCAN on April 01, 2017, 05:25:37 PM
+1 Please!

mlv_dump for mac (compressed_raw branch)

https://drive.google.com/file/d/0B-7nnVbwzGVkUmpHLW5kbEU3OWc/view?usp=sharing (https://drive.google.com/file/d/0B-7nnVbwzGVkUmpHLW5kbEU3OWc/view?usp=sharing)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: eNnvi on April 01, 2017, 08:25:29 PM
So how can we help to Port this on other cameras?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 01, 2017, 08:36:31 PM
Quote from: hyalinejim on April 01, 2017, 05:04:10 PM
[ERROR] Invalid block size at position 0x1981e600

Diagnosed the above issue (it was related to variable frame sizes) and hopefully fixed it. Tested by filling the card a couple of times and checking the validity of the recorded clip with mlv_play (it doesn't play the video yet, but from the error message it's possible to tell whether the file structure is valid or not).

New experimental build available.

Will explain how it works later.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 01, 2017, 09:11:58 PM
Nice one!

EDIT: Checked out new build and I'm happily recording at 3072 x 1286. Around 37 minutes into that take now and temperature is showing as 55 degrees. Previously I'd never really seen it go above around 45.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: lostfeliz on April 01, 2017, 10:17:02 PM
37 minutes at 3072 x 1286?! Wow. How many GB was the whole clip?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ilia3101 on April 01, 2017, 10:40:41 PM
hyalinejim is that losslessly compressed video you were able to record for 37 minutes?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 01, 2017, 11:49:00 PM
Quote from: lostfeliz on April 01, 2017, 10:17:02 PM
37 minutes at 3072 x 1286?! Wow. How many GB was the whole clip?

Just checked the card: 208GB.

Quote from: Ilia3101 on April 01, 2017, 10:40:41 PM
hyalinejim is that losslessly compressed video you were able to record for 37 minutes?

Yes, indeed. By comparison, at the same resolution on my card I got around seven seconds in 10 bit. To be clear, I stopped the lossless 14bit recording at 37 minutes. It's only limited by the size of your recording media.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on April 01, 2017, 11:53:24 PM
Lossless 14bit files 3k :o
I need my camera already.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: beauchampy on April 02, 2017, 03:06:07 AM
Hey a1ex! Really nice job! I'm still wrapping my head around all these incredible developments.

Questions about the lossless compression - it says compressed with Canon lossless JPEG, but this is still raw, right? How does the information differ to the normal 14-bit raw files?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Markus on April 02, 2017, 06:44:10 AM
Still can't believe all this incredible progress isn't just a big april fools joke.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 02, 2017, 08:12:08 AM
Quote from: beauchampy on April 02, 2017, 03:06:07 AM
Questions about the lossless compression - it says compressed with Canon lossless JPEG, but this is still raw, right? How does the information differ to the normal 14-bit raw files?

Lossless JPEG is what you have in regular CR2 files and compressed DNGs. It's not the 8-bit JPEG you are used with.

I was also fooled by this for a long time - the routines used to compress the CR2, on most camera models, had JPEG in their name, so I didn't look into them, thinking they were for regular 8-bit JPEG compression...

https://en.wikipedia.org/wiki/Lossless_JPEG
http://lclevy.free.fr/cr2/#lossless
https://bitbucket.org/baldand/mlrawviewer/src/e7abaaf4cf9be66f46e0c8844297be0e7d88c288/LJ92.py
https://bitbucket.org/hudson/magic-lantern/commits/83ced42a1009db90ff33ceae5e4e198fe5943b47
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: nikfreak on April 02, 2017, 09:25:22 AM
Have to ask it:only 14/15bit (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: bouncyball on April 02, 2017, 10:04:02 AM
@a1ex: Happiest April 1st ever! :D

Quote from: hyalinejim on April 01, 2017, 11:49:00 PM
Just checked the card: 208GB.

Yes, indeed. By comparison, at the same resolution on my card I got around seven seconds in 10 bit. To be clear, I stopped the lossless 14bit recording at 37 minutes. It's only limited by the size of your recording media.
@hyalinejim: Hmm... it's about 93mb/sec. Which CF card did you use?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on April 02, 2017, 11:14:13 AM
YEs, Canon standard CR2 compression for RAW video  :D

How hard is this to port to other cams, is it just a few lines of routines to be altered or is it a pain in the :o to port  :P ?

I hear compression ratios of 58%, is that true ? Because in that case it would be even better then 10 bit (about 70% smaller compared to 14 bit)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: hyalinejim on April 02, 2017, 11:24:09 AM
Quote from: bouncyball on April 02, 2017, 10:04:02 AM
@hyalinejim: Hmm... it's about 93mb/sec. Which CF card did you use?

OK, so the long file I recorded is 218,855,342 KB and it's 55,874 frames at 3072 x 1286 resolution.

Card is Komputerbay 256GB 1066x
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on April 02, 2017, 11:27:29 AM
That's about 57% compression ratio  :o
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: eNnvi on April 02, 2017, 01:21:08 PM
@levas no, as Alex tested a few weeks ago the compression would be the same as a 10bit stream would give the same results of a 14 bit as it's used as a 10bit with leading zeros. So it's not worth to use a 10bit stream with this method.

Just I'd like to know how I can help to Port this on other cameras and if this would get rid of focus pixels on many cams like 700D. Would be nice to know if it's the same routine to compress and delete focus pixels :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on April 02, 2017, 01:34:37 PM
QuoteOK, so the long file I recorded is 218,855,342 KB and it's 55,874 frames at 3072 x 1286 resolution.

The above gives about 4Mb frames which should be about 7Mb uncompressed, that's 57% isn't it?

Edit: I know that it doesn't work with 10 bit stream, which would be even more great.
But to me it looks like 14 bit compressed is more efficient  then 10 bit.
57% vs 71%
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: bouncyball on April 02, 2017, 02:36:00 PM
Quote from: eNnvi on April 02, 2017, 01:21:08 PM
Just I'd like to know how I can help to Port this on other cameras and if this would get rid of focus pixels on many cams like 700D. Would be nice to know if it's the same routine to compress and delete focus pixels :D
The focus pixel issue has no relation to canon HW compression at all.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: eyeland on April 04, 2017, 01:59:30 AM
This is amazing :D
Has anyone played around with cranking frame-rates up instead of resolution?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: COMMANDES on April 05, 2017, 01:14:49 PM
How soon will it be possible to test a compressed RAW at 650D?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: littlebobbytables on April 05, 2017, 02:22:43 PM
I too would be very interested in a build for my camera. I foolishly tried to build for the 50D in the compressed raw branch but the mlvlite module crashes the camera. I guess some work needs to be done for other (Digic IV) camera's?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: domasa on April 08, 2017, 04:04:11 PM
Quote...processing options such as bad pixel or vertical stripe fixes are not working yet
Would it be possible delete some rows and columns during processing (or color black for better compression)?

It will be great for record some resolution unsupported by ML.
E.g.: use supported resolution 5796x3870 25p for record UHD (3840x2160) 25p.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 12, 2017, 06:59:50 AM
Some people are playing around with adtg_gui in order to get other cameras working with the crop_rec on steroids. Of course another part of the puzzle is getting lossless compression working so I thought I'd try my luck with porting ProcessTwoInTwoOutLosslessPath to the the 700D and EOSM. First lesson, it is called ProcessTwoInTwoOutJpegath on the 700D and EOSM.

I started out with the crop_rec_4k branch working on the silent module. First problem, Lossless DNG is broken on that branch. Switch over to the compressed_raw branch and it looks like the camera specific part is limited to this:

    if (is_camera("700D", "1.1.4"))
    {
        /* ProcessTwoInTwoOutJpegath, 700D 1.1.4 */
        TTL_SetArgs     = (void *) 0xFF35F510;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF423650;  /* called right after ProcessTwoInTwoOutJpegath(R) Start; */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF423B88;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF36B2D8;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF423960;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF424C84;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF424CBC;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

    if (is_camera("EOSM", "2.0.2"))
    {
        /* ProcessTwoInTwoOutJpegath, EOSM 2.0.2 */
        TTL_SetArgs     = (void *) 0xFF361498;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF427CBC;  /* called right after ProcessTwoInTwoOutJpegath(R) Start; */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF4281F4;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF36D124;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF427FCC;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF4292F0;  /* called right after StopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF429328;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


Don't get too excited, this isn't working yet. I'm posting so others that might be working on it can compare notes and point out errors. I'm pretty sure I nailed TTL_SetArgs and TTL_Prepare the the others are rather iffy. They "look" good but neither camera can create Lossless DNG files.

Looking through lossless.c there seem to be some hardcoded values that are fine for the 5D3 but might need to be adjusted for other cameras?

By the way, for those of you who haven't combed through a disassembly there are some pretty interesting things in there. For example:
and the very profound:

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Teamsleepkid on April 12, 2017, 07:54:47 AM
i thought even the 5dii wasn't making lossless dng files...isnt it still coming out as an mlv file? and looks like you're definitely making progress:)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 12, 2017, 08:21:27 AM
The 5D2 has ProcessTwoInTwoOutJpegPath as do most other cameras. Note that the 700D and EOSM have ProcessTwoInTwoOutJpegath (typo?) while the 5D3 has both ProcessTwoInTwoOutJpegPath and ProcessTwoInTwoOutLosslessPath.

So is it possible to do lossless compression on all cameras? I don't know but probably. The 700D and EOSM compresses the CR2 files.

Quote from: Teamsleepkid on April 12, 2017, 07:54:47 AM
isnt it still coming out as an mlv file?

As far as I know lossless compression was first tried out on DNG files so that's where I'm starting.

(https://c1.staticflickr.com/4/3949/33831470232_96ba400695.jpg)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 12, 2017, 09:54:36 AM
My older attempt at 700D in QEMU (I've used it to understand how 5D3 TTJ works):


    if (is_camera("700D", "1.1.4"))
    {
        /* uses TwoInTwoOutJpegPath, but functions are mostly compatible */
        TTL_SetArgs     = (void *) 0xFF35F510;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF424BA4;      /* called right after StartTwoInTwoOutJpegPath Raw(R) Start(%d) */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF423B88;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_700D;  /* this function is inline on 700D */
        TTL_Start       = (void *) 0xFF424C4C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF423DD4;      /* called after sdfStopYuvToJpegPath(%d); unsure, need a dm-spy log */
        TTL_Finish      = (void *) 0xFF424CBC;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */

        TTL_ResLock     = (void *) MEM(0x25A60);    /* this should work outside LiveView (e.g. full-res silent pics) */
    }

static void set_flags_700D()
{
    EngDrvOut(0xC0F37010, (shamem_read(0xC0F37010) & 0xFFE3FFFF) | 0xC0000);
    EngDrvOut(0xC0F3704C, (shamem_read(0xC0F3704C) & 0xFFFFFFC0) | 0x20);
    EngDrvOut(0xC0F37074, (shamem_read(0xC0F37074) & 0xFFC0FFFF) | 0x180000);
    EngDrvOut(0xC0F37078, (shamem_read(0xC0F37078) & 0xFFC0FFFF) | 0x180000);
}


As dfort guessed, all of this is found by following Canon's CR2 compression routines (codenamed Mem1ToRaw). Unfortunately, there are differences between models, so it's not always just a pattern matching.

You've noticed ResLock requires some tweaking. What's up with it?

ResLock is a way to lock certain image processing modules (or "resources") that we are going to use (in order to prevent two tasks from using that image processing module at the same time). Each image processing "resource" is identified by a 32-bit resource ID (aka resID). From those, the most significant 16 bits are the class ID (category of resource), and the least significant bits are the resource ID within that class. Here you can find the known resource identifiers (http://www.magiclantern.fm/forum/index.php?topic=6740.0).

Each ResLock ID has an (internal) semaphore associated. Some resources (such as EDMAC channels / connections) may work without locking, as long as only one task uses them (otherwise you get corrupted data), but the compression ones refuse to work that way. Apparently, LockEngineResources also enables the clocks required by certain resources (it probably powers up the hardware required by those operations).

The first step is to try Canon's ResLock configuration outside LiveView (where all image processing resources are free, so their default reslock setup is likely to work).

What about LiveView?

In LiveView, many EDMAC channels are already assigned and locked (by Canon code); they will probably conflict with the choice used for CR2 compression (which is done outside LiveView). That means, we'll have to re-assign the EDMAC channels (and a good choice is to reuse the channels from edmac-memcpy.c, since they won't be used at the same time anyway).

On 5D3, I only had to redirect the write channel, but for a portable solution, the read channel might have to be redirected as well.

On this camera, the list of resources is found in TTL_SetArgs, for PictureSize == 0:

0x10000,             // RD1: first read channel (EDMAC channel 8) - didn't have to change it
0xD,                 // WR1: write channel #13 (look it up in edmac_chanlist - it's channel 22; changed to channel 17 for LiveView)
2,                   // WR2: write channel #2 (unused, deleted; didn't check what data it outputs)
0x30001,             // connection 1; don't remember why I've deleted it (possibly by mistake)
0x2002d ... 0x260003 // unknown, copied verbatim


On 700D, the list is at 0x334E4 (e.g. in a RAM dump).

For reference, here's the TTJ version for 5D3 1.2.3 (works only outside LiveView; code based on 700D):

    if (is_camera("5D3", "1.2.3"))
    {
        /* ProcessTwoInTwoOutJpegPath, 5D3 1.2.3 */
        TTL_SetArgs     = (void *) 0xFF327DE8;  /* using TTL version */
        TTL_Prepare     = (void *) 0xFF3D00A0;  /* called between sdfExecuteYuvToJpegPath and StartYuvToJpegPath */
        TTL_RegisterCBR = (void *) 0xFF3CF084;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_700D;   /* using 700D version */
        TTL_Start       = (void *) 0xFF3D0148;  /* called after StartYuvToJpegPath; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF3D0180;  /* called right after sssStopMem1ToJpegPath */
        TTL_Finish      = (void *) 0xFF3D01B8;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

    uint32_t resources[] = {
        0x10000,                        /* read channel 0x8 */
        edmac_channel_to_index(0x11),   /* write channel 0x11 */
        0x30002,                        /* read conn 2 */
        0x20005,                        /* write conn 5 */
        0x50034,
        0x5002D,
        0x50010,
        0x90001,
        0x90000,
        0xA0000,
        0x160000,
        0x260000,
        0x260001,
        0x260002,
        0x260003,
    };

    /* copy this one from 0x33528 until 0x33918 (700D 1.1.4) */
    TTL_Args.engio_cmd_1_prepare = prepare_700D;


Overriding resolution is a different story; let's first get it working in full-res mode (Canon hard-codes resolution parameters for a full-res raw buffer). You may have to comment out the EngDrvOut calls for overriding the resolution, and maybe also the EDMAC setup for RD1.

OK, what's up with those funny names like RD1, WR1 etc?

From the name: TwoInTwoOut means this routine accepts two inputs (RD1, RD2) and two outputs (WR1, WR2). All of them are optional (see TTL_Start and TTL_Stop - these channels are only configured if the corresponding address or memsuite from TTL_Args is nonzero).

For lossless compression, we are interested in RD1 and WR1. What the others do, I don't know.

The default configuration (CR2 with quality set to RAW) uses RD1, WR1 and WR2. On 5D3, they use EDMAC channels 8, 22 and 2 (look at this picture (http://www.magiclantern.fm/forum/index.php?topic=18315.msg176378#msg176378)).

RD1 reads from a contiguous buffer (that's the 14-bit raw buffer in the default configuration). In the picture, you can see it appears to scan the image twice, and its size looks a bit unusual. Here's how to decipher it and why we are overriding it (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176809;topicseen#msg176809).

WR1 uses a memSuite (unlike the other channels) - this is a data structure in Canon firmware that can contain one or more contiguous memChunk's (in other words, the WR1 channel can write to fragmented memory, possibly allocated with *_malloc_suite routines from exmem.c). The output size is not known in advance - it is set by TTL_Finish (actually JpCoreCompleteCBR). If the size is larger, it will not overflow the memSuite, but apparently there are weird alignment constraints on its size (https://bitbucket.org/hudson/magic-lantern/commits/bc99f7502deb196b96f565b4a7556758258d6c6d?at=crop_rec_4k).

WR2 outputs something of size 1776*2 x 394 bytes, but didn't try to decode it (probably some sort of preview). Unused in our routine.

Here's a debug log from 5D3, showing the CR2 compression (http://a1ex.magiclantern.fm/bleeding-edge/capture/5D3-startup-eeko-rawdev-takepic-edmac.LOG) (among many others). Some relevant lines:

ShootSsDev:ff162e10:a7:03: sssExecuteMem1ToRawPath(79319)
ShootSsDev:ff163230:a7:03: ProcessTwoInTwoOutLosslessPath(R) Start(79319)
ShootSsDev:ff3d3f34:16:03: [TTL][167,79319,0] RAW(5920,3950,0,14)
ShootSsDev:000afe98:00:00: *** StartEDmac(0x16, 0x0), from ff576498
ShootSsDev:000afe98:00:00: *** StartEDmac(0x2, 0x0), from ff3d3af0
ShootSsDev:000afe98:00:00: *** StartEDmac(0x8, 0x2), from ff3d3b50
**INT-A3h*:ff3d3d14:16:03: [TTL][167,79319,0] JpCoreCompleteCBR( 0x146a56b )


To be continued.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 13, 2017, 07:39:44 AM
More coming?

Good to verify some of the 700D addresses. Still can't save lossless full res DNG's but the warning messages and crashes are gone.

Looking through RAM4 and doing a lot of head scratching.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 13, 2017, 08:10:48 AM
Some logs would be helpful (set verbose=2, or merge dm-spy-experiments and call debug_intercept before/after the compression routine).

Also, some dm-spy logs covering a still image capture (with quality set to RAW).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: goldenchild9to5 on April 13, 2017, 02:53:18 PM
@a1ex WOW.. very promising.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 14, 2017, 12:51:42 PM
Quote from: dfort on April 13, 2017, 07:39:44 AM
More coming?

Quote from: a1ex on April 13, 2017, 08:10:48 AM
Some logs would be helpful (set verbose=2, or merge dm-spy-experiments and call debug_intercept before/after the compression routine).

Episode #2 is up (http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168) - on the dm-spy-experiments thread.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 16, 2017, 08:32:41 AM
Quote from: a1ex on April 13, 2017, 08:10:48 AM
Also, some dm-spy logs covering a still image capture (with quality set to RAW).

I'm having problems saving logs when attempting to shoot lossless compressed full resolution silent pictures on the 700D. However, I did record a dm-spy log shooting a single CR2 and it looks like there are some hints at how lossless compression works on this camera:

Full log file:
https://www.dropbox.com/sh/d7jyhbfk86vbxcu/AACYm04k7cTZvy_j3HzuFVaoa?dl=0

Some interesting snippets:
EEF51> ShootSsDev:ff168580:a7:03: sssExecuteMem1ToJpegPath(11296)
EEF96> ShootSsDev:ff4242a0:16:03: [TTJ][0,0,0] JPEG(5184,3456,0,0)
...
EF442> ShootSsDev:ff42463c:16:03: [TTJ] START WR1:0xa000064 WR2:0xd3e5f8
EF473> ShootSsDev:ff424668:16:03: [TTJ] START RD1:0x608e2a0 RD2:0x1082704
...
15488> ShootSsDev:ff42463c:16:03: [TTJ] START WR1:0xa06c07c WR2:0x23f8070
154BA> ShootSsDev:ff424668:16:03: [TTJ] START RD1:0x6004894 RD2:0x7f21f9c
...
15699> ShootSsDev:ff168ac8:a7:03: StartTwoInTwoOutJpegPath Raw(R) End(11296)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 22, 2017, 04:15:05 AM
@lostfeliz

You'll probably get a better response if you post that question on the crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView (http://www.magiclantern.fm/forum/index.php?topic=19300) topic. (Just don't double post  :) )

Lots of work went into getting this working in MLV Lite (https://bitbucket.org/hudson/magic-lantern/branches/compare/crop_rec_4k%0Dunified#chg-modules/mlv_lite/mlv_lite.c) though it looks like much of it can be applied to regular mlv_rec which is required for mlv_snd.

[EDIT: Original question moved to: http://www.magiclantern.fm/forum/index.php?topic=19300.msg183504#msg183504 ]
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 23, 2017, 05:59:46 PM
Quote from: dfort on April 16, 2017, 08:32:41 AM
I did record a dm-spy log shooting a single CR2

Can you take the same log, with interrupts and engio messages enabled? They are both disabled by default (first in dm-spy-extra.c, second in dm-spy.c).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 23, 2017, 08:01:26 PM
Found where to enable interrupts logging:

dm-spy-extra.c
/* this needs pre_isr_hook/post_isr_hook stubs */
#define LOG_INTERRUPTS


but don't understand how to enable engio messages. Seems to me that it is triggered if a certain condition is met:

dm-spy.c
static void my_DebugMsg(int class, int level, char* fmt, ...)
{
    if (!buf) return;
       
    if (class == 21) // engio, lots of messages
        return;

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: nikfreak on April 23, 2017, 08:20:01 PM
http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168

follow the guide "Want more details" and remove

if (class == 21) // engio, lots of messages
        return;
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 24, 2017, 12:37:19 AM
Thanks @nikfreak -- here you go @alex

https://www.dropbox.com/sh/d7jyhbfk86vbxcu/AACYm04k7cTZvy_j3HzuFVaoa?dl=0

700D_shoot_CR2_2017-04-23

Lots of information logged for a single shutter click.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 24, 2017, 12:49:53 AM
Looks like the 700D has the same problem as 5D3 - engio_write and others are silent. Can you include them as well? (they are in the same guide)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on April 24, 2017, 02:19:40 AM
Ahh--I see the error in my thinking. I thought that /* on 5D3, Canon's engio_write is silent */ only applied to the 5D3. Should have just run the patch instead typing in the changes.

Replaced the log file from earlier today with an even longer log file that has a bunch of [ENGIO] in it.

Just to make sure we're on the same page, this is a log for just a single CR2 shot in order to see where the lossless compression is taking place. I still can't save a lossless DNG on the 700D with the settings I made on this commit (https://bitbucket.org/daniel_fort/magic-lantern/commits/ba32abcfcb7e3e5fa60502a1c50190826791be10?at=compressed_raw-700D-EOSM). Never mind the EOSM settings, I'll get back to that once the 700D is working.

I'll be away from my development setup for a few days--hope I don't get ML separation anxiety!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 24, 2017, 10:28:06 AM
At the time of writing, it only applied to 5D3, from the models I've looked at.

Our logging routines are not printing [ENGIO], so the reason for larger log might have been a few more seconds allocated for the test routine. The new log doesn't appear to have any new information though.

EngDrvOut and engio_write can be taken from stubs; the others are easy to identify from strings (as they only log a debug message in case of error). Most DIGIC 4 models also log the register and the data written to it, when the call is successful.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: festr on April 24, 2017, 02:53:37 PM
Hello, is there already tool which can decompress and fix vertical stripes?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on April 24, 2017, 03:23:10 PM
There is a closed-source tool that can decode them, to my knowledge.

The decompressor is based on Baldand's LJ92 codec (from MlRawViewer), but it needs some changes. Some of these changes were shared here (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182502#msg182502), so I assume one should be able to integrate them in other converters.

So, all that's required is some manpower to do the integration. No reverse engineering is required.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Greg on April 24, 2017, 06:14:36 PM
500D :
634AA> FrontShtDe:ff085618:96:05: sdfExecuteMem1ToRawPath(84538)
6352E> FrontShtDe:ff0859a8:96:05: sdfExecuteMem1ToRawPath(84538)©(SemOK)
6356D> FrontShtDe:ff0859e0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(84538)
645F9> FrontShtDe:ff252004:16:03: [TTJ][150,84538,0] RAW(4832,3204,0,14)
64634> FrontShtDe:ff191590:15:02: [JPCORE] SetEncodeLosslessType 1
64669> FrontShtDe:ff1929d0:15:02: [JPCORE] SetEncodeLosslessParam 1208,3204              <- note: 4 slices
84993> **INT-64h*:ff251de0:16:03: [TTJ][150,84538,0] JpCoreCompleteCBR( 0xd01f99 )
8647D> FrontShtDe:ff085a10:96:05: ProcessTwoInTwoOutJpegPath(R) End(84538)


ISO 100
dark frame - 50%
tree leaves - 70-75%

UnLockEngineResources :
851E4> FrontShtDe:00092990:00:00: *** UnLockEngineResources(869d98) x31 from ff2519bc:
8521F> FrontShtDe:00092a08:00:00:      1)    10002 (read channel 0xa)
85244> FrontShtDe:00092a08:00:00:      2)        3 (write channel 0x3)
85264> FrontShtDe:00092a08:00:00:      3)        4 (write channel 0x4)
85284> FrontShtDe:00092a08:00:00:      4)    30000 (read connection 0x0)
852A8> FrontShtDe:00092a08:00:00:      5)    20005 (write connection 0x5)
852CC> FrontShtDe:00092a08:00:00:      6)    20016 (write connection 0x16)
852EA> FrontShtDe:00092a08:00:00:      7)    50003 (?)
85307> FrontShtDe:00092a08:00:00:      8)    5000d (?)
85324> FrontShtDe:00092a08:00:00:      9)    5000f (?)
85341> FrontShtDe:00092a08:00:00:     10)    5001a (?)
8535E> FrontShtDe:00092a08:00:00:     11)    80000 (?)
8537B> FrontShtDe:00092a08:00:00:     12)    90000 (?)
85399> FrontShtDe:00092a08:00:00:     13)    a0000 (?)
853B6> FrontShtDe:00092a08:00:00:     14)   160000 (?)
853D5> FrontShtDe:00092a08:00:00:     15)   130003 (?)
853F3> FrontShtDe:00092a08:00:00:     16)   130004 (?)
85412> FrontShtDe:00092a08:00:00:     17)   130005 (?)
85430> FrontShtDe:00092a08:00:00:     18)   130006 (?)
8544F> FrontShtDe:00092a08:00:00:     19)   130007 (?)
8546D> FrontShtDe:00092a08:00:00:     20)   130008 (?)
8548C> FrontShtDe:00092a08:00:00:     21)   13000c (?)
854AB> FrontShtDe:00092a08:00:00:     22)   130010 (?)
854C9> FrontShtDe:00092a08:00:00:     23)   130011 (?)
854E8> FrontShtDe:00092a08:00:00:     24)   130012 (?)
85507> FrontShtDe:00092a08:00:00:     25)   130013 (?)
85526> FrontShtDe:00092a08:00:00:     26)   130014 (?)
85544> FrontShtDe:00092a08:00:00:     27)   130015 (?)
85563> FrontShtDe:00092a08:00:00:     28)   130018 (?)
85582> FrontShtDe:00092a08:00:00:     29)   13001a (?)
855A0> FrontShtDe:00092a08:00:00:     30)   13001b (?)
855BF> FrontShtDe:00092a08:00:00:     31)   13001c (?)


Resolution :

6583E> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f08548, Data:0x c87064e)[29]
6589B> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f08554, Data:0x c8812f0)[31]
660BD> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f13068, Data:0x c8312df)[78]
661F2> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12010, Data:0x c5f064f)[85]
66224> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12014, Data:0x c5f063f)[86]
66281> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f1201c, Data:0x 18b00c9)[88]
662B2> FrontShtDe:ff190d44:15:02: [ENGIO](Addr:0xc0f12020, Data:0x 18b00c7)[89]
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on April 25, 2017, 01:38:19 AM
Quote from: a1ex on April 24, 2017, 03:23:10 PM
There is a closed-source tool that can decode them, to my knowledge.

The decompressor is based on Baldand's LJ92 codec (from MlRawViewer), but it needs some changes. Some of these changes were shared here (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182502#msg182502), so I assume one should be able to integrate them in other converters.

So, all that's required is some manpower to do the integration. No reverse engineering is required.

can you please try this patch?
https://pastebin.com/Ggm0nkw4
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: nikfreak on April 25, 2017, 06:06:47 PM
EOS 7D.203
Attached the full dm.log while taking CR2 in photo mode:
https://drive.google.com/file/d/0B9Mu66yg5QzRV3RzZ0I2Q3dMZWM/view?usp=sharing

Log snippet:

9AA01> FrontShtDe:ff09c558:96:05: sdfExecuteMem1ToRawPath(673)
9AA78> FrontShtDe:ff09c8f4:96:05: sdfExecuteMem1ToRawPath(673)i(SemOK)
9AAAE> FrontShtDe:ff09c92c:96:05: ProcessTwoInTwoOutJpegPath(R) Start(673)
9AAF2> FrontShtDe:00ca6e80:00:00: *** LockEngineResources(89dd38) x17 from ff2de6e4:
9AB33> FrontShtDe:00ca6dd8:00:00:      1)    10002 (read channel 0xa)
9AB5A> FrontShtDe:00ca6dd8:00:00:      2)        3 (write channel 0x3)
9AB7A> FrontShtDe:00ca6dd8:00:00:      3)        4 (write channel 0x4)
9AB9B> FrontShtDe:00ca6dd8:00:00:      4)    30000 (read connection 0x0)
9ABBF> FrontShtDe:00ca6dd8:00:00:      5)    20005 (write connection 0x5)
9ABE4> FrontShtDe:00ca6dd8:00:00:      6)    20016 (write connection 0x16)
9AC04> FrontShtDe:00ca6dd8:00:00:      7)    50003 (?)
9AC23> FrontShtDe:00ca6dd8:00:00:      8)    5000d (?)
9AC42> FrontShtDe:00ca6dd8:00:00:      9)    5000f (?)
9AC60> FrontShtDe:00ca6dd8:00:00:     10)    5001a (?)
9AC7F> FrontShtDe:00ca6dd8:00:00:     11)    80000 (?)
9AC9E> FrontShtDe:00ca6dd8:00:00:     12)    90000 (?)
9ACBD> FrontShtDe:00ca6dd8:00:00:     13)    a0000 (?)
9ACDC> FrontShtDe:00ca6dd8:00:00:     14)   160000 (?)
9ACFC> FrontShtDe:00ca6dd8:00:00:     15)   130003 (?)
9AD1C> FrontShtDe:00ca6dd8:00:00:     16)   130004 (?)
9AD3D> FrontShtDe:00ca6dd8:00:00:     17)   130005 (?)
9ADBE> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400008, Data:0x 36b110f) EngDrvOut
9AE07> FrontShtDe:ff1f8cf4:00:01: [CLKSAVER] ZClockSave OutZ
9AE28> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400008, Data:0x 36b110f) EngDrvOut
9AE5C> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400048, Data:0x       0) EngDrvOut
9AE86> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f01000, Data:0x    8000) EngDrvOut
9AEB4> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f01004, Data:0x       0) EngDrvOut
9AEE8> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0400008, Data:0x 36b110f) EngDrvOut
....
....
....
C20A7> **INT-5Bh*:ff1f4de8:15:01: WriteDMACInterrupt(3)(0x4)(2)
C20CD> **INT-5Bh*:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04304, Data:0x       2) EngDrvOut
C20FB> **INT-5Bh*:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04300, Data:0x       4) EngDrvOut
C2127> **INT-5Bh*:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04304, Data:0x       0) EngDrvOut
C2156> **INT-5Bh*:ff3d2424:15:02: [PackMem] PopInterrupt
C217F> **INT-5Bh*:ff2ddf74:16:03: [TTJ][150,673,0] Write1Complete(POP)CBR
C21AA> **INT-5Bh*:00000588:00:00: <<< INT-5Bh done
C21C1> **INT-5Bh*:ff3d268c:15:02: [PackMem] PopEDmacForMemorySuite 3
C21E2> **INT-5Bh*:00000588:00:00: <<< INT-64h done
C222E> FrontShtDe:ff2ddc80:16:03: [TTJ] STOP WR1:0x3095978 WR2:0x12420030
C2270> FrontShtDe:ff2ddcac:16:03: [TTJ] STOP RD1:0x5f7cf64 RD2:0x1ca25c84
C22AA> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f0300c, Data:0x       0)[0]
C22D9> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f03010, Data:0x       0)[1]
C2304> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f03014, Data:0x       0)[2]
C232E> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f03090, Data:0x       0)[3]
C2358> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f08070, Data:0x80000000)[4]
C238B> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f08540, Data:0x80000000)[5]
C23BE> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f0d000, Data:0x80000000)[6]
C23F1> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f13060, Data:0x80000000)[7]
C2424> FrontShtDe:ff1f6ba8:15:02: [ENGIO](Addr:0xc0f12000, Data:0x80000000)[8]
C2464> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04a04, Data:0x       0) EngDrvOut
C249C> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04304, Data:0x       0) EngDrvOut
C24D1> FrontShtDe:ff1f67c8:15:02: [ENGIO](Addr:0xc0f04404, Data:0x      80) EngDrvOut
C251C> FrontShtDe:00ca6e80:00:00: *** UnLockEngineResources(89dd38) x17 from ff2ddc08:
C255C> FrontShtDe:00ca6dd8:00:00:      1)    10002 (read channel 0xa)
C2582> FrontShtDe:00ca6dd8:00:00:      2)        3 (write channel 0x3)
C25A2> FrontShtDe:00ca6dd8:00:00:      3)        4 (write channel 0x4)
C25C3> FrontShtDe:00ca6dd8:00:00:      4)    30000 (read connection 0x0)
C25E7> FrontShtDe:00ca6dd8:00:00:      5)    20005 (write connection 0x5)
C260D> FrontShtDe:00ca6dd8:00:00:      6)    20016 (write connection 0x16)
C262D> FrontShtDe:00ca6dd8:00:00:      7)    50003 (?)
C264B> FrontShtDe:00ca6dd8:00:00:      8)    5000d (?)
C266A> FrontShtDe:00ca6dd8:00:00:      9)    5000f (?)
C2689> FrontShtDe:00ca6dd8:00:00:     10)    5001a (?)
C26A7> FrontShtDe:00ca6dd8:00:00:     11)    80000 (?)
C26C6> FrontShtDe:00ca6dd8:00:00:     12)    90000 (?)
C26E5> FrontShtDe:00ca6dd8:00:00:     13)    a0000 (?)
C2704> FrontShtDe:00ca6dd8:00:00:     14)   160000 (?)
C2724> FrontShtDe:00ca6dd8:00:00:     15)   130003 (?)
C2745> FrontShtDe:00ca6dd8:00:00:     16)   130004 (?)
C2765> FrontShtDe:00ca6dd8:00:00:     17)   130005 (?)





Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: festr on April 26, 2017, 01:03:38 PM
Hello again, I have tried the mlv dump for mac - it decodes compressed mlv into dng files which is very nice! But there is one major problem - it does not remove vertical strips. Is this known issue? Is there any way how to remove them currently?

edit: oh I realised that the mlv_dump is not able to handle compressed image
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on April 26, 2017, 01:14:41 PM
Yes
http://www.magiclantern.fm/forum/index.php?topic=19429.msg183795;topicseen#msg183795
Yes
http://www.magiclantern.fm/forum/index.php?topic=15108.msg183768#msg183768
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: festr on April 26, 2017, 03:14:10 PM
thank you! It works including vertical stripes removing.

Alex - is vertical stripes removing possible inside the camera before it is even compressed? Or the vertical stripes has to be always corrected out of the camera?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: DeafEyeJedi on April 26, 2017, 05:50:38 PM
Quote from: g3gg0 on April 25, 2017, 01:38:19 AM
can you please try this patch?
https://pastebin.com/Ggm0nkw4

Wonderful stuff and Thanks @g3gg0!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: g3gg0 on April 26, 2017, 07:47:23 PM
most credits go to martinhering, i just added the code he shared to mlv_dump.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Greg on May 05, 2017, 11:49:43 PM
It works with FRSP but the image is not decoded.

if (is_camera("500D", "1.1.1"))
{
    /* uses TwoInTwoOutJpegPath, but functions are mostly compatible */
    TTL_SetArgs     = (void *) 0xFF1AD47C;
    TTL_Prepare     = (void *) 0xFF252748;
    TTL_RegisterCBR = (void *) 0xFF2517D0;
    TTL_Start       = (void *) 0xFF2527B8;
    TTL_Stop        = (void *) 0xFF2527F0;
    TTL_Finish      = (void *) 0xFF252828;

    TTL_ResLock     = (void *) MEM(0x39D0);
}


500D edmac RD1 default configuration :
struct edmac_info RD1_info = {
    .xa     = 1648 * 14/8,
    .xb     = 1616 * 14/8,
    .yb     = 3208 - 1,
    .xn     = 2,
    .off1a  = 4832 * 14/8 - 1648 * 14/8,
    .off1b  = 4832 * 14/8 - 1616 * 14/8,
    .off2b  = 0xe623490,
};

struct edmac_info RD1_info = {
    .xa     = slice1 * 14/8,
    .xb     = slice3 * 14/8,
    .yb     = height + 4 - 1,
    .xn     = 2,
    .off1a  = width * 14/8 - slice1 * 14/8,
    .off1b  = width * 14/8 - slice3 * 14/8,
    .off2b  = (((width * 14/8) * (height + 4 - 1) + 56) * -1 & 0xFFFFFFE),  //digic 3/4
};


7D edmac RD1 default configuration :
struct edmac_info RD1_info = {
    .xa     = 1760 * 14/8,
    .xb     = 1920 * 14/8,
    .yb     = 3520 - 1,
    .xn     = 2,
    .off1a  = 5360 * 14/8 - 1760 * 14/8,
    .off1b  = 5360 * 14/8 - 1920 * 14/8,
    .off2b  = 0xe08556c,
};


Comparison of configuration 500D vs 7D - https://www.diffnow.com/?report=92yqa

Current interpretation - http://s24.postimg.org/uu3knpc0z/ttj-config3.png/
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Greg on May 07, 2017, 06:06:58 PM
It works with live view (not decoded) but we have a problem.

When I take a silent simple picture (LV) it will wait with processing until I get out of LV (just go to play mode).

Conflict with live view reslock :
0x5000d,
0x5000f,
0x5001a,
0x80000,
0x130006,
0x130008,
0x130012,
0x130014,
0x130018,
0x13001c,


Some notes :
int slice1 = (width + 80 + 32) / 3;
int slice2 = (width + 80 + 32) / 3;
int slice3 = (width + 80 + 32) / 3 - 32;

int sliceerr = slice1 + slice2 + slice3 - width;
   
if (sliceerr != 80)
{
    printf("ERROR slice size margin %d\n", sliceerr);
    return -2;
}

EngDrvOut(0xc0f08548, PACK32(slice3 - 2, height + 3));
EngDrvOut(0xc0f08554, PACK32(width + 16, height + 4));
EngDrvOut(0xc0f0858c, slice1 - 32);
EngDrvOut(0xc0f08594, slice1 - 2);
EngDrvOut(0xc0f0d014, PACK32(slice3, height + 3));
EngDrvOut(0xc0f0d018, slice3 + 11);
EngDrvOut(0xc0f0d01c, slice2);
EngDrvOut(0xc0f0d020, slice2 + 3);
EngDrvOut(0xc0f0d024, slice1);
EngDrvOut(0xc0f0d054, PACK32(slice3 + 7, height + 4));
EngDrvOut(0xc0f0d05c, slice2 - 9);
EngDrvOut(0xc0f0d080, PACK32(slice3 + 7, height - 1));
EngDrvOut(0xc0f0d088, slice2 - 9);
EngDrvOut(0xC0F13068, PACK32(width - 1, height - 1));
EngDrvOut(0xc0f12010, PACK32(slice1 - 33, height - 37));
EngDrvOut(0xc0f12014, PACK32(slice3 - 17, height - 37));
EngDrvOut(0xc0f1201c, PACK32((slice1 - 33) / 8, (height  - 37) / 8));
EngDrvOut(0xc0f12020, PACK32((slice3 - 17) / 8, (height  - 37) / 8));

struct edmac_info RD1_info = {
    .xa     = slice1 * 14/8,
    .xb     = slice3 * 14/8,
    .yb     = height + 4 - 1,
    .xn     = 2,
    .off1a  = width * 14/8 - slice1 * 14/8,
    .off1b  = width * 14/8 - slice3 * 14/8,
    .off2b  = (((width * 14/8) * (height + 4 - 1) + 56) * -1 & 0xFFFFFFE),  //digic 3/4
};


500D ML TTJ_Start :
*** StartEDmac(0x3, 0x0), from ff28f6e4
    addr c000064, ptr c000064, size
StartEDmac(^^)/=== (3)(0)(0xc000064) ===
[ENGIO](Addr:0xc0f04304, Data:0x20000001) EngDrvOut  //WR1
[ENGIO](Addr:0xc0f04300, Data:0x       1) EngDrvOut  //WR1
[ENGIO](Addr:0xc0f12000, Data:0x       1)[0]
[ENGIO](Addr:0xc0f13048, Data:0x       3)[1]
[ENGIO](Addr:0xc0f13060, Data:0x       0)[2]
[ENGIO](Addr:0xc0f13000, Data:0x       0) EngDrvOut
[ENGIO](Addr:0xc0f13048, Data:0x       3) EngDrvOut
[ENGIO](Addr:0xc0f1304c, Data:0x       0) EngDrvOut
[ENGIO](Addr:0xc0f13004, Data:0x       1) EngDrvOut
[ENGIO](Addr:0xc0f0d000, Data:0x       1)[0]         
[ENGIO](Addr:0xc0f08540, Data:0x       1)[1]         //RSHD_ENB
[ENGIO](Addr:0xc0f08070, Data:0x       1)[2]         //UNPACK24
*** StartEDmac(0xa, 0x2), from ff251bc0
    addr 8008484, ptr 8008484, size
StartEDmac(^^)/=== (10)(2)(0x8008484) ===
[ENGIO](Addr:0xc0f04a04, Data:0x60020000) EngDrvOut  //RD1
[ENGIO](Addr:0xc0f04a00, Data:0x       1) EngDrvOut  //RD1
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: lorenzo353 on May 09, 2017, 10:50:08 PM
Hi Greg,

how do you find slice1, slice3, height and width values, compared to this table (extracted from CR2 files)
http://lclevy.free.fr/cr2/#app, Sensor information ?
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_database.txt
and this poster:
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_lossless.pdf

Kind regards,

Laurent

Quote from: Greg on May 05, 2017, 11:49:43 PM
It works with FRSP but the image is not decoded.

500D edmac RD1 default configuration :
struct edmac_info RD1_info = {
    .xa     = 1648 * 14/8,
    .xb     = 1616 * 14/8,
    .yb     = 3208 - 1,
    .xn     = 2,
    .off1a  = 4832 * 14/8 - 1648 * 14/8,
    .off1b  = 4832 * 14/8 - 1616 * 14/8,
    .off2b  = 0xe623490,
};

struct edmac_info RD1_info = {
    .xa     = slice1 * 14/8,
    .xb     = slice3 * 14/8,
    .yb     = height + 4 - 1,
    .xn     = 2,
    .off1a  = width * 14/8 - slice1 * 14/8,
    .off1b  = width * 14/8 - slice3 * 14/8,
    .off2b  = (((width * 14/8) * (height + 4 - 1) + 56) * -1 & 0xFFFFFFE),  //digic 3/4
};


7D edmac RD1 default configuration :
struct edmac_info RD1_info = {
    .xa     = 1760 * 14/8,
    .xb     = 1920 * 14/8,
    .yb     = 3520 - 1,
    .xn     = 2,
    .off1a  = 5360 * 14/8 - 1760 * 14/8,
    .off1b  = 5360 * 14/8 - 1920 * 14/8,
    .off2b  = 0xe08556c,
};


Comparison of configuration 500D vs 7D - https://www.diffnow.com/?report=92yqa

Current interpretation - http://s24.postimg.org/uu3knpc0z/ttj-config3.png/
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Greg on May 10, 2017, 12:11:25 AM
I found this in dm-spy log and my interpretation. Edmac configuration also explains this.
I needed this to configure the image processor to live view/video mode.

Looking at your link - http://lclevy.free.fr/cr2/#app
Look at EOS Kiss X3 (500D) :
sensor width = 4832
slices[0] = 2
slices[1] = 1616
slices[2] = 1600

1616*2 + 1600 = 4832


Of course, the sizes needed for decoding may be slightly different.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: lorenzo353 on May 10, 2017, 09:52:25 PM
thanks Greg,

I added a section in  my doc:
http://lclevy.free.fr/cr2/index.html#ml
to highlight Magic Lantern work and links with Canon Raw v2 format interpretation.

please let me know if you know other useful information to open this proprietary format...

Laurent
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: festr on June 03, 2017, 11:57:38 AM
will be possible in near future using lossless compression alongside with sound recording? Anyone working on it or there are too much work to be done?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on June 03, 2017, 12:16:30 PM
http://www.magiclantern.fm/forum/index.php?topic=19300.msg184232#msg184232
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: festr on June 03, 2017, 01:47:30 PM
Is it possible to have both old raw rec with the sound (old module) with the new one in the same build? In case I want to decide recording audio without the compression but most of the time will be no audio with compression.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on June 03, 2017, 01:54:52 PM
Keep 2 cards and change them if needed.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: festr on June 05, 2017, 01:14:26 AM
good idea!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on June 30, 2017, 07:39:41 AM
Found this today on my 5d2 rom.bin string

3373. ff088c74 StartYuvToJpegPath(%d)
3374. ff088f34 sdfExecuteMem1ToRawPath(%d)
3375. ff088f54 RAW_OBWB
3376. ff088f60 ->FindObject RAW_OBWB
3377. ff088f7c SRAW_OBWB
3378. ff088f8c ->Unknown PictureSize(Mem1ToRaw)
3379. ff088fb0 sdfExecuteMem1ToRawPath(%d)
3380. ff088fcd (SemOK)
3381. ff088fd8 ProcessTwoInTwoOutJpegPath(R) Start(%d)
3382. ff089000 ProcessTwoInTwoOutJpegPath(R) End(%d)
3383. ff089028 ->FindObject SRAW_OBWB
3384. ff0892d4 sdfExecuteMem1ToJpegPath(%d)
3385. ff08930c ProcessTwoInTwoOutJpegPath(J) Start(%d)
3386. ff089334 ProcessTwoInTwoOutJpegPath(J) End(%d)

Not that I know how to use this yet but it looks like's it maybe possible to have lossless on 5d2 .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on June 30, 2017, 10:02:11 AM
First step: get a log that contains these strings. The startup-log (https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/) builds should be fine to get started; to get more details in the log file, see episode #2 earlier in this thread.

You'll need to understand:
- how these things work in vanilla firmware (qemu with "-d io" is very useful for this)
- how to call them from your code (what functions to call, what parameters they take, what preconditions they might require - see previous messages)
- what parameters and/or registers to change and how, in order to modify resolution (see previous messages, in particular the ones from Greg)
- how to fix the conflicts with LiveView, in order to call it there (ResLock; unfortunately, this step was unsuccessful on 500D).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on July 01, 2017, 06:38:07 AM
Ok --I'll do my best to port this to the 5d2 (swimming in the deep end now , hope there's life jacket near by  :P
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on July 02, 2017, 09:05:57 AM
I'd suggest to start with smaller goals - by 1 or 2 orders of magnitude :D

For example, look up "easy coding task" around the forum and pick one.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 07, 2017, 01:17:23 PM
Quote from: dfort on April 13, 2017, 07:39:44 AM
Good to verify some of the 700D addresses. Still can't save lossless full res DNG's but the warning messages and crashes are gone.

Had some fun trying to continue where dfort finished. After some searching, trying, failing and starting again I got the 700d taking full res silent pictures in Lossless DNG. Still in the default resolution, but at least it's a start.

    if (is_camera("700D", "1.1.4"))
    {
        /* ProcessTwoInTwoOutJpegath, 700D 1.1.4 */
        TTL_SetArgs     = (void *) 0xFF35F510;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF424BA4;      /* called right after ProcessTwoInTwoOutJpegath(R) Start(%d); */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF423B88;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_700D;  /* this function is inline on 700D */
        TTL_Start       = (void *) 0xFF424C4C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF423DD4;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF424CBC;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

    uint32_t resources[] = {
        0x10002,                        /* read channel 0x8 */
        edmac_channel_to_index(0x20),   /* write channel 0x11 */
        0x20005,
        0x20016,
        0x30002,
        0x50034,
        0x5002d,
        0x50010,
        0x90001,
        0x90000,
        0xa0000,
        0x160000,
        0x260000,
        0x260001,
        0x260002,
        0x260003,
    };


    TTL_Args.WR1_Channel = 0x20;


Shooting simple silent photo's is hit or miss and in the wrong aspect ratio.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 12, 2017, 09:48:28 PM
Quote from: ErwinH on August 07, 2017, 01:17:23 PM
I got the 700d taking full res silent pictures in Lossless DNG.

Nice! Do you have something on bitbucket you could share?

I tried using your changes--note that you also need this:

static void set_flags_700D()
{
    EngDrvOut(0xC0F37010, (shamem_read(0xC0F37010) & 0xFFE3FFFF) | 0xC0000);
    EngDrvOut(0xC0F3704C, (shamem_read(0xC0F3704C) & 0xFFFFFFC0) | 0x20);
    EngDrvOut(0xC0F37074, (shamem_read(0xC0F37074) & 0xFFC0FFFF) | 0x180000);
    EngDrvOut(0xC0F37078, (shamem_read(0xC0F37078) & 0xFFC0FFFF) | 0x180000);
}


Branching off the compressed_raw branch and adding your changes it does build and I was able to shoot silent stills with the Lossless DNG setting but they were not compressed. How did you do it?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 13, 2017, 07:35:35 PM
I tool the crop_rec_4k branche, took out the 5D3 specific boothacks and that runs like a charm.

I need some help with the raw buffers (for the mlv_lite) but I'll share my code with you tomorrow.

Warning, I did encounter a null pointer error during recording when the card couldn't keep up and the scan_a5a5 routines warns about a null pointer bug all the time...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 14, 2017, 09:53:06 AM
dfort: https://bitbucket.org/ehoutsma/magic-lantern/src/?at=crop_rec_4k_700d

Just took 2 pictures, one in DNG and one in Lossless DNG (both with the lenscap on), Lossless DNG was 1.4MB, the normal DNG 2.3MB.

I haven't disabled the null pointer warning ;)

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 15, 2017, 01:18:07 AM
@ErwinH

That null pointer warning makes for a wild and crazy ride:

(https://farm5.staticflickr.com/4365/36525244276_0be9847a5e.jpg) (https://flic.kr/p/XDBqAY)

I wasn't able to get a compressed Full Resoution Silent Picture DNG but it did work for Simple Silent:

(https://farm5.staticflickr.com/4433/36525246246_db850b3e7e_m.jpg) (https://flic.kr/p/XDBrbW)

The first few I shot came out blank and Adobe Camera Raw wouldn't open them but the next batch came out fine. That's to be expected with experiments like this. A while back I remember testing out the crop_rec_4k branch and finding out that FRSP was not working so it isn't surprising if these changes broke it.

Next test, compressed raw video with reduced bit-depth. Don't know if "mlv_dump -v" shows if the video is compressed and at what bit-depth it was recorded in but:

...
    Frames Video: 916
...
    Res:  1736x976
...
     Camera Name:   'Canon EOS REBEL T5i'
...


Holy Smokes! First test was 38 seconds of raw video at that resolution with no corrupt frames. Subsequent tests were even better. Adjusting the aspect ratio to 1.85 and 2.35 (the standards in cinema) recording was continuous. This works both in mv1080 (1736x936 for 1.85 and 1736x736 for 2.35) and mv1080crop (1800x968 for 1.85 and 1800x760 for 2.35).

Going even more extreme, zoom mode:

    FPS         : 23.976000
...
    Frames Video: 173
...
    Res:  2520x1072


Congratulations!

Will the crop_rec module add anything to this? It should be possible to get more resolution options but it is very doubtful that recording 4K raw video on a 700D is possible unless you adjust the frame rate way down.
Title: ProcessTwoInTwoOutLosslessPath
Post by: DeafEyeJedi on August 15, 2017, 07:59:59 AM
That's some sick work @dfort and thanks for sharing the source code @ErwinH! :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Teamsleepkid on August 15, 2017, 08:23:05 AM
You guys are awesome. Nice going ErwinH. And Dfort you got it man. Hats off to you guys :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on August 15, 2017, 08:24:40 AM
Wonderful work@dfort n ErwinH
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 15, 2017, 04:29:38 PM
I also tried to get this running on the 600D, but I can't get the dm-spy-experiments branch running on the 600D. The led blinks, once per second for a few times, then it blinks fast and it starts over. Same for the startup-log builds. (even the older ones). Am I missing something?

Finding the stubs was easy, except for the setflags bit of course.

    if (is_camera("600D", "1.0.2"))
    {
        /* ProcessTwoInTwoOutJpegPath, 600D 1.0.2 */
        TTL_SetArgs     = (void *) 0xFF2155E0;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF2FE354;      /* called right after ProcessTwoInTwoOutJpegPath(R) Start(%d); */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF2FD424;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_600D;  /* this function is also inline on 600D??? */
        TTL_Start       = (void *) 0xFF2FE3C4;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF2FE3FC;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF2FE434;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 15, 2017, 08:15:19 PM
Sorry for delay - was a bit away from ML lately, so anything that required more than a 5-minute break went to the back burner. Nice to see the progress - now the questions:

1) the null pointer warning appears to be false - both my 700D ROMs have that magic value. A sure way to tell would be to compare with a ROM from a 700D that never had ML on it - for example, the ROM backup saved during ML installation. If anyone made a backup copy of those ROMs (easily recognized by their boot flag status), they could be useful.

Detailed analysis: I get the magic value 0xA5A5A5A5 at the following offsets:
- 0xa5403c: that's in the middle of other magic values with similar patterns (e.g. 34 34 34 34 A5 A5 A5 A5 E5 E5 E5 E5 F1 F1 F1 F1). This pattern is also present in the firmware update from Canon.
- 0xc950e4, 0xc950e8, 0xc950ec, 0xc950f0. This is in the data area, not present in Canon's update and not recognized by my property parser either. Not sure what's up with it, but there are similar patterns nearby.
- 0xc95bd0, 0xc95bd4, 0xc95bd8, 0xc95bdc: same as above.
- 0xc9f1b0, 0xc9f1b4: like the first one, except it's not present in Canon's update.
- 0xfdb4b0: same as above.

2) Buffers - this was a bit of guesswork. On 5D3, I've assumed the raw buffer allocated by Canon is at least as big as the maximum resolution - the one from 5x zoom. With dm-spy-experiments I've found out a possible size for that buffer (commit 1ee9679 => 0x1cae000, so about 30 MB). Experimentally I've found out that was wrong - the buffer was overwritten by something else.

Another interesting discovery - as soon as leaving LiveView, a small part of Canon's default raw buffer (0x4d31a000) was overwritten near 0x4d600000 - about 2.9MB after its beginning. If we always stay in LiveView, things should be fine - however, we are using the "paused" LiveView for various things, so I wanted to play safe and pick a buffer that's not going to be overwritten when doing that. So I took whatever was not overwritten - found by trial and error between 0x4d600100 and 0x4ee00000: about 24 MB. Enough for 4K, but not for full-res (in 14-bit). For that (extreme) case, the only way to allocate a contiguous chunk of memory was with the SRM backend from Canon - they use it as raw buffer for still photo capture. It's very tricky to use - the buffers have to be freed in exactly the same order as allocated, there's a BUSY sign printed on the screen while this buffer is allocated, while this sign is on the screen it also locks up the UI as soon as you move a scrollwheel, and so on. For that reason, I went for something very hackish - intentional use after free for this buffer (assuming Canon code won't use or move the SRM buffers while in LiveView). It's not very robust, but it's only used with full-res LiveView (since that's the only case we need more than 25 MB for one frame).

To see the memory map, and find out what areas might be free, there is CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP. Without it, the RAM is not cleared at boot, so you'll get different results (and likely not relevant) every time you boot.

Then, once you have guessed a possibly free memory region, you'll need to test it somehow to make sure nobody overwrites it (at least during LiveView). There is some test code for that in the RscMgr_memory branch, but it's nothing fancy - you could just fill the buffer with something and check whether it's still unchanged, in a loop.

RAW_LV_BUFFER_ALLOC_SIZE is how much we need for a full-res image (max W * max H * 14/8).

3) Regarding 07573f3 - that's a bit difficult to do in C (since the arrays have different lengths for different camera models). See for example mpu.c in the QEMU branch for a similar situation.

Tip: the ResLock entries do not depend on the firmware version (they probably depend only on the DIGIC generation), so you can use is_camera("5D3", "*") to cover all versions.

4) 600D LED blink means "out of memory". You could remove some features in features.h (you can even boot with an empty features.h, also comment out CONFIG_RAW* and related entries, e.g. EDMAC, and - on 600D - also remove audio-lapis.o from Makefile.setup.default). [ todo: allow compiling without features on all models, and include this on jenkins ]

Another todo for me: integrate dfort's 700D crop_rec in the 4K branch [DONE], and merge the non-4K changes (e.g. without memory hacks) into mainline (WIP in the raw_fixes branch).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 17, 2017, 11:27:57 AM
1) I altered the scan_A5A5 routine so it won't trip over A5A5A5A5 at these addresses:
0xF8C950E4 - 0xF8C950F0
0xF8C95BD0 - 0xF8C95BDC
0xF8C9F1B0 - 0xF8C9F1B4
If they do appear at another location it will show the warning again.

I haven't had a good look at the buffers yet.

3) I've combined the resources for the 700D * and the 5D3 *. It is working on my 700D and should work on the 5D3. Hope this is a solution you can live with.

Quote from: dfort on August 15, 2017, 01:18:07 AM
I wasn't able to get a compressed Full Resoution Silent Picture DNG but it did work for Simple Silent:

Found the issue with the FRSP. Tricking the compressor in half the height, double the width makes it fail. Removing the width * 2 and height / 2 does grab the lossless dng. I don't know if they are valid, but Rawtherapee and Resolve 14 accept them.

P.S. Here are the entry points for lossless decompression on the 700D. I do get a lot of malloc errors, but replacing malloc with fio_malloc @mlv_play.c:2220 fixed those.
    if (is_camera("700D", "1.1.4"))
    {
        Setup_DecodeLosslessRawPath = (void*)0xFF4294DC;
        Start_DecodeLosslessPath = (void*)0xFF4295A4;
        Cleanup_DecodeLosslessPath = (void*)0xFF429708;
    }
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 17, 2017, 04:23:07 PM
Did a "short" testrun, recording 16:9 using 11 bits lossless compression.


    FPS         : 25.000000
...
    Frames Video: 23504
...
    Res:  1736x976
...
     ISO:        800
...
     Time: 942568.613000 ms
...
   Frame: #23503


Only thing is the battery went out of juice ;) Tomorrow I'll be recording an interview and will be using this camera as my second (backup) camera, using plugin power. Let's see if we can fill up the SD card without interruption.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on August 17, 2017, 04:30:14 PM
Eosm maybe? :P 8)
Awesome progress.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 17, 2017, 06:19:36 PM
Quote from: Danne on August 17, 2017, 04:30:14 PM
Eosm maybe? :P 8)

I was thinking the same thing. The main sticking point would be to figure out if this applies to the EOSM and if so, how to find the proper values.

static void set_flags_700D()
{
    EngDrvOut(0xC0F37010, (shamem_read(0xC0F37010) & 0xFFE3FFFF) | 0xC0000);
    EngDrvOut(0xC0F3704C, (shamem_read(0xC0F3704C) & 0xFFFFFFC0) | 0x20);
    EngDrvOut(0xC0F37074, (shamem_read(0xC0F37074) & 0xFFC0FFFF) | 0x180000);
    EngDrvOut(0xC0F37078, (shamem_read(0xC0F37078) & 0xFFC0FFFF) | 0x180000);
}


BTW -- The 700D can now use some of the crop_rec module resolutions (https://bitbucket.org/hudson/magic-lantern/commits/35b5b5e94269e22f168c81ec5ce44b9b12cbb124). With lossless compression maybe we can start experimenting with some of the other crop_rec options?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 18, 2017, 07:29:29 PM
The EOS M and 650D should be very similar. 100D uses the same pattern, only the code was refactored a bit. EOS M2 is likely similar to 100D. 70D and 6D are more like 5D3 (TwoInTwoOutLosslessPath). All these models are very good choices for low-hanging fruits.

BTW, 700D might also work with FF36B2D8 TTL_SetFlags. There are two versions of StartTwoInTwoOutJpegPath - one of them (the one I've looked at back then) has this function inline, the other one calls it.

Whatever these flags do, they also worked on 5D3 (with TTJ routines (http://www.magiclantern.fm/forum/index.php?topic=18443.msg182989#msg182989)), so they are probably common to all DIGIC 5 models.

Interesting notice about doubling the width no longer needed; will look into it, but not right now.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 18, 2017, 08:09:30 PM
The doubling only has to be removed for the FRSP, the mlv do need half the height, double the width. I'm not really sure about the simple silent pictures.

I'll test the other option for the set flags later and match it to the 600D and check if that works.

The real test today was a bit of a dissapointment. Recording at 1736x976 wasn't continyous... It stopped after 33000 frames (because the card was full)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 18, 2017, 09:17:56 PM
Recording 33000 frames was a disappointment?

I took the latest crop_rec_4k_700d branch from ErwinH's bitbucket repository (https://bitbucket.org/ehoutsma/magic-lantern/src/?at=crop_rec_4k_700d), merged in the latest crop_rec_4k changes and got the crop_rec module working. That should give you 60fps raw recording though the image size needs to be scaled down to get continuous recording. Here is a link (https://bitbucket.org/daniel_fort/magic-lantern/branch/crop_rec_4k_700d_eosm) to the branch I'm working on. Yeah, I'm going to attempt once again to get lossless compression working on the EOSM.

I'm still not able to record lossless compressed Full Resolution Silent Stills on the 700D. I was able to get a screenshot of the error message.

(https://farm5.staticflickr.com/4341/35819680704_337da1d87b_z.jpg) (https://flic.kr/p/Wzge8N)

Just to make sure that the 5D3 didn't break with all these changes I compiled for the 5D3.113 from my working branch and YIKES! The null pointer warning message came up! Rather unnerving but I went ahead and shot some FRSP frames and lossless compression is working on that camera.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 18, 2017, 09:25:28 PM
Oops. I forgot to change the test for 0XE5E5E5E5. it should continue if it is 0XE5E5E5E5 and not if it''s something else.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 18, 2017, 09:35:22 PM
Quote from: dfort on August 18, 2017, 09:17:56 PM
Rather unnerving but I went ahead and shot some FRSP frames and lossless compression is working on that camera.

Really? I'm currently trying to take some full-res frames and I'm getting the above errors on 5D3.123...

(I've started from the current 4K experimental build, and went back to first known good (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183489#msg183489) build with lossless silent pictures - but that one isn't working either...)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 18, 2017, 10:04:06 PM
For the 700D the solution was to remove the * 2 and / 2 in these statements.
TTL_Args.xRes = width * 2;
    TTL_Args.yRes = height / 2;
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 18, 2017, 10:14:45 PM
Interesting -- I just grabbed the current 5D3.113 crop_rec_4k build from the experiments download page and it displays that "Raw error, falling back to YUV overlays" message though it does save a lossless full-res DNG frame. I might have missed the message because the LiveView screen was displaying the null pointer message which pretty much takes over the entire screen.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 18, 2017, 11:21:02 PM
Figured it out - I've been compressing the raw data in place (reusing the memory buffer); however, the overlays no longer had a valid raw buffer. Will disable that (though I might have to re-think the memory management).

Back then, I must have tested it with global draw off...

A bigger trouble I have is with full-res LiveView snapshots (crop_rec enabled and set to Full-res LV, silent pictures set to Simple). Regular (uncompressed) DNG works fine. Lossless DNG usually locks up the camera, though it's not 100% repeatable.

QuoteI might have missed the message because the LiveView screen was displaying the null pointer message which pretty much takes over the entire screen.

If that was on 5D3, I should take a look at your ROM.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 19, 2017, 01:50:30 AM
Quote from: ErwinH on August 18, 2017, 10:04:06 PM
For the 700D the solution was to remove the * 2 and / 2 in these statements.
TTL_Args.xRes = width * 2;
    TTL_Args.yRes = height / 2;


Yep--that works. Able to capture full-res silent with that "fix" on both the 700D and 5D3.

Found most of the addresses on the EOSM but am stumped in a few spots.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 19, 2017, 03:19:41 AM
Lossless compression working on the EOSM.

https://bitbucket.org/daniel_fort/magic-lantern/branch/crop_rec_4k_700d_eosm

A bit hackish. When in doubt I just used the 700D values but it is working. Did continuous recording, 23.98 fps at 1648 x 696 using the crop_rec module.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on August 19, 2017, 03:26:38 AM
 :o :o :o
Hopefully reach my eosm on sunday. This thread is rockin'! And in the crop_rec module. The best.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 19, 2017, 03:34:48 AM
Processed in Switch of course.


Sent from my iPhone using Tapatalk
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: DeafEyeJedi on August 19, 2017, 05:59:36 AM
Exciting news. On my way home from work. Can't wait to grab the EOSM out and do you have a build to share or do I need to try compiling again?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: mothaibaphoto on August 19, 2017, 09:17:19 AM
Quote from: a1ex on August 18, 2017, 11:21:02 PM
Figured it out - I've been compressing the raw data in place (reusing the memory buffer); however, the overlays no longer had a valid raw buffer.
Is that a reason I get the "Raw error, falling back to YUV overlays" message when trying to shoot full-res silent compressed DNG? After searching the forum i decided that it was my camera sensor fault.
Here is detailed description of what i did and what i get:
with crop_rec_4k.2017Jun19.5D3113 i activated ettr and silent modules, Auto ETTR configured to "Press Set".
Than, i can expose with Set and take image with half press.
I see that "Raw error, falling back to YUV overlays" message on preview, but everything is OK. Next, i activated intervalometer, and first what i noticed is Auto ETTR now "Always ON" on its own.
As far as i understand, ETTR can calculate exposure with low res liveview image and after taking image with that hi res image - this is how it used to work on previous builds with intervalometer and ETTR configured to "Press Set".
This explains why in poor light conditions when liveview is extremely amplified, ETTR failed to get correct exposure until you start intervalometer and take couple frames.
And now it looks like i can't get correct exposure either - because ETTR is liveview only, and, moreover, not RAW based?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 19, 2017, 09:28:43 AM
Correct - ETTR also relies on the raw buffer, which was overwritten. Just pushed the fix (but didn't test this scenario).

Also went ahead and pulled ErwinH's and dfort's changes for 700D/EOSM and did some minor cleanups. Didn't pull the null pointer changes yet (want to test them in QEMU first).

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ErwinH on August 19, 2017, 03:03:24 PM
The null pointer changes are incorrect. Both tests are reversed. Do other cameras then the 5D3 incorporate this test since they can't be affected by the same bug.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 19, 2017, 08:27:16 PM
Quote from: a1ex on August 19, 2017, 09:28:43 AM
Also went ahead and pulled ErwinH's and dfort's changes for 700D/EOSM and did some minor cleanups.

Great!

Noticed that a change I made to mlv_play for the EOSM that I later reverted got merged. Also, ErwinH's 700D change didn't get merged. As a result that module won't build.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 19, 2017, 11:43:53 PM
The 5D3 likes the latest changes but 700D and EOSM are complaining.

Only mlv_lite active, Lossless 14bit.

ML ASSERT:
fullsize_buffers[1] == UNCACHEABLE(raw_info.buffer)
at mlv_lite.c:1379 (free_buffers), task raw_rec_task
lv:1 mode:3

raw_rec_task stack: 1acd08 [1acdd0-1abdd0]
0xUNKNOWN  @ bba0:1acdc8
0x00ABCC14 @ ac1140:1acd48
0x0007F5E4 @ abcc74:1acd38
0x0007EF68 @ 7f644:1acd08

Magic Lantern version : Nightly.2017Aug19.700D114
Mercurial changeset   : 024351174cb1 (crop_rec_4k) tip
Built on 2017-08-19 21:34:59 UTC by rosiefort@RosieFoComputer.
Free Memory  : 128K + 3206K


ML ASSERT:
fullsize_buffers[1] == UNCACHEABLE(raw_info.buffer)
at mlv_lite.c:1379 (free_buffers), task raw_rec_task
lv:1 mode:3

raw_rec_task stack: 1ded18 [1dede0-1ddde0]
0xUNKNOWN  @ c380:1dedd8
0x00AB4A34 @ ab8f60:1ded58
0x0009EBC8 @ ab4a94:1ded48
0x0009E548 @ 9ec28:1ded18

Magic Lantern version : Nightly.2017Aug19.EOSM202
Mercurial changeset   : 024351174cb1 (crop_rec_4k) tip
Built on 2017-08-19 21:39:42 UTC by rosiefort@RosieFoComputer.
Free Memory  : 161K + 3404K
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: mothaibaphoto on August 20, 2017, 04:38:55 AM
I posted some feedback as comments to appropriate commits on bitbucket in hope that this way is more convenient for developers.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 20, 2017, 09:49:30 PM
Hopefully sorted now - experimental builds available.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on August 20, 2017, 11:14:21 PM
Cool stuff. Gonna try this on my eosm.
Pulled the latest from crop_rec_4k and gcc moved in make?
dans-MBP:EOSM.202 dan$ make zip
/bin/sh: /Users/dan/bin/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3: No such file or directory

Created a bin folder in home folder and put the gcc-arm-none-eabi-4_8-2013q4 int it and it compiles. Anyway. Shoud be /Users/dan/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3 without the first bin?

*Just filmed crop_rec 14bit lossless on my eosm(continuous). Really amazing :o. Lovely stuff. Thanks all involved. The best. mlv_play works flawlessly too!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: nikfreak on October 08, 2017, 01:32:25 PM
tried to match stubs for digic4 (7D) and it seems it can be done but TTL_SetFlags seems to be the only one making troubles. At least i can't find / match it.
Any hint on that one? I guess (not checked) it could be same for 60d, 5d2 etc.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on November 23, 2017, 01:24:51 AM
@nikfreak

Just checking notes -- this is what I've got. Not that I have this working yet but it looks about right.

TTL_SetFlags is called after TTL_RegisterCBR:
    if (is_camera("7D", "2.0.3"))
    {
        /* ProcessTwoInTwoOutJpegPath, 7D 2.0.3 */
        TTL_SetArgs     = (void *) 0xFF224324;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF2DE720;  /* called right after ProcessTwoInTwoOutJpegPath */
                                                /* calls [TTL] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF2DDA1C;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF0CA978;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF2DEA04;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF2DEA3C;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF2DEA74;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


The decode, "RequestDecodeLosslessRawPath" on the 7D, seems pretty straightforward:

    if (is_camera("7D", "2.0.3"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xFF2E8EAC;
        Start_DecodeLosslessPath    = (void *) 0xFF2E8F5C;
        Cleanup_DecodeLosslessPath  = (void *) 0xFF2E906C;
    }
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: nikfreak on November 23, 2017, 01:52:51 PM
You'll need this too:

http://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741

No time atm myself to verify / test anything.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: esas on December 31, 2017, 05:05:15 PM
I've tried to confirm the stubs for 70D by pattern matching with the EOSM, but that camera uses TTJ instead of lossless. Could someone please PM me a ROM1.bin from a 5D3, or at least a disassembly showing the relevant stubs.

Have also been trying to confirm the stubs on 650D. I hoped they would match the EOSM one, but they don't. Since the lossless videos from both the 650D and 70D looks very similar (same faults in both) I was hoping that if I managed to get one working I would know how to fix the other.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: 12georgiadis on January 02, 2018, 07:52:17 PM
Quote from: nikfreak on November 23, 2017, 01:52:51 PM
You'll need this too:

http://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741

No time atm myself to verify / test anything.

Hello Nick, Dfort, I'd like to know where was the work in progress on 7D's lossless ?
This feature would be awesome to get enough bandwidth for H264's proxy recording alongside 10 bits lossless. In this way, we also solve the sound issue. That allows filmmakers to review clip on set in real time and can be used for fiction. Of course, if this is working on 7D, I imagine it can be done on others Digic 4 too !
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on January 03, 2018, 09:48:02 PM
Quote from: 12georgiadis on January 02, 2018, 07:52:17 PM
Hello Nick, Dfort, I'd like to know where was the work in progress on 7D's lossless ?

In Reply #204 (http://www.magiclantern.fm/forum/index.php?topic=18443.msg193534#msg193534) I posted the addresses I found for the 7D but @nickfreak pointed out that there is more to it (http://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741) in order to get it working. I'm not really sure what to do with that information but it might be that the 7D doesn't fit into the resources[] array that was established for the other cameras.

lossless.c
    lossless_sem = create_named_semaphore(0, 0);
   
    if (is_camera("700D", "*") || is_camera("650D", "*") || is_camera("EOSM", "*") || is_camera("100D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x20005,
            0x20016,
            0x30002,
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x90000,
            0xa0000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };

        TTL_ResLock = CreateResLockEntry(resources, COUNT(resources));
    }
    else if (is_camera("5D3", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };


We were able to get lossless working on the 6D (more or less) using the resources[] for the 5D3 but I wasn't successful in getting the 7D working with either set.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: 12georgiadis on January 04, 2018, 01:08:56 AM
Quote from: dfort on January 03, 2018, 09:48:02 PM
In Reply #204 (http://www.magiclantern.fm/forum/index.php?topic=18443.msg193534#msg193534) I posted the addresses I found for the 7D but @nickfreak pointed out that there is more to it (http://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741) in order to get it working. I'm not really sure what to do with that information but it might be that the 7D doesn't fit into the resources[] array that was established for the other cameras.

lossless.c
    lossless_sem = create_named_semaphore(0, 0);
   
    if (is_camera("700D", "*") || is_camera("650D", "*") || is_camera("EOSM", "*") || is_camera("100D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x20005,
            0x20016,
            0x30002,
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x90000,
            0xa0000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };

        TTL_ResLock = CreateResLockEntry(resources, COUNT(resources));
    }
    else if (is_camera("5D3", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };


We were able to get lossless working on the 6D (more or less) using the  resources[] for the 5D3 but I wasn't successful in getting the 7D working with either set.
Thanks Dfort for clarifying it ! I understand now that there is a difference between digic 5 and digic 4 cameras regarding resources[] to get lossless rec working and porting it successfully with older cameras. This would be crazy if someone get it to work because it's not just 7D that can have this benefit but also 5d2 and other digic 4 dslr.


Envoyé de mon iPhone en utilisant Tapatalk
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: VivaLaTip on January 04, 2018, 10:07:13 PM
Great work! Do I hear 6D and lossless?  :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 09, 2018, 08:40:29 AM
Info on the 5D2 cr2 from cr2_database (https://raw.githubusercontent.com/lclevy/libcraw2/master/docs/cr2_database.txt)
I guess I  need to find studs  for this now .

Edit: This give some very good info on Canon Compression cr2_lossless.pdf (https://github.com/lclevy/libcraw2/raw/master/docs/cr2_lossless.pdf)
(https://thumb.ibb.co/kRx7kc/cr2_compression_1.png) (https://ibb.co/kRx7kc)  (https://thumb.ibb.co/c1nwdx/cr2_compression_2.png) (https://ibb.co/c1nwdx)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: DeafEyeJedi on February 09, 2018, 09:21:21 AM
Great details @reddeercity especially w that cr2_lossless.pdf sheet.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 10, 2018, 08:29:22 AM
Here some good info form my rom
ff088fd8 ProcessTwoInTwoOutJpegPath(R) Start(%d)
ff089000 ProcessTwoInTwoOutJpegPath(R) End(%d)
ff089028 ->FindObject SRAW_OBWB
ff0892d4 sdfExecuteMem1ToJpegPath(%d)
ff08930c ProcessTwoInTwoOutJpegPath(J) Start(%d)
ff089334 ProcessTwoInTwoOutJpegPath(J) End(%d)

Need to get a log file while taken cr2 photo so I find the addresses
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 12, 2018, 11:02:13 AM
Ok after reading 25,000 lines from my startup log  on 5D2 with taken a cr2 photo I think I have it finial  :D
83523> ShootPreDe:ff8ab340:00:03: [JOB] PostNextStageForDevelop (ID = 5694, PictType = 0x400)
8361A>FrontShtDe:ff88a0fc:96:05: ->sdsPostJob InsertStageJob(5694)
8366B> FrontShtDe:ff8aacd0:00:03: [JOB] PostNextStage (ID = 5694, Class = 150, Pos = 4)
836D7> FrontShtDe:ff88a170:96:05: sdsPostJob Start(5694)
8370F> FrontShtDe:ff88a19c:96:05: sdfGetDevelopDevType(5694)(0x7000)
83755> FrontShtDe:ff88a210:96:05: ENABLE(Poster:0)(Ohyear:0)
83793> FrontShtDe:ff8893e4:96:05: sdfAllocateMemoryDevelopSuite(5694)


4A90> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5694)
84B1E> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5694)��(SemOK)
84B5D> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5694)

84EB6> FrontShtDe:ffa59694:16:03: [TTJ][150,5694,0] RAW(5792,3804,0,14)
84F6E> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
84FAF> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1ae2c0f0
84FE9> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864

850B0> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
851C2> RearShtDev:ff88738c:97:05: ->sdsPostJob InsertStageJob(5694)
85216> RearShtDev:ff8874f8:97:05: sdsPostJob Start(5694)
85246> RearShtDev:ff887524:97:05: sdfGetDevelopDevType(5694)(0x7000)
85347> ShootPreDe:ff886170:95:05: ->spsComplete ExtractStageJob(5694)

AF6FE> FrontShtDe:ff888f10:96:05: ProcessTwoInTwoOutJpegPath(R) End(5694)
AFE40> FrontShtDe:ff8893e4:96:05: sdfAllocateMemoryDevelopSuite(5694)
B07BC> FrontShtDe:ff88b008:96:05: sdsMem1ToJpegDevelop(5694)
B0805> FrontShtDe:ff8890a0:96:05: sdfExecuteMem1ToJpegPath(5694)
B0882> FrontShtDe:ff889290:96:05: ProcessTwoInTwoOutJpegPath(J) Start(5694)
DB219> FrontShtDe:ff8892c0:96:05: ProcessTwoInTwoOutJpegPath(J) End(5694)
DBF48> FrontShtDe:ff888418:96:05: sdfCompleteFrontDevelop(5694)
DBF96> FrontShtDe:ff888444:96:05: ->spsComplete ExtractStageJob(5694)
DC05D> RearShtDev:ff887f40:97:05: sdsYuvToDcfCompression(5694)(0,0)��(0x80000006)
DC0DF> RearShtDev:ff887f40:97:05: sdsYuvToDcfCompression(5694)(0,1)��(0x80000003)
DC128> RearShtDev:ff8869a8:97:05: sdsExecuteYuvToDcfPath(5694)
DC1F5> RearShtDev:ff886a4c:97:05: ProcessYuvToDcfPath(5694)
DC540> RearShtDev:ffa59524:16:03: [TTJ][151,5694,0] JPEG(160,120,8,0)
DC628> RearShtDev:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(4)
DC664> RearShtDev:ffa59888:16:03: [TTJ] START WR1:0x37f0094 WR2:0x37f0094
DC698> RearShtDev:ffa598b4:16:03: [TTJ] START RD1:0x1f10000 RD2:0x1f10000
DCC29> **INT-C8h*:ff9a42b0:00:02: [ENG] ReadDMACInterrupt(13)(0x2)(0)
DCC7B> **INT-C8h*:ffa59428:16:03: [TTJ][151,5694,0] Read1CompleteCBR
DCCD8> **INT-64h*:ffa59470:16:03: [TTJ][151,5694,0] JpCoreCompleteCBR( 0x338f )
DCD1C> **INT-5Ch*:ff9a407c:00:02: [ENG] WriteDMACInterrupt(4)(0x4)(2)
DCD5D> **INT-5Ch*:ffa593b8:16:03: [TTJ][151,5694,0] Write1Complete(POP)CBR
DCDBF> RearShtDev:ffa590d4:16:03: [TTJ] STOP WR1:0x37f3424 WR2:0x37f3424
DCE05> RearShtDev:ffa59100:16:03: [TTJ] STOP RD1:0x1f4c36c RD2:0x1f4c36c
DD0BD> RearShtDev:ff886a7c:97:05: (ISR)sdcCompleteYuvToDcfPath(5694)



DD59B> RearShtDev:ff8866c8:97:05: sdfPostNextStage(5694)
DD5DE> RearShtDev:ff8aacd0:00:03: [JOB] PostNextStage (ID = 5694, Class = 151, Pos = 5)
DD618> RearShtDev:ff81cc2c:89:03: bindDecideSize
DD7C2> RearShtDev:ff8aacd0:00:03: [JOB] PostNextStage (ID = 5694, Class = 151, Pos = 6)
DD811> RearShtDev:ff8ab3f4:00:03: [JOB] NotifyDevelopJob (ID = 5694, 0x400, 0x0 | 0x400)
DD846> RearShtDev:ff8ab44c:00:03: [JOB] NotifyDevelopJob : Free MEM1 (ID = 5694)


So I guess we are looking for ,
84B5D> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5694)
AF6FE> FrontShtDe:ff888f10:96:05: ProcessTwoInTwoOutJpegPath(R) End(5694)
B0882> FrontShtDe:ff889290:96:05: ProcessTwoInTwoOutJpegPath(J) Start(5694)
DB219> FrontShtDe:ff8892c0:96:05: ProcessTwoInTwoOutJpegPath(J) End(5694)


So what would be my next step ?


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 13, 2018, 08:06:10 AM
slices for full res cr2
ff1a6ef0 [JPCORE] JP62_SLCR1 %#lx
ff1a6f0c [JPCORE] JP62_SLCR2 %#lx

from 5d2 rom ,  If I'm not mistaken this is for sRAW 3872x2574 (422lossless) & (420lossless) is 2808x1872 sRAW1 but at 15bit jpeg lossless
would be nice to get 3872 422 lossless for video  :) and just crop it .
ff1a74cc [JPCORE] SetEncodeYuv422LosslessParam %d,%d
ff1a7628 [JPCORE] SetEncodeYuv420LosslessParam %d,%d


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 14, 2018, 07:46:28 AM
From my startup dm-0001_cr2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0001_cr2_2018_2_12.log) I can't find really anything to do with JPCore , but instead I think in the 5d2 it's called TTJ , no idea what it stand for except the "J" I think something to do with Jpeg stuff . I tried my best to break down the Shoot Capture , ProcessTwoInTwoOutLosslessPath Compressed Raw(TTJ->JPCore? (I think)) , Front shutter development , Rear shutter development  & shoot pre development processes from the dm-log
TTJ.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/TTJ.log) , ShootPreDe_5d2.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/ShootPreDe_5d2.log) , FrontShtDe.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/FrontShtDe.log) , RearShtDev.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/RearShtDev.log) & shoot_capture.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/shoot_capture.log) .
I found ProcessTwoInTwoOutLosslessPath process in all of the LOGS I posted of cr2 process mostly with TTJ see  below

68464> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x4000048 RD2:0x64d1864
9342C> **INT-5Ch*:ffa59370:16:03: [TTJ][150,5696,0] Write2CompleteCBR
93460> **INT-5Fh*:ff9a42b0:00:02: [ENG] ReadDMACInterrupt(10)(0x2)(0)
93497> **INT-5Fh*:ffa59428:16:03: [TTJ][150,5696,0] Read1CompleteCBR
934F3> **INT-64h*:ffa59470:16:03: [TTJ][150,5696,0] JpCoreCompleteCBR( 0x17e5e98 )
93540> **INT-5Bh*:ff9a407c:00:02: [ENG] WriteDMACInterrupt(3)(0x4)(2)
9357C> **INT-5Bh*:ffa593b8:16:03: [TTJ][150,5696,0] Write1Complete(POP)CBR
935E1> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x37edf14 WR2:0x1aea8248
9362C> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x64cf564 RD2:0x64d1864
94FFA> FrontShtDe:ffa59524:16:03: [TTJ][150,5696,0] JPEG(5616,3744,0,0)
9512B> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x37f0088 WR2:0x1f13504
9515E> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x40882fa RD2:0x64d1c64
95283> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[0]:0x800
952B2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[2]:0xc02
952E0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[4]:0x1004
9530C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[6]:0x1404
95338> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[8]:0x1805
95364> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[10]:0x1806
95392> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[12]:0x1c07
953C0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[14]:0x2007
953ED> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[16]:0x2408
9541A> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[18]:0x2809
95447> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[20]:0x2c0a
95475> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[22]:0x300b
954A3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[24]:0x340c
954D0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[26]:0x380d
954FF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[28]:0x3c0e
9552D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[30]:0x400f
95559> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[32]:0x4410
9558A> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[34]:0x4811
955B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[36]:0x4c12
955E4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[38]:0x5013
95611> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[40]:0x5414
9563F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[42]:0x5815
9566C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[44]:0x5c16
9569D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[46]:0x6017
956CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[48]:0x6418
956F7> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[50]:0x681a
95724> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[52]:0x6c1b
95754> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[54]:0x741c
95781> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[56]:0x7c1e
957AE> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[58]:0x8420
957DC> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[60]:0x9022
9580D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[62]:0x9825
95839> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[64]:0xa027
95866> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[66]:0xa829
95894> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[68]:0xb42b
958C4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[70]:0xbc2e
958F3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[72]:0xc430
95921> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[74]:0xd032
9594E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[76]:0xd835
9597C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[78]:0xe437
959A9> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[80]:0xec3a
959D7> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[82]:0xf83d
95A06> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[84]:0x1043f
95A33> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[86]:0x10c42
95A60> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[88]:0x11845
95A8D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[90]:0x12448
95ABF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[92]:0x1304b
95AED> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[94]:0x13c4d
95B1C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[96]:0x14851
95B4D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[98]:0x15454
95B7A> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[100]:0x16057
95BA8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[102]:0x1705a
95BD8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[104]:0x17c5d
95C06> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[106]:0x18c61
95C34> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[108]:0x19c65
95C63> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[110]:0x1b069
95C93> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[112]:0x1c46e
95CC3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[114]:0x1d874
95CF2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[116]:0x1ec79
95D22> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[118]:0x2047e
95D4F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[120]:0x21884
95D7D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[122]:0x23089
95DAB> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[124]:0x2488f
95DD9> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[126]:0x26095
95E07> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[128]:0x2789b
95E35> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[130]:0x290a1
95E62> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[132]:0x2aca8
95E90> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[134]:0x2c4ae
95EBF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[136]:0x2e0b5
95EEE> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[138]:0x2fcbc
95F1E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[140]:0x318c3
95F4D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[142]:0x334ca
95F80> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[144]:0x354d1
95FB0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[146]:0x374d9
95FE0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[148]:0x394e0
96012> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[150]:0x3c0ea
9603F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[152]:0x3ecf6
96071> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[154]:0x41d01
960A2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[156]:0x44d0d
960D0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[158]:0x48119
96100> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[160]:0x4b526
9612F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[162]:0x4ed34
9615C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[164]:0x52541
9618C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[166]:0x55d50
961BD> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[168]:0x5995f
961EB> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[170]:0x5d96e
96219> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[172]:0x6197e
96247> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[174]:0x65d8f
96275> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[176]:0x65d97
962A4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[178]:0x65d97
962D4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[180]:0x65d97
96305> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[182]:0x65d97
96335> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[184]:0x65d97
96364> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[186]:0x65d97
96393> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[188]:0x65d97
963C2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[190]:0x65d97
963F0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[192]:0x65d97
96420> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[194]:0x65d97
96455> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[196]:0x65d97
96484> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[198]:0x65d97
964B3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[200]:0x65d97
964E2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[202]:0x65d97
96510> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[204]:0x65d97
9653E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[206]:0x65d97
96570> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[208]:0x65d97
965A1> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[210]:0x65d97
965D0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[212]:0x65d97
96602> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[214]:0x65d97
96633> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[216]:0x65d97
96662> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[218]:0x65d97
96690> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[220]:0x65d97
966C1> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[222]:0x65d97
966EF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[224]:0x65d97
96720> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[226]:0x65d97
96752> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[228]:0x65d97
96781> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[230]:0x65d97
967AF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[232]:0x65d97
967DD> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[234]:0x65d97
9680E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[236]:0x65d97
9685C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[238]:0x65d97
96894> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[240]:0x65d97
968C6> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[242]:0x65d97
968F7> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[244]:0x65d97
96927> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[246]:0x65d97
96957> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[248]:0x65d97
96987> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[250]:0x65d97
969B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[252]:0x65d97
969E8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[254]:0x65d97
B1616> **INT-6Eh*:ffa593f0:16:03: [TTJ][150,5696,0] Read2CompleteCBR
BED79> **INT-5Ch*:ffa59370:16:03: [TTJ][150,5696,0] Write2CompleteCBR
BEDE7> **INT-5Fh*:ffa59428:16:03: [TTJ][150,5696,0] Read1CompleteCBR
BEE3D> **INT-64h*:ffa59470:16:03: [TTJ][150,5696,0] JpCoreCompleteCBR( 0x17f1f1 )
BEEBD> **INT-5Bh*:ffa593b8:16:03: [TTJ][150,5696,0] Write1Complete(POP)CBR
BEF21> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x396f27a WR2:0x1f48c94
BEF68> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x64b91fa RD2:0x64e9464
C0739> RearShtDev:ffa59524:16:03: [TTJ][151,5696,0] JPEG(160,120,8,0)
C085B> RearShtDev:ffa59888:16:03: [TTJ] START WR1:0x3970094 WR2:0x3970094
C0893> RearShtDev:ffa598b4:16:03: [TTJ] START RD1:0x1f10000 RD2:0x1f10000
C0E6F> **INT-C8h*:ffa59428:16:03: [TTJ][151,5696,0] Read1CompleteCBR
C0EC6> **INT-64h*:ffa59470:16:03: [TTJ][151,5696,0] JpCoreCompleteCBR( 0x3e9f )
C0F4B> **INT-5Ch*:ffa593b8:16:03: [TTJ][151,5696,0] Write1Complete(POP)CBR
C0FAB> RearShtDev:ffa590d4:16:03: [TTJ] STOP WR1:0x3973f34 WR2:0x3973f34
C0FF1> RearShtDev:ffa59100:16:03: [TTJ] STOP RD1:0x1f4c36c RD2:0x1f4c36c
E1BA3> FrontShtDe:ffa59694:16:03: [TTJ][150,5697,0] RAW(5792,3804,0,14)
E1C9B> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1afb00f0
E1CD4> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864
0CDA3> **INT-5Ch*:ffa59370:16:03: [TTJ][150,5697,0] Write2CompleteCBR
0CE0B> **INT-5Fh*:ffa59428:16:03: [TTJ][150,5697,0] Read1CompleteCBR
0CE64> **INT-64h*:ffa59470:16:03: [TTJ][150,5697,0] JpCoreCompleteCBR( 0x17f656f )
0CEE6> **INT-5Bh*:ffa593b8:16:03: [TTJ][150,5697,0] Write1Complete(POP)CBR
0CF7F> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x3982610 WR2:0x1b02c248
0CFD0> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x124cf564 RD2:0x124d1864
0EA6C> FrontShtDe:ffa59524:16:03: [TTJ][150,5697,0] JPEG(5616,3744,0,0)
0EB96> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x39840ac WR2:0x1f13504
0EBCE> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x100882fa RD2:0x124d1c64
0ECF8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[0]:0x800
0ED26> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[2]:0xc02
0ED50> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[4]:0x1004
0ED7B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[6]:0x1404
0EDA5> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[8]:0x1805
0EDD0> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[10]:0x1806
0EDFC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[12]:0x1c07
0EE27> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[14]:0x2007
0EE52> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[16]:0x2408
0EE7E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[18]:0x2809
0EEA8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[20]:0x2c0a
0EED3> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[22]:0x300b
0EEFF> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[24]:0x340c
0EF2C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[26]:0x380d
0EF57> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[28]:0x3c0e
0EF81> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[30]:0x400f
0EFAE> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[32]:0x4410
0EFD9> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[34]:0x4811
0F004> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[36]:0x4c12
0F031> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[38]:0x5013
0F05E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[40]:0x5414
0F08B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[42]:0x5815
0F0B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[44]:0x5c16
0F0E3> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[46]:0x6017
0F110> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[48]:0x6418
0F13E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[50]:0x681a
0F16A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[52]:0x6c1b
0F196> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[54]:0x741c
0F1C0> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[56]:0x7c1e
0F1EB> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[58]:0x8420
0F219> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[60]:0x9022
0F244> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[62]:0x9825
0F26F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[64]:0xa027
0F29C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[66]:0xa829
0F2C6> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[68]:0xb42b
0F2F8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[70]:0xbc2e
0F324> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[72]:0xc430
0F34F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[74]:0xd032
0F37F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[76]:0xd835
0F3AC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[78]:0xe437
0F3D9> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[80]:0xec3a
0F405> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[82]:0xf83d
0F432> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[84]:0x1043f
0F465> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[86]:0x10c42
0F492> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[88]:0x11845
0F4BF> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[90]:0x12448
0F4EC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[92]:0x1304b
0F51B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[94]:0x13c4d
0F549> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[96]:0x14851
0F576> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[98]:0x15454
0F5A4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[100]:0x16057
0F5D1> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[102]:0x1705a
0F5FF> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[104]:0x17c5d
0F62C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[106]:0x18c61
0F659> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[108]:0x19c65
0F687> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[110]:0x1b069
0F6B4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[112]:0x1c46e
0F6E4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[114]:0x1d874
0F711> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[116]:0x1ec79
0F740> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[118]:0x2047e
0F76D> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[120]:0x21884
0F79B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[122]:0x23089
0F7CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[124]:0x2488f
0F7F7> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[126]:0x26095
0F825> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[128]:0x2789b
0F877> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[130]:0x290a1
0F8AB> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[132]:0x2aca8
0F8DB> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[134]:0x2c4ae
0F909> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[136]:0x2e0b5
0F938> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[138]:0x2fcbc
0F966> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[140]:0x318c3
0F993> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[142]:0x334ca
0F9C2> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[144]:0x354d1
0F9F2> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[146]:0x374d9
0FA1F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[148]:0x394e0
0FA4F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[150]:0x3c0ea
0FA7D> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[152]:0x3ecf6
0FAAA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[154]:0x41d01
0FAD7> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[156]:0x44d0d
0FB04> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[158]:0x48119
0FB32> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[160]:0x4b526
0FB5F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[162]:0x4ed34
0FB8C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[164]:0x52541
0FBB8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[166]:0x55d50
0FBE6> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[168]:0x5995f
0FC15> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[170]:0x5d96e
0FC43> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[172]:0x6197e
0FC71> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[174]:0x65d8f
0FC9E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[176]:0x65d97
0FCCC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[178]:0x65d97
0FCFA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[180]:0x65d97
0FD29> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[182]:0x65d97
0FD57> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[184]:0x65d97
0FD86> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[186]:0x65d97
0FDB4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[188]:0x65d97
0FDE1> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[190]:0x65d97
0FE11> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[192]:0x65d97
0FE3F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[194]:0x65d97
0FE6E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[196]:0x65d97
0FE9A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[198]:0x65d97
0FECA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[200]:0x65d97
0FEF8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[202]:0x65d97
0FF26> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[204]:0x65d97
0FF56> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[206]:0x65d97
0FF83> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[208]:0x65d97
0FFB1> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[210]:0x65d97
0FFDD> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[212]:0x65d97
1000B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[214]:0x65d97
1003B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[216]:0x65d97
1006A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[218]:0x65d97
1009A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[220]:0x65d97
100CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[222]:0x65d97
100F8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[224]:0x65d97
10129> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[226]:0x65d97
10158> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[228]:0x65d97
10187> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[230]:0x65d97
101B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[232]:0x65d97
101E8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[234]:0x65d97
10216> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[236]:0x65d97
1024A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[238]:0x65d97
1027B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[240]:0x65d97
102A9> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[242]:0x65d97
102DA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[244]:0x65d97
1030B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[246]:0x65d97
10338> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[248]:0x65d97
1036A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[250]:0x65d97
1039B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[252]:0x65d97
103CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[254]:0x65d97
2B00B> **INT-6Eh*:ffa593f0:16:03: [TTJ][150,5697,0] Read2CompleteCBR
38742> **INT-5Fh*:ffa59428:16:03: [TTJ][150,5697,0] Read1CompleteCBR
387AD> **INT-5Ch*:ffa59370:16:03: [TTJ][150,5697,0] Write2CompleteCBR
38803> **INT-64h*:ffa59470:16:03: [TTJ][150,5697,0] JpCoreCompleteCBR( 0x17e47a )
3888A> **INT-5Bh*:ffa593b8:16:03: [TTJ][150,5697,0] Write1Complete(POP)CBR
388E7> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x3b02526 WR2:0x1f48c94
38928> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x124b91fa RD2:0x124e9464
3A0E6> RearShtDev:ffa59524:16:03: [TTJ][151,5697,0] JPEG(160,120,8,0)
3A200> RearShtDev:ffa59888:16:03: [TTJ] START WR1:0x3b040b8 WR2:0x3b040b8
3A237> RearShtDev:ffa598b4:16:03: [TTJ] START RD1:0x1f10000 RD2:0x1f10000
3A81E> **INT-C8h*:ffa59428:16:03: [TTJ][151,5697,0] Read1CompleteCBR
3A87E> **INT-64h*:ffa59470:16:03: [TTJ][151,5697,0] JpCoreCompleteCBR( 0x3ec8 )
3A904> **INT-5Ch*:ffa593b8:16:03: [TTJ][151,5697,0] Write1Complete(POP)CBR
3A966> RearShtDev:ffa590d4:16:03: [TTJ] STOP WR1:0x3b07f80 WR2:0x3b07f80
3A9A7> RearShtDev:ffa59100:16:03: [TTJ] STOP RD1:0x1f4c36c RD2:0x1f4c36c

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 14, 2018, 07:58:57 AM
After reading @nikfreak post for 7D here (https://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741) I see the FrontShtDe may be more important

CEE83> FrontShtDe:ff889d28:96:05: sdsInit
CEE83> FrontShtDe:ff889d28:96:05: sdsInit
66A99> FrontShtDe:ff88a0fc:96:05: ->sdsPostJob InsertStageJob(5696)
66AE7> FrontShtDe:ff8aacd0:00:03: [JOB] PostNextStage (ID = 5696, Class = 150, Pos = 4)
66B53> FrontShtDe:ff88a170:96:05: sdsPostJob Start(5696)
66B8C> FrontShtDe:ff88a19c:96:05: sdfGetDevelopDevType(5696)(0x7000)
66BD5> FrontShtDe:ff88a210:96:05: ENABLE(Poster:0)(Ohyear:0)
66C14> FrontShtDe:ff8893e4:96:05: sdfAllocateMemoryDevelopSuite(5696)
674DF> FrontShtDe:ff88ad78:96:05: sdsMem1ToRawCompression
67EA5> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5696)
67F2E> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5696)��(SemOK)
67F6A> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5696)
6801D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
68060> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
68094> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
680C0> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
680E9> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
68111> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
6813F> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
6816C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
68199> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
681C8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
68257> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
6828A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
682B4> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
682E1> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
68327> FrontShtDe:ffa59694:16:03: [TTJ][150,5696,0] RAW(5792,3804,0,14)
683E3> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
68428> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1ae2c0f0
68464> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x4000048 RD2:0x64d1864
68526> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
935E1> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x37edf14 WR2:0x1aea8248
9362C> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x64cf564 RD2:0x64d1864
936D1> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
9371F> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93755> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93782> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
937AF> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
937DC> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
9380A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93838> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93867> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93893> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
938BD> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
938E3> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93909> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
93958> FrontShtDe:ff888f10:96:05: ProcessTwoInTwoOutJpegPath(R) End(5696)
94081> FrontShtDe:ff8893e4:96:05: sdfAllocateMemoryDevelopSuite(5696)
94965> FrontShtDe:ff88b008:96:05: sdsMem1ToJpegDevelop(5696)
949FE> FrontShtDe:ff8890a0:96:05: sdfExecuteMem1ToJpegPath(5696)
94A85> FrontShtDe:ff889290:96:05: ProcessTwoInTwoOutJpegPath(J) Start(5696)
94B7D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94BC8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94BF8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94C27> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94C56> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94C83> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94CAC> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94CD5> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94D04> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94D32> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94D5D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94D87> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94DB0> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94DDA> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94E00> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94E2C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94E57> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94E81> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94EAE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94ED7> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94F01> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94F2D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94F59> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94F84> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
94FFA> FrontShtDe:ffa59524:16:03: [TTJ][150,5696,0] JPEG(5616,3744,0,0)
950EC> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
9512B> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x37f0088 WR2:0x1f13504
9515E> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x40882fa RD2:0x64d1c64
95283> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[0]:0x800
952B2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[2]:0xc02
952E0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[4]:0x1004
9530C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[6]:0x1404
95338> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[8]:0x1805
95364> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[10]:0x1806
95392> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[12]:0x1c07
953C0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[14]:0x2007
953ED> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[16]:0x2408
9541A> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[18]:0x2809
95447> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[20]:0x2c0a
95475> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[22]:0x300b
954A3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[24]:0x340c
954D0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[26]:0x380d
954FF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[28]:0x3c0e
9552D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[30]:0x400f
95559> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[32]:0x4410
9558A> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[34]:0x4811
955B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[36]:0x4c12
955E4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[38]:0x5013
95611> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[40]:0x5414
9563F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[42]:0x5815
9566C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[44]:0x5c16
9569D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[46]:0x6017
956CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[48]:0x6418
956F7> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[50]:0x681a
95724> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[52]:0x6c1b
95754> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[54]:0x741c
95781> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[56]:0x7c1e
957AE> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[58]:0x8420
957DC> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[60]:0x9022
9580D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[62]:0x9825
95839> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[64]:0xa027
95866> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[66]:0xa829
95894> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[68]:0xb42b
958C4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[70]:0xbc2e
958F3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[72]:0xc430
95921> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[74]:0xd032
9594E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[76]:0xd835
9597C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[78]:0xe437
959A9> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[80]:0xec3a
959D7> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[82]:0xf83d
95A06> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[84]:0x1043f
95A33> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[86]:0x10c42
95A60> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[88]:0x11845
95A8D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[90]:0x12448
95ABF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[92]:0x1304b
95AED> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[94]:0x13c4d
95B1C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[96]:0x14851
95B4D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[98]:0x15454
95B7A> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[100]:0x16057
95BA8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[102]:0x1705a
95BD8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[104]:0x17c5d
95C06> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[106]:0x18c61
95C34> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[108]:0x19c65
95C63> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[110]:0x1b069
95C93> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[112]:0x1c46e
95CC3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[114]:0x1d874
95CF2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[116]:0x1ec79
95D22> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[118]:0x2047e
95D4F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[120]:0x21884
95D7D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[122]:0x23089
95DAB> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[124]:0x2488f
95DD9> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[126]:0x26095
95E07> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[128]:0x2789b
95E35> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[130]:0x290a1
95E62> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[132]:0x2aca8
95E90> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[134]:0x2c4ae
95EBF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[136]:0x2e0b5
95EEE> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[138]:0x2fcbc
95F1E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[140]:0x318c3
95F4D> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[142]:0x334ca
95F80> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[144]:0x354d1
95FB0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[146]:0x374d9
95FE0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[148]:0x394e0
96012> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[150]:0x3c0ea
9603F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[152]:0x3ecf6
96071> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[154]:0x41d01
960A2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[156]:0x44d0d
960D0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[158]:0x48119
96100> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[160]:0x4b526
9612F> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[162]:0x4ed34
9615C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[164]:0x52541
9618C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[166]:0x55d50
961BD> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[168]:0x5995f
961EB> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[170]:0x5d96e
96219> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[172]:0x6197e
96247> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[174]:0x65d8f
96275> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[176]:0x65d97
962A4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[178]:0x65d97
962D4> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[180]:0x65d97
96305> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[182]:0x65d97
96335> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[184]:0x65d97
96364> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[186]:0x65d97
96393> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[188]:0x65d97
963C2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[190]:0x65d97
963F0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[192]:0x65d97
96420> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[194]:0x65d97
96455> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[196]:0x65d97
96484> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[198]:0x65d97
964B3> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[200]:0x65d97
964E2> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[202]:0x65d97
96510> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[204]:0x65d97
9653E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[206]:0x65d97
96570> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[208]:0x65d97
965A1> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[210]:0x65d97
965D0> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[212]:0x65d97
96602> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[214]:0x65d97
96633> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[216]:0x65d97
96662> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[218]:0x65d97
96690> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[220]:0x65d97
966C1> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[222]:0x65d97
966EF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[224]:0x65d97
96720> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[226]:0x65d97
96752> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[228]:0x65d97
96781> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[230]:0x65d97
967AF> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[232]:0x65d97
967DD> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[234]:0x65d97
9680E> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[236]:0x65d97
9685C> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[238]:0x65d97
96894> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[240]:0x65d97
968C6> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[242]:0x65d97
968F7> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[244]:0x65d97
96927> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[246]:0x65d97
96957> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[248]:0x65d97
96987> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[250]:0x65d97
969B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[252]:0x65d97
969E8> FrontShtDe:ffa59960:16:01: [TTJ][150,5696,0] PPR Wr[254]:0x65d97
96A87> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
96C37> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
96D27> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
BEF21> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x396f27a WR2:0x1f48c94
BEF68> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x64b91fa RD2:0x64e9464
BF01A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF065> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF099> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF0C9> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF0F3> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF11E> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF14A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF175> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF1A7> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF1D5> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF200> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF22C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF25B> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF285> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF2AE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF2D6> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF305> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF331> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF358> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF381> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF3AC> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF3D8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF403> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
BF487> FrontShtDe:ff8892c0:96:05: ProcessTwoInTwoOutJpegPath(J) End(5696)
BFB89> FrontShtDe:ff887068:97:05: (ISR)sdcCompleteFrontDevelop::DisplayJpegComplete
C01E6> FrontShtDe:ff888418:96:05: sdfCompleteFrontDevelop(5696)
C022C> FrontShtDe:ff888444:96:05: ->spsComplete ExtractStageJob(5696)
E0226> FrontShtDe:ff88a0fc:96:05: ->sdsPostJob InsertStageJob(5697)
E0275> FrontShtDe:ff8aacd0:00:03: [JOB] PostNextStage (ID = 5697, Class = 150, Pos = 4)
E02D9> FrontShtDe:ff88a170:96:05: sdsPostJob Start(5697)
E030D> FrontShtDe:ff88a19c:96:05: sdfGetDevelopDevType(5697)(0x7000)
E0351> FrontShtDe:ff88a210:96:05: ENABLE(Poster:0)(Ohyear:0)
E038C> FrontShtDe:ff8893e4:96:05: sdfAllocateMemoryDevelopSuite(5697)
E0D25> FrontShtDe:ff88ad78:96:05: sdsMem1ToRawCompression
E178E> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5697)
E1816> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5697)��(SemOK)
E1858> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5697)
E1907> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E194D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E197D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E19A8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E19D2> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E19FB> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1A2B> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1A5A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1A8A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1AB8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1AE5> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1B0F> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1B38> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1B62> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
E1BA3> FrontShtDe:ffa59694:16:03: [TTJ][150,5697,0] RAW(5792,3804,0,14)
E1C5C> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
E1C9B> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1afb00f0
E1CD4> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864
E1D97> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
0CF7F> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x3982610 WR2:0x1b02c248
0CFD0> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x124cf564 RD2:0x124d1864
0D080> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D0CF> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D107> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D135> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D194> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D1C8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D1F7> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D224> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D253> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D280> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D2A9> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D2CE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D2F5> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0D344> FrontShtDe:ff888f10:96:05: ProcessTwoInTwoOutJpegPath(R) End(5697)
0DBBA> FrontShtDe:ff8893e4:96:05: sdfAllocateMemoryDevelopSuite(5697)
0E454> FrontShtDe:ff88b008:96:05: sdsMem1ToJpegDevelop(5697)
0E49D> FrontShtDe:ff8890a0:96:05: sdfExecuteMem1ToJpegPath(5697)
0E51C> FrontShtDe:ff889290:96:05: ProcessTwoInTwoOutJpegPath(J) Start(5697)
0E615> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E65A> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E689> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E6B1> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E6DE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E704> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E731> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E75E> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E78B> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E7B7> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E7E2> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E80B> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E836> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E85C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E883> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E8AC> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E8D8> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E902> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E92B> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E953> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E97E> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E9AB> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E9D2> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0E9FC> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
0EA6C> FrontShtDe:ffa59524:16:03: [TTJ][150,5697,0] JPEG(5616,3744,0,0)
0EB58> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
0EB96> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x39840ac WR2:0x1f13504
0EBCE> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x100882fa RD2:0x124d1c64
0ECF8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[0]:0x800
0ED26> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[2]:0xc02
0ED50> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[4]:0x1004
0ED7B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[6]:0x1404
0EDA5> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[8]:0x1805
0EDD0> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[10]:0x1806
0EDFC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[12]:0x1c07
0EE27> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[14]:0x2007
0EE52> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[16]:0x2408
0EE7E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[18]:0x2809
0EEA8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[20]:0x2c0a
0EED3> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[22]:0x300b
0EEFF> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[24]:0x340c
0EF2C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[26]:0x380d
0EF57> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[28]:0x3c0e
0EF81> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[30]:0x400f
0EFAE> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[32]:0x4410
0EFD9> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[34]:0x4811
0F004> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[36]:0x4c12
0F031> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[38]:0x5013
0F05E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[40]:0x5414
0F08B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[42]:0x5815
0F0B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[44]:0x5c16
0F0E3> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[46]:0x6017
0F110> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[48]:0x6418
0F13E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[50]:0x681a
0F16A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[52]:0x6c1b
0F196> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[54]:0x741c
0F1C0> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[56]:0x7c1e
0F1EB> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[58]:0x8420
0F219> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[60]:0x9022
0F244> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[62]:0x9825
0F26F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[64]:0xa027
0F29C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[66]:0xa829
0F2C6> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[68]:0xb42b
0F2F8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[70]:0xbc2e
0F324> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[72]:0xc430
0F34F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[74]:0xd032
0F37F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[76]:0xd835
0F3AC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[78]:0xe437
0F3D9> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[80]:0xec3a
0F405> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[82]:0xf83d
0F432> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[84]:0x1043f
0F465> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[86]:0x10c42
0F492> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[88]:0x11845
0F4BF> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[90]:0x12448
0F4EC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[92]:0x1304b
0F51B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[94]:0x13c4d
0F549> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[96]:0x14851
0F576> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[98]:0x15454
0F5A4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[100]:0x16057
0F5D1> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[102]:0x1705a
0F5FF> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[104]:0x17c5d
0F62C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[106]:0x18c61
0F659> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[108]:0x19c65
0F687> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[110]:0x1b069
0F6B4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[112]:0x1c46e
0F6E4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[114]:0x1d874
0F711> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[116]:0x1ec79
0F740> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[118]:0x2047e
0F76D> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[120]:0x21884
0F79B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[122]:0x23089
0F7CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[124]:0x2488f
0F7F7> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[126]:0x26095
0F825> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[128]:0x2789b
0F877> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[130]:0x290a1
0F8AB> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[132]:0x2aca8
0F8DB> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[134]:0x2c4ae
0F909> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[136]:0x2e0b5
0F938> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[138]:0x2fcbc
0F966> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[140]:0x318c3
0F993> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[142]:0x334ca
0F9C2> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[144]:0x354d1
0F9F2> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[146]:0x374d9
0FA1F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[148]:0x394e0
0FA4F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[150]:0x3c0ea
0FA7D> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[152]:0x3ecf6
0FAAA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[154]:0x41d01
0FAD7> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[156]:0x44d0d
0FB04> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[158]:0x48119
0FB32> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[160]:0x4b526
0FB5F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[162]:0x4ed34
0FB8C> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[164]:0x52541
0FBB8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[166]:0x55d50
0FBE6> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[168]:0x5995f
0FC15> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[170]:0x5d96e
0FC43> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[172]:0x6197e
0FC71> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[174]:0x65d8f
0FC9E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[176]:0x65d97
0FCCC> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[178]:0x65d97
0FCFA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[180]:0x65d97
0FD29> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[182]:0x65d97
0FD57> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[184]:0x65d97
0FD86> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[186]:0x65d97
0FDB4> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[188]:0x65d97
0FDE1> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[190]:0x65d97
0FE11> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[192]:0x65d97
0FE3F> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[194]:0x65d97
0FE6E> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[196]:0x65d97
0FE9A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[198]:0x65d97
0FECA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[200]:0x65d97
0FEF8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[202]:0x65d97
0FF26> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[204]:0x65d97
0FF56> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[206]:0x65d97
0FF83> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[208]:0x65d97
0FFB1> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[210]:0x65d97
0FFDD> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[212]:0x65d97
1000B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[214]:0x65d97
1003B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[216]:0x65d97
1006A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[218]:0x65d97
1009A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[220]:0x65d97
100CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[222]:0x65d97
100F8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[224]:0x65d97
10129> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[226]:0x65d97
10158> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[228]:0x65d97
10187> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[230]:0x65d97
101B8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[232]:0x65d97
101E8> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[234]:0x65d97
10216> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[236]:0x65d97
1024A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[238]:0x65d97
1027B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[240]:0x65d97
102A9> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[242]:0x65d97
102DA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[244]:0x65d97
1030B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[246]:0x65d97
10338> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[248]:0x65d97
1036A> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[250]:0x65d97
1039B> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[252]:0x65d97
103CA> FrontShtDe:ffa59960:16:01: [TTJ][150,5697,0] PPR Wr[254]:0x65d97
1046B> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
1061E> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
1070C> FrontShtDe:00096178:00:00: *** register_interrupt("BLTDMA", 0x2f, 0xff872820, 0x29794), from ff872904
388E7> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x3b02526 WR2:0x1f48c94
38928> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x124b91fa RD2:0x124e9464
389E2> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38A2C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38A62> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38A92> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38ABE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38AE9> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38B14> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38B41> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38B6E> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38B9C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38BC9> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38BF3> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38C1B> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38C44> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38C6F> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38C96> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38CC2> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38CEB> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38D16> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38D42> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38D6D> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38D95> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38DBF> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
38E43> FrontShtDe:ff8892c0:96:05: ProcessTwoInTwoOutJpegPath(J) End(5697)
39571> FrontShtDe:ff887068:97:05: (ISR)sdcCompleteFrontDevelop::DisplayJpegComplete
39BA1> FrontShtDe:ff888418:96:05: sdfCompleteFrontDevelop(5697)
39BF0> FrontShtDe:ff888444:96:05: ->spsComplete ExtractStageJob(5697)

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 16, 2018, 12:40:42 AM
Starting define 5d2 in "lossless.c ,
Looking for "Mem1toRaw" (picture size) and in my log I have two (Mem1toRaw)
67EA5> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5696)
67F2E> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5696)��(SemOK)

So is the first one I need ?
ff888cc0 

67F6A> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5696)

ff888ee0 I think maybe ?



if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF888CC0;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF888EE0;  /* called right after ProcessTwoInTwoOutLosslessPath(R) Start; */
                                                /* calls [TTL] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF******;  /* RegisterTwoInTwoOutLosslessPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF******;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF******;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF******;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF******;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


This what have found so far , once I found other 5 I'll post a update and test if it works I'll post a test build but just for 5d2 as that's what I have (but don't hold your breath  :D  )
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ilia3101 on February 16, 2018, 12:50:11 AM
Ok this is starting to smell exciting again :o If the mk2 goes lossless... that will be really exciting

I can help with/test whatever tomorrow. And remember: upload the code if you have something :)

@a1ex How does it look to you?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 16, 2018, 04:43:51 AM

if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF888CC0;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF888EE0;  /* called right after ProcessTwoInTwoOutLosslessPath(R) Start; */
                                                /* calls [TTL] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF******;  /* RegisterTwoInTwoOutLosslessPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF******;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF******;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF******;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF******;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }



Reading this I just realized there's no "TTL" in Digic 4 just "TTJ" at least I could not finds it , not being the greatest coder but instead of "TTL"  I thinking "TTJ"
as that where I was find address , so this is how I think it should bee( but could be wrong)
if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 5D2 2.1.2 */
        TTJ_SetArgs     = (void *) 0xFF888CC0;  /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTJ_Prepare     = (void *) 0xFF888EE0;  /* called right after ProcessTwoInTwoOutLosslessPath(R) Start; */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTJ_RegisterCBR = (void *) 0xFF******;  /* RegisterTwoInTwoOutLosslessPathCompleteCBR */
        TTJ_SetFlags    = (void *) 0xFF******;  /* called next, with PictureType as arguments */
        TTJ_Start       = (void *) 0xFF******;  /* called next; starts the EDmac transfers */
        TTJ_Stop        = (void *) 0xFF******;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF******;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }
 

What do you @a1ex ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on February 16, 2018, 08:29:23 AM
To be taken with a grain of salt--this is what I got doing pattern matching on the disassembly:

    if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutJpegPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF1BEB18;  /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF259B58;  /* called right after ProcessTwoInTwoOutJpegPath */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF2DDA1C;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF0AA9D4;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF259E3C;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF259E74;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF259EAC;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


You can't just substitute TTJ for TTL for the variable names. The comments might not be accurate but in some cases you can look up the strings in the disassembly like "ProcessTwoInTwoOutJpegPath" -- note that there's also the DecodeLosslessPath stuff at the end of lossless.c that you still need to find. I got lost looking for it in the 5D2 disassembly.

Oh yeah - then there's the "resources[]" which are probably different on the 5D2 from the choices that are in lossless.c.

By the way, the 5D2 ROM is a bit quirky. My disassembly starts at 0xff000000 and it is mirrored at 0xff800000. That means you can use 0xFF1BEB18 or 0xFF9BEB18 interchangeably for TTL_SetArgs.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 17, 2018, 09:19:59 AM
Thank @dfort , for setting me straight  ;D
I check in disassembly and there is "TTL" Argument
so I'm back to this now.

if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutLosslessPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF9BECF4;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF088FD8;  /* called right after ProcessTwoInTwoOutLosslessPath(R) Start; */
                                                /* calls [TTL] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF258F8C; /* RegisterTwoInTwoOutLosslessPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF026AB0;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF******;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF9F3410;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF25907C;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


Now this looks better , for the my disassembly

****I used the first address**** ff9becf4
ff9becf4: STRING:  '->Unknown PictureSize(Mem1ToRaw)'
ff888f8c: STRING:  '->Unknown PictureSize(Mem1ToRaw)'
ff888e44: e28f2d05 add r2, pc, #320 ; *'->Unknown PictureSize(Mem1ToRaw)'
ff1becf4: STRING:  '->Unknown PictureSize(Mem1ToRaw)'
ff1bec2c: e28f20c0 add r2, pc, #192 ; *'->Unknown PictureSize(Mem1ToRaw)'
ff088f8c: STRING:  '->Unknown PictureSize(Mem1ToRaw)'
ff088e44: e28f2d05 add r2, pc, #320 ; *'->Unknown PictureSize(Mem1ToRaw)'

Maybe not so , I also found this but it's not the "Mem1toRaw" so just guess which one ?
ff1bc1d8: e28f0f56 add r0, pc, #344 ; *'[Picture Size?( 0(L) 2(S) 5(M) 9(RAW))] --- '
ff1bc338: STRING:  '[Picture Size?( 0(L) 2(S) 5(M) 9(RAW))] --- '
ff9bc1d8: e28f0f56 add r0, pc, #344 ; *'[Picture Size?( 0(L) 2(S) 5(M) 9(RAW))] --- '


ff258f8c: STRING:  '[TTJ][%d,%d,%d] RegisterTwoInTwoOutJpegPathCompleteCBR'
ff25907c: e28f2f8b add r2, pc, #556 ; *'[TTJ][%d,%d,%d] UnLockEngineResources(%d)'
****Now sure about this one****
ff9f3410: STRING:  'SSS'

ff088fd8: STRING:  'ProcessTwoInTwoOutJpegPath(R) Start(%d)'


Ok , I'll continue this tomorrow .
Edit:
ff026ab0: STRING:  '(PictType123 = %#x, %#x, %#x)'
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 18, 2018, 12:34:48 AM
@dfort , I've being checking your addresses for 5d2 and all go a "Pointer" then to  a different address in the .dis file , how are you looking these up ?
I couldn't find them in the string.htm at all. Just wondering if I'm doing this right , the ones I found seem to line up with the description in the ML code.
I use the 5d2.212.dis in MS Visual Studio Code (https://code.visualstudio.com/download)  (free) and search for different variable to find addresses .
Visual Studio is really cool (supported in Win 7,8,10, Mac OS 10.9+ & Linux OS , it look for mistake/problems & make suggestions in the code , add extension
(clang etc. ... )when need them , can upset for "Github" (maybe bitbucket too, no sure)to clone code and even (if I'm not mistaken)
I can setup the tool chain to compile ml , but I haven got to it yet -- makes working with code a breeze .


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on February 18, 2018, 03:57:02 AM
Quote from: reddeercity on February 18, 2018, 12:34:48 AM
I've being checking your addresses for 5d2 and all go a "Pointer" then to  a different address in the .dis file , how are you looking these up ?

I was going to ask you the same question, how are you looking these up? Are you checking the addresses you found against disassemblies for the cameras that already have lossless compression working?

Your logic seems to make sense:

ML code
        /* ProcessTwoInTwoOutLosslessPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF9BECF4;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */


Looking up that address:
Quoteff9bec2c:    redacted    add   r2, pc, #192   ; ff9becf4: (redacted)  *"->Unknown PictureSize(Mem1ToRaw)"

So if you look up that same line in the 5D3.113 disassembly you would come up with 0xFF3234E4 for TTL_SetArgs but---

        /* ProcessTwoInTwoOutLosslessPath, 5D3 1.1.3 */
        TTL_SetArgs     = (void *) 0xFF32330C;  /* fills TTL_Args struct; PictureSize(Mem1ToRaw) */


Look up that address in the 5D3.113 disassembly and you'll see this:

loc_ff32330c: ; 10 refs

That's the start of a function. Now look up the address that I found in the 5D2 disassembly:

loc_ff1beb18: ; 5 refs

This won't make any sense unless you can flip between the 5D2 and the 5D3 disassemblies. It is almost a perfect match. Keep scrolling through that function and just before you get to the next function (2 refs on both 5D2 and 5D3) you should have an "ah ha" moment.

It doesn't always work out this way but basically the code that does the job on one camera should "look" pretty much the same as the code that does the same job on another camera.

BTW -- Never heard of MS Visual Studio Code so I downloaded it and took a look at it. I might be missing something but I don't see how it can be useful for working with disassembled ARM code. There are some powerful apps out there but I'm just using the disassembly.pl script as recommended in Tutorial: finding stubs (https://www.magiclantern.fm/forum/index.php?topic=12177.0) by a1ex and a text editor.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 18, 2018, 07:32:49 AM
Quote from: dfort on February 18, 2018, 03:57:02 AM
BTW -- Never heard of MS Visual Studio Code so I downloaded it and took a look at it. I might be missing something but I don't see how it can be useful for working with disassembled ARM code. There are some powerful apps out there but I'm just using the disassembly.pl script as recommended in Tutorial: finding stubs (https://www.magiclantern.fm/forum/index.php?topic=12177.0) by a1ex and a text editor.
No , I use it to search the disassemble .dis and strings.htm file for address & as a code editor , it's very fast in searching .
(https://thumb.ibb.co/bOHuWS/Visual_Studio_Coder_2.png) (https://ibb.co/bOHuWS)  (https://thumb.ibb.co/hEk9y7/Visual_Studio_Coder_1.png) (https://ibb.co/hEk9y7) 
(https://preview.ibb.co/bvWhJ7/Visual_Studio_Coder_small.png) (https://ibb.co/gCsBBS)

Quote from: dfort on February 18, 2018, 03:57:02 AM
I was going to ask you the same question, how are you looking these up? Are you checking the addresses you found against disassemblies for the cameras that already have lossless compression working?

No , I just looked at the code and started to search by "key word statements" e.g. mem1toraw as you can see in the images & it searches all files loaded (5d2.212.dis , lossless.c , strings.htm)
Then I would read the disassemble , and see what was going , which gives me a very,  very basic understanding , the more I read the more I discovery  :D



Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 18, 2018, 10:19:27 PM
Thanks for the help @dfort , sent you a pm
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ilia3101 on February 18, 2018, 11:18:43 PM
Can confirm VSCode is only good thing Microsoft ever made.

I'm a bit confused as to how dfort found the addresses it seems more complex than just finding strings in the disassembly :( Can't imagine I'll ever figure this out.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 19, 2018, 07:14:44 AM
I must have a bad 5d2 disassembly or bad rom1, I can't find
loc_ff1beb18: ; 5 refs
I have not problem finding
loc_ff32330c: ; 10 refs
and all the other stuff make sense now , I can't even find any phase/statement , etc. ... about "loc_" or "5refs"
in my 5d2 disassembly , but I did find
ff1beb18: e92d41f0 push {r4, r5, r6, r7, r8, lr}
which is very close to 5d3-1.1.3
ff1beb18: e92d4070 push {r4, r5, r6, lr}
QuoteThis won't make any sense unless you can flip between the 5D2 and the 5D3 disassemblies.
It is almost a perfect match. Keep scrolling through that function and just before you get to the next function (2 refs on both 5D2 and 5D3) you should have an "ah ha" moment.
I wish could say that , but sadly no there must be a problem with my disassembly , or is there a state that the camera needs to be in .e.g. "Photo Mode" or "Video Mode"
for compressed raw ? to save to the ROM ,  does that make a different ? I think I was always in video mode so maybe those routines/are not present .

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on February 19, 2018, 09:03:32 AM
You might have used a different disassembler. I used the perl script from the Tutorial: finding stubs (https://www.magiclantern.fm/forum/index.php?topic=12177.0). Hint, just search for the address, not the entire line. Yours might be formatted differently.

Quote from: reddeercity on February 19, 2018, 07:14:44 AM
I think I was always in video mode so maybe those routines/are not present .

I doubt that. The firmware should have everything you need no matter what camera settings were active when the dump was saved. It does make a difference in QEMU. Just checked out the 5D2 in QEMU and it looks like the emulation has gotten a lot better:

(https://farm5.staticflickr.com/4671/39643340124_0cbf1fafe9.jpg) (https://flic.kr/p/23p9tSd)

Though I can't seem to be able to get to the ML menu and this probably won't help you figure out lossless compression so -- never mind!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: theBilalFakhouri on February 19, 2018, 12:48:18 PM
Hello a1ex

at 23.976fps 1736x976 resolution - ISO 100:

14-bit uncompressed = 14-bit lossless (with compression ratio of ~57%).
12-bit uncompressed ≠ 12-bit lossless (negative gain fake lossless with enhanced compression ratio of ~50% compared to native 14 bit ).
10-bit uncompressed ≠ 11-8 bit lossless (negative gain fake lossless with enhanced compression ratio of ~45% compared to native 14 bit).

So the 67.7mb/s became ~38.4mb/s (in 14-bit).

Now the 10-bit uncompressed has write speed of 48.4mb/s ,if it was native 10bit lossless, Will we have same as 57% compression ratio
of 14-bit/14-bit lossless compared to 10-bit/10-bit native lossless?

The 48.4mb/s (10-bit) will become ~27mb/s (10-bit native lossless) instead of the current 11-bit lossless with write speed of ~32.4mb/s ?

Or it's just the same ?

I hope I talked well  :P
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 20, 2018, 01:33:34 AM
Well guess what
******5D2.2.1.2******
loc_ff1beb18: ; 5 refs

found it now , thanks @dfort  ;D
(https://thumb.ibb.co/iRfmfn/disassembly5d2_1.png) (https://ibb.co/iRfmfn)

*****5D3.1.1.3******
loc_ff32330c: ; 10 refs


(https://thumb.ibb.co/ns1pY7/disassembly5_D3_113.png) (https://ibb.co/ns1pY7)
AH Ha !! 
This should help me advance this faster for D4
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 20, 2018, 08:42:09 AM
****5d2.2.1.2****from dfort****
TTL_RegisterCBR = (void *) 0xFF2DDA1C;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */

Checking address FF2DDA1C  in 5d2 and does not really add up like the other one from 5d3

(https://image.ibb.co/caQrXc/disassembly5d2_3.png) (https://imgbb.com/)

This the way it is for digic5
****5d3.1.1.3****
TTL_RegisterCBR = (void *) 0xFF3D3774;  /* RegisterTwoInTwoOutLosslessPathCompleteCBR */


(https://image.ibb.co/k9EUJH/disassembly5d3_4.png) (https://imgbb.com/)

After some time searching etc. .... I think I found the pattern for 5d2 but it has a different name/operation  then digic 5/5d3
I believe  FF023288 is the one , but not 100% sure
TTL_RegisterCBR = (void *) 0xFF023288;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */

(https://image.ibb.co/gtP1Xc/disassembly5d2_2.png) (https://imgbb.com/)
Looks the same (almost) expect for the address , how I'm I convinced ? I searched "e92d401c" (there was 150  :o , lucky guess)
and looked for something to do with  "TwoInTwoOutJpeg" I didn't find it but I found something that's close (I hope)
*"PreDevelop\PreDevelop.c"
the digic5 has
./DevelopPath/TwoInTwoOutLosslessPath.c
I will not know for sure 100% until I test but looks like maybe .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on February 20, 2018, 02:07:55 PM
Quote from: reddeercity on February 20, 2018, 08:42:09 AM
After some time searching etc. .... I think I found the pattern for 5d2 but it has a different name/operation  then digic 5/5d3
I believe  FF023288 is the one , but not 100% sure
TTL_RegisterCBR = (void *) 0xFF023288;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */

Good you're checking my work. I was way off but take a look at this one, it seems to be a better match than what you found:

TTL_RegisterCBR = (void *) 0xFF258E70;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */

Now this is where we're at:

    if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutJpegPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF1BEB18;  /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF259B58;  /* called right after ProcessTwoInTwoOutJpegPath */
                                                /* calls [TTJ] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF258E70;  /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF0AA9D4;  /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF259E3C;  /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF259E74;  /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF259EAC;  /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


When I first looked at this I thought maybe I made a mistake with TTL_SetArgs and TTL_SetFlags because those addresses are quite a distance from the rest of the addresses and the comments seem to indicate that this is a block of code with the functions living very close to each other. Let's do a different kind of pattern matching and look at some other cameras. Do their TTL_SetArgs and TTL_SetFlags functions also come quite a bit earlier in the disassembly?

    if (is_camera("EOSM", "2.0.2"))
    {
        /* ProcessTwoInTwoOutJpegath, EOSM 2.0.2 */
        TTL_SetArgs     = (void *) 0xFF361498;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF429210;      /* called right after ProcessTwoInTwoOutJpegath(R) Start(%d); */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF4281F4;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF36D124;      /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF4292B8;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF428440;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF429328;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }


Yes. Note that like the 5D2 the EOSM also uses ProcessTwoInTwoOutJpegath while the 5D3 uses ProcessTwoInTwoOutLosslessPath. Probably a brother from a different mother but they are still related.

When looking for stubs I often look for the pattern on a few different cameras. You might stumble onto a solution for an issue you weren't even looking for. That's what happened when I was hunting for the EOSM2 stubs. Start with this post (https://www.magiclantern.fm/forum/index.php?topic=9741.msg186151#msg186151) then jump to this one  (https://www.magiclantern.fm/forum/index.php?topic=9741.msg186289#msg186289) and what a1ex posted right after it.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ilia3101 on February 20, 2018, 08:30:15 PM
How necessary is DecodeLosslessPath for lossless recording? Also would it be dangerous to compile and run it "where we're at" right now?

Is the pattern matching just finding really similar blocks of assembly instructions in the disassembled code of different cameras?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on February 20, 2018, 10:49:58 PM
There are three sections in lossless.c that need to be figured out. My suggestion would be to look at a recent pull request that got this working on another camera, for example:

https://bitbucket.org/hudson/magic-lantern/pull-requests/864/crop-rec-4k-eos100d-lossless/diff

Is DecodeLosslessPath necessary? Would it be dangerous to compile and run this? Try it and let us know if you and your camera survive.  ;D

The pattern matching I've been talking about is pretty much as you described it. Sometimes it is an obvious match that is easy to find, other times it is like looking for a needle in a haystack.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 21, 2018, 06:12:39 AM

FF258E70
Yes , that much better , I see I didn't look anymore after I found that one .
e92d401c
this seem to be a very important , as all stubs so far has it in the chain .
Did a quick check and there 750 references with e92d401c , that address was about half way thought.

Yea ,  as I search for the two in two out stuff I keep getting side tracked -- I see stuff that I can use in 3.5k/UHD
and compressed Full Lv , 4K etc. ... (mjpeg) and the best so far I see the hdmi stuff I want to change (hopefully loss the pillarbox at least 5d2)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 22, 2018, 09:52:49 AM
Working on

if (is_camera("5D2", "2.1.2"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xff******;
        Start_DecodeLosslessPath    = (void *) 0xff******;
        Cleanup_DecodeLosslessPath  = (void *) 0xff******;
    }

I can't seem to find DecodeLosslessRawPath yet but coming close
ff253b88: e28f0f75 add r0, pc, #468 ; ff253d64: (44776152)  *"RawDecodePath"
ffa48448: e28f2f69 add r2, pc, #420 ; ffa485f4: (4e495754)  *"TWIN DecodeMode == DECODE_TWIN"
ffa48d68: e28f2fb5 add r2, pc, #724 ; ffa49044: (4e495754)  *"TWIN CompJpCoreDecode"
ff1889e0: e28f2f7e add r2, pc, #504 ; ff188be0: (61745320)  *" StartJpegTwinDecoding %d"

Nothing about LossLess , wondering if TWIN DecodeMode may be related ?

the search continues
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 23, 2018, 03:20:03 AM
@dfort how did you find "DecodeLosslessRawPath" on 7D
Quote from: dfort on November 23, 2017, 01:24:51 AM
The decode, "RequestDecodeLosslessRawPath" on the 7D, seems pretty straightforward:

    if (is_camera("7D", "2.0.3"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xFF2E8EAC;
        Start_DecodeLosslessPath    = (void *) 0xFF2E8F5C;
        Cleanup_DecodeLosslessPath  = (void *) 0xFF2E906C;
    }

I have trouble finding any good reference  to it on 5d2 , or is it called something else I'm not a ware of
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 23, 2018, 05:19:36 AM
Found it in the 7D ,
  *"RequestDecodeLosslessRawPath"
but not in 5D2 , must be called something else
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ilia3101 on February 24, 2018, 02:00:03 AM
I feel we're getting close to lossless on the 5D2.

Quote from: dfort on February 20, 2018, 10:49:58 PM
Is DecodeLosslessPath necessary? Would it be dangerous to compile and run this? Try it and let us know if you and your camera survive.  ;D
:D
I tried, but there seems to be a function that is missing from the 5D2 codebase that won't let it link. I'll try to get rid of that error tomorrow.

@reddeercity have you had any luck compiling, is it even time to try it yet?

I've also been thinking about a pattern matching script...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 27, 2018, 08:46:25 AM
@Ilia3101 it's not ready yet , it's missing stubs for
*"RequestDecodeLosslessRawPath

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 27, 2018, 09:00:21 AM
@a1ex It seem D4 (5d2) doesn't use
*"RequestDecodeLosslessRawPath
or I can't find it like the 7d but I did find something very close it maybe the same not sure .
ff24a178: e28f2f4d add r2, pc, #308 ; ff24a2b4: (4e495754)  *"TWIN StartResWinResizeDecode"
ff24a1d0: e28f2c01 add r2, pc, #256 ; ff24a2d8: (4e495754)  *"TWIN StartTwinDecode "
ff24a1f8: e28f20f0 add r2, pc, #240 ; ff24a2f0: (4e495754)  *"TWIN DECODE_WINRES "
ff24a218: e28f20e4 add r2, pc, #228 ; ff24a304: (4e495754)  *"TWIN DECODE_TWIN "
ff24a260: e28f20d4 add r2, pc, #212 ; ff24a33c: (4e495754)  *"TWIN StartTwinDecode->DecodeEngineStart"
ff24a374: e28f2e13 add r2, pc, #304 ; ff24a4ac: (4e495754)  *"TWIN StopDecWinPath Error:%d"

and this also
ff248240: e28f2e2e add r2, pc, #736 ; ff248528: (4e495754)  *"TWIN DecodeEngineStart EngineCB:0x%lX"
ff248270: e28f2fb6 add r2, pc, #728 ; ff248550: (4e495754)  *"TWIN 1_WrCH:%#lx, 2_WrCH:%#lx"
ff248330: e28f2f8e add r2, pc, #568 ; ff248570: (4e495754)  *"TWIN EngineStop EngineCB: 0x%lX"
ff248368: e28f2e22 add r2, pc, #544 ; ff248590: (4e495754)  *"TWIN EngineCB = READ_DMAC_BIT"
ff24840c: e28f2f6f add r2, pc, #444 ; ff2485d0: (4e495754)  *"TWIN EngineState == STATE_DECODE"
ff248448: e28f2f69 add r2, pc, #420 ; ff2485f4: (4e495754)  *"TWIN DecodeMode == DECODE_TWIN"

Do you think these could be what we are after on digic 4 models ?
there is no lossless decode I can see it all this *"TWIN stuff , hope you can shed light on this  :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on February 28, 2018, 02:11:32 AM
Quote from: reddeercity on February 27, 2018, 09:00:21 AM
...there is no lossless decode I can see it all this *"TWIN stuff...

That *"TWIN stuff is also in the 7D so it doesn't really bring you any closer to finding DecodeLossless in the 5D2. Maybe a1ex can verify this idea, use the dm-spy-experiments branch to log what happens viewing a CR2 still. CR2 files use lossless compression so viewing them is probably decoding the raw file. Then again maybe the LCD is just displaying the jpeg that's embedded in the CR2 file.

Quote from: a1ex on April 12, 2017, 09:54:36 AM
As dfort guessed, all of this is found by following Canon's CR2 compression routines (codenamed Mem1ToRaw). Unfortunately, there are differences between models, so it's not always just a pattern matching.

Mem1ToRaw is in the 5D2 but I keep getting lost trying to figure out where the decoding functions are located.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 28, 2018, 02:47:39 AM
Quote from: dfort on February 28, 2018, 02:11:32 AM
Maybe a1ex can verify this idea, use the dm-spy-experiments branch to log what happens viewing a CR2 still. CR2 files use lossless compression so viewing them is probably decoding the raw file.
I did this a couple of weeks ago but I wasn't sure how to decipher until I start to read the disassembly
it make more sense to me now .
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0001_cr2_2018_2_12.log
found this  so far  , after I finish analyzing the dm-cr2 log I'll perform another test .

50EA1> ImgPlayDrv:ff98b9e8:19:03:  ClearDecodeJpegPath 414
50EDF> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
50F0C> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
50F35> ImgPlayDrv:ffa3b688:1a:02: DEC CancelDecodeJpeg Rdmac:1 Wdmac:0 Jp:0 Er:0
50F76> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
50F9D> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
52061>    CtrlSrv:ff98acd0:19:03:  (PUB) ExecuteJpegDecoding 2057
52330> ImgPlayDrv:ff9886a8:19:03:  StartJpegDecoding 614


524D9> ImgPlayDrv:ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:1) 193
527CB> ImgPlayDrv:ffa3b324:1a:04: DEC RequestDecodeJpeg Before Lock
52803> ImgPlayDrv:ffa3b340:1a:04: DEC RequestDecodeJpeg After Lock
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 28, 2018, 04:27:57 AM
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/dm-0003_Photo_Liveview_reviewing_cr2.log
much more here now , dm-log in Photo Liveview reviewing cr2  , Found this , it not in the other log
717A3> FrontShtDe:ff88ad78:96:05: sdsMem1ToRawCompression
this also
72185> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5702)
72207> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5702)��(SemOK)
7224B> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5702)


725FC> FrontShtDe:ffa59694:16:03: [TTJ][150,5702,0] RAW(5792,3804,0,14)
726FC> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1ae2c0f0
72733> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864
72998> RearShtDev:ff887524:97:05: sdfGetDevelopDevType(5702)(0x7000)
96EBC>    ml_init:ff9b1dc8:91:01: SendDataToDfe[0x180e0000]
9C5F3> **INT-64h*:ffa59470:16:03: [TTJ][150,5702,0] JpCoreCompleteCBR( 0x14469ba )
9C719> FrontShtDe:ffa590d4:16:03: [TTJ] STOP WR1:0x344ea36 WR2:0x1aea8248
9C763> FrontShtDe:ffa59100:16:03: [TTJ] STOP RD1:0x124cf564 RD2:0x124d1864
9CAA5> FrontShtDe:ff888f10:96:05: ProcessTwoInTwoOutJpegPath(R) End(5702)
9DB0B> FrontShtDe:ff88b008:96:05: sdsMem1ToJpegDevelop(5702)
9DB53> FrontShtDe:ff8890a0:96:05: sdfExecuteMem1ToJpegPath(5702)
9DBD5> FrontShtDe:ff889290:96:05: ProcessTwoInTwoOutJpegPath(J) Start(5702)


This looks very interesting
C8C90> FrontShtDe:ff887068:97:05: (ISR)sdcCompleteFrontDevelop::DisplayJpegComplete
2AC> FrontShtDe:ff888418:96:05: sdfCompleteFrontDevelop(5702)
C92F9> FrontShtDe:ff888444:96:05: ->spsComplete ExtractStageJob(5702)
C93BB> RearShtDev:ff887f40:97:05: sdsYuvToDcfCompression(5702)(0,0)��(0x80000006)
C9435> RearShtDev:ff887f40:97:05: sdsYuvToDcfCompression(5702)(0,1)��(0x80000003)
C947E> RearShtDev:ff8869a8:97:05: sdsExecuteYuvToDcfPath(5702)
C954A> RearShtDev:ff886a4c:97:05: ProcessYuvToDcfPath(5702)

:o :o :o something close
DEEE5> ImgPlayDrv:ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:0) 193
DF1D2> ImgPlayDrv:ffa3b324:1a:04: DEC RequestDecodeJpeg Before Lock
DF206> ImgPlayDrv:ffa3b340:1a:04: DEC RequestDecodeJpeg After Lock

getting warmer
E05F4> ImgPlayDrv:ff98b9e8:19:03:  ClearDecodeJpegPath 414
E0635> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
E0660> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
E068B> ImgPlayDrv:ffa3b688:1a:02: DEC CancelDecodeJpeg Rdmac:1 Wdmac:0 Jp:0 Er:0
E06C9> ImgPlayDrv:ffa3b5e4:1a:02: DEC UnregisterCBROfCompleteDecodeJpeg 475
E06EF> ImgPlayDrv:ffa3b5a0:1a:02: DEC UnregisterCBROfStopDecodeJpeg 494
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on February 28, 2018, 04:58:38 AM
this seems to be the one closest 
E1C28> ImgPlayDrv:ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:1) 193
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 01, 2018, 01:16:40 AM
@reddeercity -- Keep at it and you'll have the entire log posted.  :P

There is a branched called compressed_raw where the experiments were done on the 5D3. I tried making changes on it and compiling for the 7D but it doesn't work at all. Then I tried getting the EOSM working on that branch. Should be easy, right? After all the EOSM is already has lossless compression working--yeah, not so easy. The compressed DNG's are coming out all white or all black. Can't seem to figure it out. I'll keep trying because this branch is all about getting just lossless compression working. The crop_rec_4k branch has lossless but it also includes 8...12 bit reduced bit depth and the crop_rec module so you're going to have to juggle several babies to get it working.

I looked for *"RequestDecodeLosslessRawPath" and it isn't in any of these Digic 4 cameras:

5D2
50D
500D
550D


However, it is in these:

7D
60D
600D


The "interesting" stuff you are posting are functions that are also in the 7D firmware and these routines aren't being used for decoding lossless.

My suggestion is not to worry about the decoding stuff for now. Check the commits made on lossless.c and you'll find out that it only appears to be useful in mlv_play so don't worry about it until you can successfully save a lossless compressed MLV clip.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 01, 2018, 05:41:21 AM
compiled the "compressed_raw" branch and it locks up the camera (5d2) when trying to load either mlv_lite.mo or silent.mo
Locks up at the sense cleaning , need to pull battery , but full mlv loads find and works records 10, 12 & 14 uncompressed raw etc.
Needs more work to find what's locking it up. I'll try the "crop_rec_4k branch" with lossless and see if it makes a difference

Edit: "crop_rec_4k branch" with lossless will not compile (raw.mo errors)
keeps messing up on  raw.c "digital gain" stuff line 2336  :-X

Edit:#2 Got it to compile finally , but mlv_lite didn't make , all the rest did .
loaded silent.mo , still lockups the camera - a little good new 10 ,12bit full mlv  works better in 1:1 but that's off topic .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 02, 2018, 03:06:02 AM
@reddeercity -- Getting it to compile is rather easy, getting it to work is much harder. Did you put in the 5D2 specific changes in lossless.c? One thing that is still missing for the 5D2 is the "resources[]" which should be in your dm logs but they aren't. Check out Reply #148 (https://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741) for an example and Reply #130 (https://www.magiclantern.fm/forum/index.php?topic=18443.msg182989#msg182989) for a deeper dive into how this code works.

Today I thought I'd see what changes that need to be made so that the 7D will compile on the crop_rec_4k branch with a working silent module. Turned out there were only a few things to change:

src/gdb.c
-    unpatch_memory(bkpt->address);
+//    unpatch_memory(bkpt->address); // patch and unpatch not working on 7D?

-    patch_instruction(bkpt->address, MEM(bkpt->address), GDB_BKPT_OPCODE, "GDB hook");
+//    patch_instruction(bkpt->address, MEM(bkpt->address), GDB_BKPT_OPCODE, "GDB hook");  // patch and unpatch not working on 7D?


Not that I know what this code does so I might have short circuited the whole thing here?

In addition:

src/raw.c
+#if defined(LV_STATE)  /* older Digic 4 */
+#define BLACK_LEVEL 2047
+#endif


This is copied from the EVF_STATE code -- all cameras that work on crop_rec_4k at this point are EVF_STATE, right? Again, I'm not sure what I'm doing but it won't compile without defining BLACK_LEVEL. I made it a separate block of code in case LV_STATE cameras need a different value though I have a feeling that it will probably be the same.

Never mind that many modules fail to build, all I'm looking for right now is a silent module that can save a simple lossless compressed DNG. It will save an uncompressed DNG but hangs when saving a lossless. No crash logs.

I made a fake pull request (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff) to make it easy to see what changes I made. Note that I also copied in sections of the dm log files to show where I got the "resources[]" addresses.

This feels so close, yet so far.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 02, 2018, 07:30:21 AM
Quote from: dfort on March 02, 2018, 03:06:02 AM
@reddeercity -- Getting it to compile is rather easy, getting it to work is much harder. 
yes that for sure
Quote from: dfort on March 02, 2018, 03:06:02 AM
@reddeercity --  Did you put in the 5D2 specific changes in lossless.c? One thing that is still missing for the 5D2 is the "resources[]" which should be in your dm logs but they aren't.
Yes  I added the lossless.c changes , but messed up on the resources , more then likely that's the lockup for me I think .
To get it to compile with the crop_rec_4k branch had to comment out this in raw.c
//if (gain)
    //{
        //lv_raw_gain = gain;
       // raw_info.white_level = get_default_white_level();
        //raw_info.black_level = BLACK_LEVEL;
    //}
        //else

and in lv-img-engio.c

//total_movie_gain *= _raw_lv_get_iso_post_gain();


I do see other thing I missed , Thanks -- but I didn't have to comment out anything in the "compressed_raw" branch .
it seemed that all the problems with compiling was with the "digital gain" stuff . Mainly I was interested if it even would compile at all .
I'll check some other LOG's I have and see if I can resolve/found the "resources[]" , if not I'll make some new  Log's
Quote from: dfort on March 02, 2018, 03:06:02 AM
I'm looking for right now is a silent module that can save a simple lossless compressed DNG
Agreed , I've been focusing on trying to get a Lossless silent dng
Yea it feels close 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 02, 2018, 09:10:22 PM
Studied the code some more. There is this section that only applies to the 5D3 that I can't figure out:

modules/silent/lossless.c
    if (is_camera("5D3", "*"))
    {
        /* resolution is hardcoded in some places; patch them */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F13068, PACK32(width*2  - 1,  height/2  - 1));  /* 0xF6D171F */
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }


It has been in compressed_raw branch from the beginning. None of the other cameras need it but maybe something similar is necessary for the Digic 4 cameras?

Searched the forum and we've come across it before on the 6D (https://www.magiclantern.fm/forum/index.php?topic=15088.msg193290#msg193290). In any case, this isn't causing the 7D lockup issues when shooting a lossless DNG so I did this to try and track it down:

modules/silent/silent.c
    /* fixme: not all models are able to allocate such a large contiguous chunk */
    int max_compressed_size = ((uint64_t) raw_info->frame_size * 80 / 100) & ~0xFFF;
   
    bmp_printf( FONT_MED, 0, 83, "max_compressed_size = %d", max_compressed_size);


This is what it printed in mv1080 mode:

(https://farm5.staticflickr.com/4665/25707914037_79608cc7d1.jpg) (https://flic.kr/p/FaHMcc)

and in mv720 mode:

(https://farm5.staticflickr.com/4795/39683877795_6f179a441e.jpg) (https://flic.kr/p/23sJfiz)

The reason I was interested in mv720 mode was so I could compare it with the EOSM which is the other camera I've got with me at the moment.

(https://farm5.staticflickr.com/4659/39868204454_b85309543d.jpg) (https://flic.kr/p/23K1Ycq)

I repeated this at several different settings and it didn't change so it is probably a fixed value. Why are they different on the two cameras? IDK.

Whatever the issue, it is probably somewhere in save_lossless_dng. Redirecting to save_dng is working fine.

@reddeercity, still need to find resources[]  for the 5D2. Here's what I did on the 7D--not sure if this is correct but it seems to make sense according to the dm log file.

modules/silent/lossless.c
        else if (is_camera("7D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10002 | edmac_channel_to_index(edmac_read_chan),
            0x30000,    /* read  connection  0 */
            0x20005,    /* write connection  5 */
            0x20016,    /* write connection 22 */
            0x50003,
            0x5000d,
            0x5000f,
            0x5001a,
            0x80000,
            0x90000,
            0xa0000,
            0x160000,
            0x130003,
            0x130004,
            0x130005,
        };
/*       
1)    10002 (read channel 0xa)
2)        3 (write channel 0x3)
3)        4 (write channel 0x4)
4)    30000 (read connection 0x0)
5)    20005 (write connection 0x5)
6)    20016 (write connection 0x16)
7)    50003 (?)
8)    5000d (?)
9)    5000f (?)
10)    5001a (?)
11)    80000 (?)
12)    90000 (?)
13)    a0000 (?)
14)   160000 (?)
15)   130003 (?)
16)   130004 (?)
17)   130005 (?)
*/


I added the 5D2 code in my branch in case anyone wants to check out what max_compressed_size is on that camera.

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 02, 2018, 10:09:48 PM
The resolution section covers registers that seem to be related to width and height (possibly by some scaling factor). They are pretty much guesswork (tweaked them until they happened to work), but there are hints that not all of them might be needed. However, the exact meaning of these registers is still unknown (possibly configuring resolution for various image processing modules in the compression path).

Before, I've got these register values from QEMU, but the procedure is not very straightforward (it can emulate only a part of the lossless compression process). Now you can get the full conversation with the hardware, by using io_trace (https://www.magiclantern.fm/forum/index.php?topic=2388.msg197313#msg197313). Some of the registers (but maybe not all) are also present in the regular logs.

Look up the above registers in the photo capture log - you'll find the values from the comments.

Capture similar logs for other camera models (possibly with different sensor resolutions) to identify which registers might have to be changed.

I have some (incomplete) logs from QEMU with register diffs, but with the current tools (which evolved a lot during last year), I think it's best to capture them from real hardware using the latest io_trace_full branch. If it crashes, try reducing the logged range (to capture only the registers we are actually interested in).

Recommended models for comparison:
- 5D3 vs 6D vs 70D (all of them using ProcessTwoInTwoOutLosslessPath)
- all other D5 models (using ProcessTwoInTwoOutJpegPath)
- all other D4 models (caveat: io_trace is more likely to crash)

To get these logs in QEMU, the basic procedure is to call the compression routine with fake data. It won't get emulated completely anyway, so you can just plug some random address (0x12345678 will do), and give valid width and height to the raw_info structure passed to lossless_compress_raw. You don't have to capture any image. The test code can be executed from silent.c's init function, for example:

static unsigned int silent_init()
{
    /* keep the old stuff */

    /* code for running only in QEMU - no error checking done, no valid input, no valid results expected */
    struct memSuite * out_suite = shoot_malloc_suite_contig(16 * 1024 * 1024);
    struct raw_info out_raw_info = {
       .buffer = 0x12345678,
       .width = 1920,
       .height = 1080,
    };
    lossless_compress_raw(&out_raw_info, out_suite);

    return 0;
}


Then, run with -d io:

# from platform/CAM.FW
make install_qemu ML_MODULES=silent

# from qemu-eos
./run_canon_fw.sh CAM,firmware="boot=1" -d debugmsg,io [...]


The emulation will fail at the semaphore, but you will see the registers set by the lossless compression routine (and you can compare them with the ones obtained from io_trace). If they are identical, that's a very good sign.

TODO: automate this task, run it in QEMU for all single-DIGIC models that have lossless routines identified, then diff all these logs. Difficulty: easy, just time-consuming (edit: done for DIGIC 5, took only 2 hours; I've expected it would take a lot more).

lossless-1080-qemu-5D3.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-5D3.log)
lossless-1080-qemu-6D.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-6D.log)
lossless-1080-qemu-70D.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-70D.log)
lossless-1080-qemu-700D.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-700D.log)
lossless-1080-qemu-650D.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-650D.log)
lossless-1080-qemu-EOSM.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-EOSM.log)
lossless-1080-qemu-100D.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-100D.log)
lossless-1080-qemu-EOSM2.log (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-EOSM2.log)
lossless-1080-qemu-5D3-vs-6D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-5D3-vs-6D.html)
lossless-1080-qemu-5D3-vs-70D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-5D3-vs-70D.html)
lossless-1080-qemu-6D-vs-70D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-6D-vs-70D.html)
lossless-1080-qemu-5D3-vs-700D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-5D3-vs-700D.html)
lossless-1080-qemu-700D-vs-650D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-700D-vs-650D.html)
lossless-1080-qemu-700D-vs-EOSM.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-700D-vs-EOSM.html)
lossless-1080-qemu-700D-vs-100D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-700D-vs-100D.html)
lossless-1080-qemu-100D-vs-EOSM2.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-100D-vs-EOSM2.html)

In particular, the differences between 700D (working) and 650D (having artifacts) are quite small. No idea what they are, but... try overriding the highlighted registers and see if it helps.

The differences between 700D and EOSM (both working) are not relevant (one is some clock enabled for some unknown device, the other is from a malloc'ed address). The differences between 700D and 100D look a little more interesting, but they are probably related to how powersaving is implemented in hardware (TLDR: not relevant). The 100D and EOSM2 are also nearly identical (as expected).

For 70D, registers 0xC0F37300 and 0xC0F373E8 seem to be resolution-related, but are not used on 5D3/6D; @esas - please try. The subsequent registers smell like some possible Bayer configuration...

Emulating the full photo capture can be also useful, as it will use the unaltered Canon configuration, but it's a lot harder (it's much easier to get this one with io_trace).

One bit of mystery: on 5D3, capturing a still photo from movie mode's LiveView breaks the lossless compressor from mlv_lite. Investigating this one could give even more clues for figuring out what happens on the other models.

Quote
Why are they different on the two cameras? IDK.

You have pasted the answer a few lines above that ;)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 03, 2018, 06:32:28 AM
@dfort , I'll try the resource you posted , they seem right for 5d2 from what I see in my new dm-log .
**INT-5Ch*
not sure if this mean it's using channel #5 , I'll compile and see what happens
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 03, 2018, 08:48:30 AM
Some success  :D I can load the silent.mo and no lockups on D4 5D2

(https://preview.ibb.co/nq9Rhn/VRAM3.png) (https://ibb.co/nysOTS)

It gets better , I can take a photo with DNG Lossless compression

(https://preview.ibb.co/hajyv7/small_IMG_2362.png) (https://ibb.co/iF2U2n)

But for some reason it doesn't save a DNG as the screen shot shows , just locks up the camera
everything is unresponsive , had a solid blue led on the Liveview button and a frozen Liveview image
needed to pull the battery . After a reboot , every thing is working again -- If I take a full res lossless photo
locks up again so need to find out why it's locking up after a half shutter press . On a side note
the mlv_lite module didn't compile , but I never look at it so until I can save a lossless image that is last on the list .

Edit: I didn't add the raw.c changes that @dfort had in the https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff
I just left the changes I made in post https://www.magiclantern.fm/forum/index.php?topic=18443.msg197970#msg197970 for the digital gain stuff .
But everything else I add to my source.
 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 03, 2018, 11:32:17 AM
For solving the 650D and 70D lossless compression puzzle:

Quote from: a1ex on March 03, 2018, 10:51:53 AM
[...] what lossless compression rates are you getting on 700D, EOSM and 100D? Do they depend on horizontal resolution?

I've tried to trick it on 5D3, by commenting out the resolution patching block, but could not notice any significant change in compression rate.

However, I did some more experiments. I took the following changes from the 5D3 vs 6D log:


        EngDrvOut(0xC0F37634, 0x72000);
        EngDrvOut(0xC0F3763C, 0x1000000);
        EngDrvOut(0xC0F37640, 0x2000000);
        EngDrvOut(0xC0F37644, 0x4000000);
        EngDrvOut(0xC0F37648, 0x8000000);


and got the same artifacts as on 6D - both this (https://www.magiclantern.fm/forum/index.php?topic=15088.msg193290#msg193290) (only partly fixable by overriding 0xC0F375B4) and the per-channel black level variations!

Also got these changes from the 700D vs 650D log, and tried them on 5D3:

        EngDrvOut(0xC0F37610, 0x11);
        EngDrvOut(0xC0F37628, 0x72000);
        EngDrvOut(0xC0F3762c, 0x72000);
        EngDrvOut(0xC0F37630, 0x72000);
        EngDrvOut(0xC0F37634, 0x72000);
        EngDrvOut(0xC0F3763C, 0x1000000);
        EngDrvOut(0xC0F37640, 0x2000000);
        EngDrvOut(0xC0F37644, 0x4000000);
        EngDrvOut(0xC0F37648, 0x8000000);


and the artifacts were a lot like on 650D (but also had the per-channel black levels).

=> we've got stuff to try on 6D, 650D and 70D :D

It's probably going to be a bit of trial and error, but I'm pretty sure this is the right track.

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 03, 2018, 04:31:28 PM
Possible patch that might handle all D5 models (to be tested; can be applied with "hg import"):

d5-check.patch (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/d5-check.patch)
lossless-res-d5.patch (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-res-d5.patch)

To avoid conflicts, the easiest way is to undo any (half-working) changes to the resolution patching block from the pull requests (for example, on 6D there is one such change). The other changes should be taken from the PRs.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 03, 2018, 05:02:29 PM
Ha ha--I was posting this:

So if you are able to reproduce the 650D artifacts on the 5D by patching the differences between those log files would it be reasonable to try and patch the 650D  to the 700D?

modules/lossless.c
    if (is_camera("650D", "*"))
    {
        /* patch 650D to 700D values */
        EngDrvOut(0xC0F37610, 0x0);
        EngDrvOut(0xC0F37628, 0x71000);
        EngDrvOut(0xC0F3762c, 0x71000);
        EngDrvOut(0xC0F37630, 0x71000);
        EngDrvOut(0xC0F37634, 0x71000);
        EngDrvOut(0xC0F3763C, 0x0);
        EngDrvOut(0xC0F37640, 0x0);
        EngDrvOut(0xC0F37644, 0x0);
        EngDrvOut(0xC0F37648, 0x0);
    }


Of course you took this to a whole new level. Thanks for the patches. I'll test on the 700D and EOSM. Need volunteers for the 6D and 650D -- @Levas and @saulbass ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: saulbass on March 03, 2018, 05:05:41 PM
happy to help
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 03, 2018, 05:31:46 PM
Tried it on the EOSM before posting test builds--found a problem with simple silent lossless compressed DNG's:

(https://farm5.staticflickr.com/4755/39884232264_8986405a03.jpg) (https://flic.kr/p/23Lr7GY)

No DNG or log saved. However, Full-res silent lossless compressed DNG's are working fine.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 03, 2018, 05:47:50 PM
Similar issues on the 700D. In addition, it is not saving FRSP lossless DNG's.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 03, 2018, 06:05:31 PM
mlv_lite lossless compression seems to be working fine with all bit depths on both EOSM and 700D. I believe most people have been testing video recording so I'll put up the test builds for 650D and 6D. It will be interesting to see if we can get some good lossless MLV's out of these cameras. As usual, look for the test builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 03, 2018, 06:29:05 PM
Quote from: a1ex on March 03, 2018, 10:51:53 AM
Also wondering: what lossless compression rates are you getting on 700D, EOSM and 100D? Do they depend on horizontal resolution? (a bit off-topic, but pretty sure dfort is reading). I'd expect bad compression rates if the declared width does not match the actual image width (but that won't affect decoding).

This is probably the right topic. What test should we run to answer this question?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 03, 2018, 06:57:26 PM
Quote from: a1ex on March 03, 2018, 11:32:17 AM

and got the same artifacts as on 6D - both this (https://www.magiclantern.fm/forum/index.php?topic=15088.msg193290#msg193290) (only partly fixable by overriding 0xC0F375B4) and the per-channel black level variations!


Wow, new insights  :o

Tried the test build, for 6d, made by Dfort, but unfortunately it doesn't work for lossless recording and silent lossless...
I can't get no simple and full-ress lossless silent pics.
Lossless recording with mlv_lite gives data corruption error at slot 24  :(
error log:

ML ASSERT:
0
at mlv_lite.c:2497 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1df738 [1df7c8-1de7c8]
0x0044C93C @ c056bc:1df768
0x0044C478 @ 44c998:1df738

Magic Lantern version : Nightly.2018Mar03.6D116
Mercurial changeset   : e8dba6222cd4 (crop_rec_4k_Digic5_patches) tip
Built on 2018-03-03 17:07:06 UTC by rosiefort@RosieFoComputer.
Free Memory  : 379K + 2320K


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 03, 2018, 08:03:40 PM
Sorry, my mistake. I applied the patches to a clean crop_rec_4k branch to prevent conflicts. That should be fine for the 650D but not for the 6D because the pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/875/crop_rec_4k-for-the-6d116/diff) hasn't been merged yet. That PR is getting a bit messy so I'll make a new one once we figure this out. Posting a new build and going offline for a while. If you want to keep tweaking the code it is in my crop_rec_4k_Digic5_patches (https://bitbucket.org/daniel_fort/magic-lantern/src/e8dba6222cd4965989466e31e6519789b82b2f0a/?at=crop_rec_4k_Digic5_patches) branch.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 03, 2018, 08:12:52 PM
Thanks, will try it out and download your source  :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 03, 2018, 08:21:37 PM
The patch WORKS :D

Did a quick test with latest build from Dfort.
I have recorded a lossless 14 bit MLV file in 1824x1016 1026 resolution and the resulting DNG's are normal!

Thanks Alex  :D

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 03, 2018, 10:15:26 PM
Previously, the 700D and EOSM worked without having to tweak any registers. That means, one or more patches are probably interfering somehow with the encoder setup.

Checked the 700D in QEMU, from changeset e8dba6222cd4 - can't see anything wrong with it: lossless-1080-qemu-700D-before-vs-after.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-700D-before-vs-after.html).

Which one(s) are causing the issue? No idea, but can be found by trial and error.

Got another hypothesis - the blocks with width/10, width/12 etc might be for the second path from the TTL, which we don't use - likely used for http://lclevy.free.fr/cr2/ section 2.6. If true, that would remove a large amount of code from the patch (all the 0xC0F12* registers). The numbers seem to match very well on 5D3:

exiftool TEST_5D3.CR2 -IFD2:ImageWidth -IFD2:ImageHeight -IFD2:StripByteCounts -IFD2:StripOffsets
Image Width                     : 592
Image Height                    : 395
Strip Byte Counts               : 1403040
Strip Offsets                   : 981116



dd if=TEST_5D3.CR2 bs=1 count=1403040 skip=981116 of=uncompressed.bin
convert -size 592x395 -depth 16 RGB:uncompressed.bin -auto-level uncompressed.png



[0xC0F12020] <- 0x18A0127  -> 0x18B = 395, 0x128 * 2 = 592


Quote from: a1ex on April 12, 2017, 09:54:36 AM
WR2 outputs something of size 1776*2 x 394 bytes

Still unsure what's up with 0xC0F13068.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 03, 2018, 11:00:53 PM
Simplified patch (to be applied from scratch):

d5-check.patch (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/d5-check.patch) (unchanged)
lossless-res-d5-v2.patch (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-res-d5-v2.patch)

Please try on 6D, 700D, 650D, 70D, 100D, EOSM and (if possible) EOSM2. If it works, I'll commit this one.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 04, 2018, 01:18:50 AM
I'm trying to figure out how the silent Full res lossless DNG is saved or in my case not being save .
I step thought the lossless.c and from what I can tell it doing it thing . The thing I'm not sure about  is the
EngDrvOut in lossless.c362 (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff#Lmodules/silent/lossless.cF278T362) or is the saving routines some where else like "raw.c"? I'll try a uncompressed raw full res and see it it save a dng and doesn't lockup .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 04, 2018, 02:31:00 AM
I can save a uncompressed Full Res silent photo what out problems , so the saving routines and not a issue must be in lossless.c or silent.c
I did try another lossless FRSP after taking a uncompressed and it's trying to save , gives the resolution and max compression size plus say it's saving but locks up/frozen Liveview
needs a battery pull  .

(https://preview.ibb.co/k9CHq7/small_IMG_2366.png) (https://ibb.co/g2HTxn)

funny it started to save a dng etc. ... only after I took a uncompressed frsp  very puzzling .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 07:45:28 AM
@reddeercity - That's pretty much the same on the 7D. Putting aside Digic 4 for now though this new Digic 5 code will most likely help get lossless compression working on these older cameras.

The latest changes are working on the EOSM and 700D. No problem saving lossless silent stills and mlv_lite lossless compression is working at all bit depths. Posted test builds on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) for 6D, 700D, 650D, 70D, 100D, EOSM. Took the 70D code from nikfreak's repository. Also created a new pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/909/lossless-compression-updates-for-all-digic/diff) that supersedes the 6D PR for crop_rec_4k.

The EOSM2 is also saving lossless simple silent stills. Lossless with FRSP isn't working yet and mlv_lite displays "Compression error -3" the frames are corrupted and it saves assert logs.

ML ASSERT:
0
at mlv_lite.c:2499 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 216780 [216810-215810]
0x0044C938 @ adc3e8:2167b0
0x0044C478 @ 44c994:216780

Magic Lantern version : Nightly.2018Mar03.EOSM2103
Mercurial changeset   : 60bb31cb0539+4b3cf949679e+ (EOSM2.103_crop_rec_4k_wip) tip
Built on 2018-03-04 05:48:50 UTC by rosiefort@RosieFoComputer.
Free Memory  : 384K + 1982K


Of course there are lots of other stability issues with the EOSM2 but hey, the simple silent feature is working.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on March 04, 2018, 10:40:46 AM
Quote from: a1ex on March 03, 2018, 11:00:53 PM
Simplified patch (to be applied from scratch):

d5-check.patch (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/d5-check.patch) (unchanged)
lossless-res-d5-v2.patch (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-res-d5-v2.patch)

Please try on 6D, 700D, 650D, 70D, 100D, EOSM and (if possible) EOSM2. If it works, I'll commit this one.


Added above patch to crop_rec_4k branch(hg import) and changes were added according to patches. Tested on a 100D. All lossless workflows works which I tested:
mv1080p 8-11, 12, 14bit check
3xzoom 8-11, 12, 14bit check
crop_rec(mv720p) 8-11, 12, 14bit check
Silent dng
Fullres lossless silent dng check
lossless silent dng mv1080p check

Now how do one revert a patch when done testing?

Anyway. Great to see progress on 6D and also eosm2 and simply following the discussion is very exciting.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 04, 2018, 11:58:00 AM
Checked out Dfort latest build for 6d (made 5 hours ago, so I'm guessing this one is with the simplified patch)

Lossless recording in 14, 12 and 11-8 bit depth works in 1080 mode, 720 mode and zoom mode  8)
Simple silent lossless also works.
But, full-res silent lossless doesn't work, it doesn't create a DNG file  ???
Normal full-res silent, without lossless compression, does work in this build.

The previous build, Dfort made for 6d, with the first patches from Alex, does have a working full-res silent lossless mode.
So with first patches, 6d has working lossless full-res silent pictures.
With simplified patches, 6d has non working lossless full-res silent pictures (simple lossless does work and normal-nonlossless full-res also works)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 01:58:12 PM
First guess: does it have to do with 0xC0F13068? However, I have a feeling this one caused the issues on 700D and EOSM.

What happens if you move it out of the if (0) ?

If it doesn't help, you could try the remaining ones (although I believe they are only for RD2/WR2, which we don't use). The 6D and 70D use width/12 and height/24; the small cameras use width/8 and height/16 (see the initial patch).




I've tried to override it on Canon's CR2 compression routine (dm-spy-experiments, dm-spy-extra.c):


#ifdef CONFIG_5D3_113
    { 0xFF3D46F0, "TTL_Start", 1, ttl_start_log },
#endif

static void ttl_start_log(uint32_t* regs, uint32_t* stack, uint32_t pc)
{
    /* log the original call as usual */
    generic_log(regs, stack, pc);

    EngDrvOut(0xC0F13068, 0x0x1edb0b8f);  /* 0xF6D171F */
    EngDrvOut(0xC0F12010, 0xB8F);  /* 0xB8F     */
    EngDrvOut(0xC0F12014, 0xF6D0B8F);  /* 0xF6D0B8F */
    EngDrvOut(0xC0F1201C, 0x127);  /* 0x127     */
    EngDrvOut(0xC0F12020, 0x18A0127);  /* 0x18A0127 */

    DryosDebugMsg(0, 0, "*** TTL regs overridden");
}


If I change it to something that has the same area (in the above example: 5920x3950 -> 2960x7900), it still works (without any obvious changes in the output). If I change it by one or two pixels in either direction, the CR2 no longer gets saved. Same change in lossless.c, with RD2/WR2 enabled, would result in error -2 (failed at semaphore).

Fun stuff: the image processing block that uses 0xC0F13068 is called RABBIT (http://magiclantern.wikia.com/wiki/Register_Map#sht_mirror). No idea what it actually does.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 04:38:12 PM
Ok--let's see if the 6D needs the RABBIT.

    if (is_camera("6D", "*"))
    {
        /* 6D and 5D3 - maybe needed for full frame cameras? */
        EngDrvOut(0xC0F13068, PACK32(width*2  - 1,  height/2  - 1));  /* 0xF6D171F */
    }


If this caused issues on 700D and EOSM it will probably also cause issues on 650D.

New 6D test build uploaded (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 04, 2018, 04:55:59 PM
This build from Dfort has working full-res-silent lossless pictures.
Simple lossless also works
And I have recorded a 14 bit lossless mlv (1824x1026) which gave normal frames.

So this means that the 6d doesn't need RABBIT, right ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 05:09:50 PM
Careful with the RABBIT -- it could bite your head off! (Yeah, I'm a Monty Python nerd.)

Need some feedback from 650D and 70D users.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 05:15:43 PM
What did the RABBIT on 700D and EOSM?

Does it work with plain width and height, rather than width*2 and height/2 ? (for 6D and 700D/M)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 05:29:25 PM
You mean like this?

    if (is_camera("[camera to test]", "*"))
    {
        EngDrvOut(0xC0F13068, PACK32(width  - 1,  height  - 1));  /* 0xF6D171F */
    }


Or also take out the "-1" ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 04, 2018, 06:00:29 PM
Made the above suggested changes and all is still working normal on the 6d.
Full-ress-silent lossless -> works
1824x1026 MLV 14 bit lossless -> works
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 06:14:06 PM
@dfort: keep the -1.

Quote from: a1ex on March 04, 2018, 01:58:12 PM
    EngDrvOut(0xC0F13068, 0x0x1edb0b8f);  /* 0xF6D171F */

If I change it [0xC0F13068] to something that has the same area (in the above example: 5920x3950 -> 2960x7900), it still works

If you get it working, the question also applies to full-res lossless encoding without height-1 (https://www.magiclantern.fm/forum/index.php?topic=19300.msg196786;topicseen#msg196786), and if that doesn't work, with height+1, height+3, height+5 etc (as the encoder might require even heights).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 07:15:55 PM
The EOSM tests came out pretty much like what @Levas found on the 6D. I also tested removing the -1 and it didn't work.

EOSM: (width*2  - 1,  height/2  - 1) -- will not save lossless compressed DNG's
700D: (width*2  - 1,  height/2  - 1) -- will not save lossless compressed FRSP DNG's but works with simple silent
EOSM: (width  - 1,  height  - 1)     -- works with simple silent and FRSP lossless compression and uncompressed
700D: (width  - 1,  height  - 1)     -- works with simple silent and FRSP lossless compression and uncompressed


Both the 700D and EOSM are also doing fine without manipulating C0F13068 as in the current PR and the current crop_rec_4k.

Now for the strange part. While testing the 700D I would sometimes get DNG's that would not open in Adobe Camera Raw, though dcraw has no problems with the files. I could not reproduce these errors consistently. They might be build errors because it wouldn't happen after running "make clean" from the top directory. (I was taking a shortcut an only rebuilding the silent module.)

More quirks with the 700D. The DNG files, both compressed and uncompressed, aren't viewable with the Mac's quick view feature but they usually open fine in ACR. Another thing I never noticed before is that if you shoot a simple silent DNG (compressed or uncompressed) in photo mode it will use the mv720 raw buffer even though the movie setting is on 1920x1080 but if you shoot the silent still in movie mode it will use the mv1080 buffer.

Ok--so, what should we do with the RABBIT?

[EDIT] Seems like it would be fine to add it to the general Digic 5 stuff:

EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
EngDrvOut(0xC0F13068, PACK32(width - 1,  height  - 1));       /* 0xF6D171F on 5D3 */
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 07:22:58 PM
Quote from: dfort on March 04, 2018, 07:15:55 PM
Ok--so, what should we do with the RABBIT?

If the version with width-1 and height-1 helps breaking the encoder height limit on 700D/100D/M/etc (see the updated question in previous post), I'm tempted to place it in the common block (for all D5 models, since all of them use this register).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 07:54:38 PM
Quote from: a1ex on March 04, 2018, 06:14:06 PM
If you get it working, the question also applies to full-res lossless encoding without height-1 (https://www.magiclantern.fm/forum/index.php?topic=19300.msg196786;topicseen#msg196786), and if that doesn't work, with height+1, height+3, height+5 etc (as the encoder might require even heights).

This is working:

        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height));         /* 0xF6D171F on 5D3 */


I'm updating the PR and making new test builds.

Note that I haven't been testing the 5D3 because I've got enough on my plate for now and you are already checking it.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 08:03:06 PM
I meant, can you save full-res lossless DNG with height greater than 3528 (https://www.magiclantern.fm/forum/index.php?topic=19300.msg196749#msg196749) on 700D/M?

(the answer will be useful as documentation, regarding what this register does - what we know about it; I don't care about that one pixel difference, I just want to know where that limit comes from)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 04, 2018, 08:10:20 PM
@Alex.
The FRSP lossless I get with all working builds from today are all in 5496x3670 resolution.
While CR2's are in 5472x3648 resolution  ???
Could it be that this difference in resolution, 'break' full res silent in the first place?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 08:25:27 PM
Yes, this is the limit I'm talking about. Encoding an image larger than a CR2 works on some models (including 6D and 5D3), but not on others (700D/M/100D), and that's the point of these tests - to figure out why.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 08:25:48 PM
I think the idea is to try some of these:

EngDrvOut(0xC0F13068, PACK32(width    - 1,  height));
EngDrvOut(0xC0F13068, PACK32(width    - 1,  height +1));
EngDrvOut(0xC0F13068, PACK32(width    - 1,  height +3));
EngDrvOut(0xC0F13068, PACK32(width    - 1,  height +5));


And keep going until it breaks, right?

Hard to get my head back to where it was a month ago.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 04, 2018, 08:53:36 PM
Running out of time for today but ran a test on the EOSM and turned the volume up all the way to 11.

Using ACR:

CR2 - 5184x3456
FRSP DNG +0 through +11 - 5208x3476

In other words, no difference in the size of the image by changing the height here:

EngDrvOut(0xC0F13068, PACK32(width    - 1,  height +[0...11]));
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 04, 2018, 08:56:50 PM
No, the idea is to feed larger images into the encoder (not necessarily with valid data - it will compress whatever it finds in memory), and keep height-1 for 0xC0F13068. Click on the links.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 05, 2018, 04:55:19 AM
#dfort  tried installing your latest crop rec4k for 70D. after installing  i turn On crop rec and mlv lite. restart my camera. no buttons would work after that. i tried reinstalling your build 2 times but no luck
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 05, 2018, 06:21:07 AM
Quote from: andy kh on March 05, 2018, 04:55:19 AM
turn On crop rec and mlv lite. restart my camera. no buttons would work after that.

Pull the card and battery, delete the ML/settings directory from the card and try again with only the silent module. Try shooting simple silent DNG's, both regular and lossless. Once we get that working we can move on to MLV. The crop_rec module isn't working AFAIK.


Sent from my iPad using Tapatalk
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 05, 2018, 06:51:57 AM
Quote from: dfort on March 05, 2018, 06:21:07 AM
Try shooting simple silent DNG's, both regular and lossless.

for lossless i think i have to activate the mlv lite. once i activate mlv lite all buttons dont response
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 05, 2018, 07:26:27 AM
Quote from: andy kh on March 05, 2018, 06:51:57 AM
for lossless i think i have to activate the mlv lite.

No. The lossless code is actually a part of the silent module so let's start with just a simple silent DNG.


Sent from my iPad using Tapatalk
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 05, 2018, 08:01:31 AM
Did some body say something about a "RABBIT" :)

(https://thumb.ibb.co/kRfnsn/disassembly5d2_rabbit.png) (https://ibb.co/kRfnsn)

from Digic 4 5d2 dm-log
F4D30> ImgPlayDrv:ffa3bf78:1a:02: DEC Jpeg Format:1 W:5616 H:3744
52BC5> ImgPlayDrv:ffa3be08:1a:02: DEC CalculateRabbitParameter 895
52BF2> ImgPlayDrv:ffa3bf04:1a:02: DEC CalculateRabbitParameter

Seems to be doing pixel binding & line skipping ? more then likely see log info below
F4DB0> ImgPlayDrv:ffa3bf2c:1a:02: DEC XA:160 XB:16 XN:35 YA:8 YB:8 YN:467
F4DF2> ImgPlayDrv:ffa3ba30:1a:02: DEC SetResampleParametersForJuno 991

from what I can tell this is the full operation with RABBIT haven't found the controlling resistor yet
ffa42720:18:03: SingleWorkWidth:2106, SingleWorkHeight:1404 (line:1575)
ff98a81c:19:03:  <<<<<< ImagePlayDriverTask ProcessID = 10
ff98a81c:19:03:  <<<<<< ImagePlayDriverTask ProcessID = 1
ff9886a8:19:03:  StartJpegDecoding 614
ff9886d4:19:01: [IMPD] add1:0x436C4088 sz:1228112
ff9886f0:19:01: [IMPD] BaseAdd:0x40D00000 BaseW:2106
ff988710:19:01: [IMPD] x:0 y:0 w:2106 h:1404
ff98872c:19:01: [IMPD] Pfil1:1 PreRes:1
ff98874c:19:01: [IMPD] xxa:8 xxb:3 xya:8 xyb:3
ffa3c1e4:1a:02: DEC RegisterCBROfCompleteDecodeJpeg 466
ffa3c214:1a:02: DEC RegisterCBROfStopDecodeJpeg 485
ffa3ae5c:1a:02: DEC SetParameterDecodeJpeg 148
ffa3b290:1a:02: DEC RequestDecodeJpeg(mode:1) 193
ffa3b2c0:1a:02: DEC add:0x436C4088 size:1228112
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
ffa3b324:1a:04: DEC RequestDecodeJpeg Before Lock
ffa3b340:1a:04: DEC RequestDecodeJpeg After Lock
ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(11)
ff988408:19:03:  CompleteJpegAnalysing 428
ffa3b020:1a:02: DEC NotifyCompleteByAnalysis 640
ff9883a4:19:03:  StopEventCallBack 396
ffa3bf50:1a:02: DEC UpdateParameterDecodeJpeg 380
ffa3bf78:1a:02: DEC Jpeg Format:1 W:5616 H:3744
ffa3be08:1a:02: DEC CalculateRabbitParameter 895
ffa3bf04:1a:02: DEC CalculateRabbitParameter
ffa3bf2c:1a:02: DEC XA:160 XB:16 XN:35 YA:8 YB:8 YN:467
ffa3ba30:1a:02: DEC SetResampleParametersForJuno 991
ffa3ba60:1a:02: DEC XXA:8 XXB:3 YXA:8 YXB:3
ffa3bbc8:1a:02: DEC LES_H  XA:60 XB:6 XN:35 YA:8 YB:8 YN:467
ffa3bbf0:1a:02: DEC PFIL1  XA:60 XB:6 XN:35 YA:8 YB:8 YN:467
ffa3bc18:1a:02: DEC LES_V  XA:60 XB:6 XN:35 YA:3 YB:3 YN:467
ffa3bc40:1a:02: DEC P_RES  XA:60 XB:6 XN:35 YA:3 YB:3 YN:467


I don't know if D4's RABBIT is the same on D5 but I seems to be a encore/resizer

DEC SetResampleParametersForJuno 991
Ok , more "code names"now there a JUNO  ???

I know this has been mention before that ml captures a bigger area of the sensor  then cr2's
canon code  is only decoding for this size "DEC Jpeg Format:1 W:5616 H:3744" and not 5632x3752
So I understand why a1ex is changing thing up a bit , at least for the 5d2 it seems as if it has a preset ,
I'll try and explain , as 5d2 doesn't really fit in to the resources[] (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff#Lmodules/silent/lossless.cT251) unless I'm off in left field digic 4 has four preset engine resource
Which  makes senses -- PiP quick preview(320x240 I think) HQ preview window(3xcrop mode 1:1 2k etc. ..) full Livevew(photo mode H5616 H:3744 ) full Liveveiw (video mode w:2106 h:1404)
I could be wrong ......
02A15> LiveViewMg:ffa0708c:98:02: ResumeEngineResource
02A40> LiveViewMg:ffa06df8:98:02: ReleaseEngineResource Res:0, Free:ffffffff Count:4
02A86> LiveViewMg:ffa06ef4:98:02: ReleaseEngineResource GetResource Res:2, Free:fffffffd Count:3
02ACD> LiveViewMg:ffa06ef4:98:02: ReleaseEngineResource GetResource Res:4, Free:fffffff9 Count:2
02B0C> LiveViewMg:ffa06ef4:98:02: ReleaseEngineResource GetResource Res:1, Free:fffffff8 Count:1

I wonder if the ml code was change to capture the same area as canon would there be the same problems ?
not sure where to look right now , just a thought ,
sorry for the long post
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 05, 2018, 08:11:43 AM
lossless dng has something to do with black level. uploaded a file
https://files.fm/u/eu2tfak9

silent simple dng below
https://files.fm/u/jqr7fcna
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 05, 2018, 10:34:12 AM
Looks slightly better than this (https://www.magiclantern.fm/forum/index.php?topic=14309.msg195269#msg195269) (no more "diagonal" artifacts). However, I don't really know what to tweak for the black level fix. The processing path it uses is pretty different from all other D5 models (different enough to confuse me), but closer to 6D, so the relevant log diff is this (https://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-6D-vs-70D.html).

Candidates: C0F37074 (try 0x180000), 0xC0F37290/94 (maybe 0? no idea)

How does the image look like if you try to review it (file_man + pic_view)?

The RD1 EDMAC also shows some differences in slice configuration, but that appears to be just a larger optical black area (though the difference is unusually high). Still, the slice size is no different from 6D, so this difference is likely not relevant.
6D: (4872, skip 4872) x 3707, 4872, skip -36121008,  (4872, skip 4872) x 3708
70D: (4872, skip 5012) x 3707, 4872, skip -36639988,  (4872, skip 5012) x 3708
6D: C0F06804: 5568 x 3723 (from raw_res.txt (https://www.magiclantern.fm/forum/index.php?topic=16054.msg195213#msg195213))
70D: C0F06804: 5648 x 3719

Uncompressed full-res silent DNG, please (ideally the full set of VRAM dumps (http://www.magiclantern.fm/forum/index.php?topic=12375.0)).

For reference: 70D photo capture log (https://drive.google.com/drive/folders/0B9Mu66yg5QzRVXpjZFYwNkpzVkU) from nikfreak.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 05, 2018, 02:50:38 PM
Quote from: dfort on March 05, 2018, 07:26:27 AM
How does the image look like if you try to review it (file_man + pic_view)?

when i try to review in the camera "no image"
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: saulbass on March 05, 2018, 03:29:11 PM
New test 650D using dfort's  crop_rec_4k.2018Mar04.650D104.

A significant advance in the loss less capabilities using this camera / build combination.

1080p & crop mode (1920x1080) tested so far.

14bit lossless, 12bit lossless, 11bit lossless - all working much better - the blocking cadence has gone - brilliant.

However there is still this behaviour that the brightness of the realtime live-view image decreases as I step from 14 > 12 > 11bit lossless. This doesn't happen without lossless compression engaged in the mlv-lite menu. This characteristic also appears to be encoded into the compressed mlv recordings so they actually appear to get brighter and brighter as the compression ratio increases.
The other issue here is that the highlights become pinker and pinker and even using the highlight reconstruction of the mlv014 app it won't go away.

Also
silent DNG compressed / uncompressed working - except there is a slight greenish cast to the compressed version. So something is going on with the lossless on the 650D.

14bit lossless - slight pinkish cast to highlights
12bit lossless - darker live view / brighter mlv / pinker highlights
11bit lossless - even darker live view / even brighter mlv / even pinker highlights

silent - faint green cast to lossless DNG

Will do some more tests later.


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 05, 2018, 04:16:08 PM
Quote from: andy kh on March 05, 2018, 02:50:38 PM
when i try to review in the camera "no image"

capture image, then use file_man & pic_view

Quote from: saulbass on March 05, 2018, 03:29:11 PM
However there is still this behaviour that the brightness of the realtime live-view image decreases as I step from 14 > 12 > 11bit lossless. This doesn't happen without lossless compression engaged in the mlv-lite menu. This characteristic also appears to be encoded into the compressed mlv recordings so they actually appear to get brighter and brighter as the compression ratio increases.

Recordings shouldn't get brighter and brighter - they should look the same at all bit depths (more or less - it doesn't match perfectly yet).

However, LiveView should be brightened back - this step doesn't seem to work. Try Display -> LV display gain - is this feature working (in photo mode) ? Also try Movie -> Image fine-tuning -> ML digital ISO (in movie mode).

If it doesn't work, I'm afraid one would have to follow this procedure to diagnose it:

https://www.magiclantern.fm/forum/index.php?topic=10111.msg191218#msg191218

QuoteThe other issue here is that the highlights become pinker and pinker and even using the highlight reconstruction of the mlv014 app it won't go away.

Please run this test: http://www.magiclantern.fm/forum/index.php?topic=16054.msg195395#msg195395

(also applies to all other cameras; only 100D done so far)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 05, 2018, 05:39:01 PM
Quote from: andy kh on March 05, 2018, 08:11:43 AM
lossless dng has something to do with black level.

Well the lossless DNG is smaller than the uncompressed one and it isn't totally black so that's a good sign. That bottle with the skull and crossbones on the label next to your computer, that's a bad sign!

One thing to try is to switch back and forth between regular DNG and lossless DNG. I had an issue where if the camera was started with lossless active it wouldn't work. That has since been resolved but maybe there's something similar going on with the 70D? Sorry, couldn't find the post where this issue was discussed.

Are you able to compile? That would help speed up the process because there are several things we can try but it is best to take them like your poison--just a little drop.

Quote from: a1ex on March 05, 2018, 10:34:12 AM
Candidates: C0F37074 (try 0x180000), 0xC0F37290/94 (maybe 0? no idea)

Seems that 0xC0F37078 could also use 0x180000 so we can try doing this with the 70D adjustments:

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37290, 0);       /* 0x400 on 70D, not set on all other D5 */
        EngDrvOut(0xC0F37294, 0);       /* 0x800 on 70D, not set on all other D5 */
    }


I'll post a new 70D test build with these changes.

Quote from: saulbass on March 05, 2018, 03:29:11 PM
However there is still this behaviour that the brightness of the realtime live-view image decreases as I step from 14 > 12 > 11bit lossless.

For sure you don't have Auto ISO set? I noticed that in certain situations when ISO is set to Auto, switching bit depths affects the brightness of the LiveView.

As far as the pink highlights, that is also happening on the 700D (https://www.magiclantern.fm/forum/index.php?topic=16054.msg195408#msg195408). Looks like there's a test to run (https://www.magiclantern.fm/forum/index.php?topic=16054.msg195395#msg195395) that might eventually resolve this issue.

Quote from: a1ex on March 04, 2018, 08:56:50 PM
No, the idea is to feed larger images into the encoder (not necessarily with valid data - it will compress whatever it finds in memory), and keep height-1 for 0xC0F13068. Click on the links.

Understand I did not. Ok--put back the height-1 and will try to figure out what we were talking about last month on this discussion (https://www.magiclantern.fm/forum/index.php?topic=19300.msg196749#msg196749).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: saulbass on March 05, 2018, 06:31:35 PM
@ dfort & a1ex: I can confirm I had idiotmode activated.

with auto iso off(!)

Again using dfort's recent crop_rec_4k.2018Mar04.650D104. 650D in 1080p:
14, 12, 10, 14LL, 12LL, 10LL - all working - no differences in exposure level & no pink highlights.

Fabulous. Thank you so much - this is amazing!!

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on March 05, 2018, 06:31:57 PM
QuoteUnderstand I did not
:P :D :-*

@saulbass
Great stuff going on!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 06, 2018, 12:22:49 AM
Found the correct resources[] for 5d2  :D but I little confuse on how to implement as it's similar to 7d/D4 & 5d3/D5
ShootCapture:ff87ffec:93:03: GetCaptureResources start.
     ShootCapture:00096480:00:00: *** LockEngineResources(68007c) x38 from ffa384f8:
     ShootCapture:00096534:00:00:      1)    50001 (?)
     ShootCapture:00096534:00:00:      2)    50005 (?)
     ShootCapture:00096534:00:00:      3)    50007 (?)
     ShootCapture:00096534:00:00:      4)    50009 (?)
     ShootCapture:00096534:00:00:      5)    5000e (?)
     ShootCapture:00096534:00:00:      6)    50011 (?)
     ShootCapture:00096534:00:00:      7)    50019 (?)
     ShootCapture:00096534:00:00:      8)    5001b (?)
     ShootCapture:00096534:00:00:      9)    5001e (?)
     ShootCapture:00096534:00:00:     10)    50013 (?)
     ShootCapture:00096534:00:00:     11)    50020 (?)
     ShootCapture:00096534:00:00:     12)    50021 (?)
     ShootCapture:00096534:00:00:     13)    70000 (?)
     ShootCapture:00096534:00:00:     14)    f0000 (?)
     ShootCapture:00096534:00:00:     15)    b0000 (?)
     ShootCapture:00096534:00:00:     16)   200002 (?)
     ShootCapture:00096534:00:00:     17)   140000 (?)
     ShootCapture:00096534:00:00:     18)   120000 (?)
     ShootCapture:00096534:00:00:     19)   130009 (?)
     ShootCapture:00096534:00:00:     20)   13000a (?)
     ShootCapture:00096534:00:00:     21)   13000b (?)
     ShootCapture:00096534:00:00:     22)   13000d (?)
     ShootCapture:00096534:00:00:     23)   13000e (?)
     ShootCapture:00096534:00:00:     24)   13000f (?)
     ShootCapture:00096534:00:00:     25)   130016 (?)
     ShootCapture:00096534:00:00:     26)   22001b (?)
     ShootCapture:00096534:00:00:     27)        0 (write channel 0x0)
     ShootCapture:00096534:00:00:     28)        1 (write channel 0x1)
     ShootCapture:00096534:00:00:     29)        2 (write channel 0x2)
     ShootCapture:00096534:00:00:     30)        6 (write channel 0x6)
     ShootCapture:00096534:00:00:     31)    10000 (read channel 0x8)
     ShootCapture:00096534:00:00:     32)    10001 (read channel 0x9)
     ShootCapture:00096534:00:00:     33)    20000 (write connection 0x0)
     ShootCapture:00096534:00:00:     34)    20002 (write connection 0x2)
     ShootCapture:00096534:00:00:     35)    20012 (write connection 0x12)
     ShootCapture:00096534:00:00:     36)    2001c (write connection 0x1c)
     ShootCapture:00096534:00:00:     37)    30008 (read connection 0x8)
     ShootCapture:00096534:00:00:     38)    3000f (read connection 0xf)


This is the 7D resources[] which 5d2 was just add to see if it would work

  uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10002 | edmac_channel_to_index(edmac_read_chan),
            0x30000,    /* read  connection  0 */
            0x20005,    /* write connection  5 */
            0x20016,    /* write connection 22 */
            0x50003,
            0x5000d,
            0x5000f,
            0x5001a,
            0x80000,
            0x90000,
            0xa0000,
            0x160000,
            0x130003,
            0x130004,
            0x130005,
        };
/*       
1)    10002 (read channel 0xa)
2)        3 (write channel 0x3)
3)        4 (write channel 0x4)
4)    30000 (read connection 0x0)
5)    20005 (write connection 0x5)
6)    20016 (write connection 0x16)
7)    50003 (?)
8)    5000d (?)
9)    5000f (?)
10)    5001a (?)
11)    80000 (?)
12)    90000 (?)
13)    a0000 (?)
14)   160000 (?)
15)   130003 (?)
16)   130004 (?)
17)   130005 (?)
*/

I didn't run the right dm-spy test etc. ... Thank to a1ex for so very old work on the 5d2
If I would have listen to a1ex mouths ago , he told me to run a FA_Capture Test but couldn't understand at the time why .
Now I do there all the info I need for compressed raw & even very good info for 3.5k/UHD for D4's cam's
I use this FA_CaptureTestImage.log (http://a1ex.magiclantern.fm/bleeding-edge/FA_CaptureTestImage.log) for the first page on FRSP -- should have known , I didn't search enough.
so do I use all the 32 entries ? or stop at 17 ?
if anyone can  help that would great  :)

Edit: seem incomplete , doesn't show the resources[] for compression but the over all resources etc. ... need to run a fa-capture test myself see if I can extract the info needed .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 06, 2018, 04:53:28 AM
@reddeercity -- Looks like you're getting close but wow, that log file is very different from the one @nickfreak posted from his 7D (https://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741). What's going on with the Digic 5 lossless code is likely to apply to Digic 4. Of course getting any Digic 4 camera working with lossless compression is still quite a challenge. I would have thought that the 60D would be the first to work. Someone we know who is really good with ML code owns one.

One tip, the 5D2 should compile in the crop_rec_4k branch with this change:

src/raw.c
-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
#define BLACK_LEVEL 2047
#endif


So never mind my previous comment about juggling babies, we might as well jump onto this branch because it is where much of the current development is happening.

In the meantime I've been trying to run some tests in the hopes that the Lossless Compression updates for all Digic 5 cameras pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/909/lossless-compression-updates-for-all-digic/diff) gets accepted. (a1ex did all the heavy lifting so...)

Quote from: a1ex on March 04, 2018, 08:56:50 PM
No, the idea is to feed larger images into the encoder (not necessarily with valid data - it will compress whatever it finds in memory), and keep height-1 for 0xC0F13068. Click on the links.

Thought I figured it out so I set up this elaborate test on the 700D and half-way through realized I was doing it all wrong because of this:

Quote from: a1ex on March 04, 2018, 08:03:06 PM
I meant, can you save full-res lossless DNG with height greater than 3528 (https://www.magiclantern.fm/forum/index.php?topic=19300.msg196749#msg196749) on 700D/M?

(the answer will be useful as documentation, regarding what this register does - what we know about it; I don't care about that one pixel difference, I just want to know where that limit comes from)

Still, the test I ran was interesting so here it is.

What I did was to add this to see what was going on:

modules/silent/silent.c
    ASSERT(out_suite->size == max_compressed_size);
       
+   printf("out_raw_info.width = %d\n", out_raw_info.width);
+   printf("out_raw_info.height = %d\n", out_raw_info.height);
+   printf("out_raw_info.active_area.y2 = %d\n", out_raw_info.active_area.y2);

    out_raw_info.frame_size = lossless_compress_raw(&out_raw_info, out_suite);


With the console open it prints out the frame size being fed into lossless compression.

So, how to increase the image height? What we were working on a month ago culminated in this:

src/raw.c
#if defined(CONFIG_700D) || defined(CONFIG_650D)
    (*height)++;
#endif


So, I decided to play around here:

(*height)--;
(*height)+=2;
etc...


Yeah--that wasn't the right thing to do. I probably should have done this right before the printf statements:

modules/silent/silent.c
(out_raw_info.height)+=3;
(out_raw_info.active_area.y2)+=3;


@a1ex - Is this what you want me to do with FRSP lossless?

In any case, the results I got were interesting, at least to me. Testing on the 700D.

(*height)--;
Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  725
out_raw_info.active_area.y2 =  721
image = 1736x693

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1188
out_raw_info.active_area.y2 = 1184
image = 1736x1156

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1058
out_raw_info.active_area.y2 = 1056
image = 1800x1028

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1106
out_raw_info.active_area.y2 = 1106
image = 2520x1080

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  725
out_raw_info.active_area.y2 =  721
image = 1736x693

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476


No height adjustment
Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  726
out_raw_info.active_area.y2 =  722
image = 1736x694

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1189
out_raw_info.active_area.y2 = 1185
image = 1736x1157

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1059
out_raw_info.active_area.y2 = 1057
image = 1800x1029

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1107
out_raw_info.active_area.y2 = 1107
image = 2520x1081

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  726
out_raw_info.active_area.y2 =  722
image = 1736x694

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476


(*height)++; - This is the current code
Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  727
out_raw_info.active_area.y2 =  723
image = 1736x695

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1190
out_raw_info.active_area.y2 = 1186
image = 1736x1158

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1060
out_raw_info.active_area.y2 = 1058
image = 1800x1030

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1108
out_raw_info.active_area.y2 = 1108
image = 2520x1082


Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  727
out_raw_info.active_area.y2 =  723
image = 1736x695

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476


(*height)+=2;
Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  728
out_raw_info.active_area.y2 =  724
image = 1736x696 - ng

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1191
out_raw_info.active_area.y2 = 1187
image = 1736x1159

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1061
out_raw_info.active_area.y2 = 1059
image = 1800x1031 - ng

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1109
out_raw_info.active_area.y2 = 1109
image = 2520x1083 - ng

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  728
out_raw_info.active_area.y2 =  724
image = 1736x696 - ng

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476


(*height)+=3;
Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  729
out_raw_info.active_area.y2 =  725
image = 1736x697 - white line bottom of frame

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1192
out_raw_info.active_area.y2 = 1188
image = 1736x1160 - ng

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1062
out_raw_info.active_area.y2 = 1060
image = 1800x1032

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1110
out_raw_info.active_area.y2 = 1110
image = 2520x1084 - ng

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  729
out_raw_info.active_area.y2 =  725
image = 1736x697 - ng

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476


Ok--you get the idea. Things are starting to break and FRSP isn't budging so let's really push it:

(*height)+=9;
Movie Mode

mv720:
out_raw_info.width          = 1808
out_raw_info.height         =  735
out_raw_info.active_area.y2 =  731
image = 1736x703 - ng

mv1080:
out_raw_info.width          = 1808
out_raw_info.height         = 1198
out_raw_info.active_area.y2 = 1194
image = 1736x1166 - ng

mv1080crop:
out_raw_info.width          = 1872
out_raw_info.height         = 1068
out_raw_info.active_area.y2 = 1066
image = 1800x1038 - ng

zoom:
out_raw_info.width          = 2592
out_raw_info.height         = 1116
out_raw_info.active_area.y2 = 1116
image = 2520x1090 - ng

Photo Mode

Simple silent - same as mv720?
out_raw_info.width          = 1808
out_raw_info.height         =  735
out_raw_info.active_area.y2 =  731
image = 1736x703 - ng

FRSP
out_raw_info.width          = 5280
out_raw_info.height         = 3528
out_raw_info.active_area.y2 = 3528
image = 5208x3476


The raw buffer height and image size kept changing but it wasn't creating valid simple silent lossless compressed DNG files while FRSP wasn't affected in the least.

I'll try doing this test the right way tomorrow.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 06, 2018, 06:00:11 AM
Quote from: a1ex on March 05, 2018, 04:16:08 PM
capture image, then use file_man & pic_view

lossless dng is completely black
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 06, 2018, 06:30:18 AM
Quote from: dfort on March 06, 2018, 04:53:28 AM
@reddeercity -- Looks like you're getting close but wow, that log file is very different from the one @nickfreak posted from his 7D (https://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741). What's going on with the Digic 5 lossless code is likely to apply to Digic 4. Of course getting any Digic 4 camera working with lossless compression is still quite a challenge. I would have thought that the 60D would be the first to work. Someone we know who is really good with ML code owns one.
Yup , closer -- I bet @nickfreak did a FA_Capture_Test LOG , I'm still trying to find more info on that I think it only done in QEMU , I was hoping to do it in cam .
I may finally have to get QEMU running if I can't do it in cam.
Quote from: dfort on March 06, 2018, 04:53:28 AM
One tip, the 5D2 should compile in the crop_rec_4k branch with this change:
src/raw.c
-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
#define BLACK_LEVEL 2047
#endif

Yes , I'm working off "crop_rec_4k branch" , I  comment out the "digital gain" stuff for lower bit lossless raw (8-12)
in raw.c
//if (gain)
    //{
        //lv_raw_gain = gain;
       // raw_info.white_level = get_default_white_level();
        //raw_info.black_level = BLACK_LEVEL;
    //}
        //else

and in lv-img-engio.c
//total_movie_gain *= _raw_lv_get_iso_post_gain();

Added all the changes that you had in pull-requests/18/loss-compression-for-digic-4-cameras/diff (https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/18/loss-compression-for-digic-4-cameras/diff) except I didn't add
]-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
#define BLACK_LEVEL 2047
#endif

Complied OK , thou the mlv_lite didn't make but silent.mo did so I been working & testing off it .
Quote
Magic Lantern Nightly.2018Mar03.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 27345b3c1ad6+ (crop_rec_4k)
Built on : 2018-03-03 06:58:42 by ml@ml-pc

diff -r 27345b3c1ad6 src/gdb.c
--- a/src/gdb.c   Sat Feb 24 17:06:06 2018 +0100
+++ b/src/gdb.c   Sat Mar 03 07:58:42 2018
Maybe , that why the mlv_lite didn't make , I'll add that also and see if it make a difference .

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 06, 2018, 07:08:49 AM
For the 5D2 the black level is 1792 , 2047 if I remember right that's the black level for 5D3 & all D5 cams.
-#if defined(EVF_STATE)  /* 60D and newer */
+#if defined(EVF_STATE) || defined(CONFIG_7D)  /* 60D and newer */
#define BLACK_LEVEL 2047
#endif

I did run full mlv , & had no black level problems (green or pink cast)
And silent photo are also ok
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 06, 2018, 03:06:19 PM
Quote from: reddeercity on March 06, 2018, 07:08:49 AM
For the 5D2 the black level is 1792 , 2047 if I remember right that's the black level for 5D3 & all D5 cams.

Then do this:

#if defined(EVF_STATE)  /* 60D and newer */
#define BLACK_LEVEL 2047
#endif

#if defined(LV_STATE)  /* older Digic 4 */
#define BLACK_LEVEL 1792
#endif


The 60D, 600D, 1100D are Digic 4 EVF_STATE cameras which are newer and thus closer to the Digic 5 cameras than the 50D, 5D2, 500D, 550D, 7D which are LV_STATE cameras.

Note that defining BLACK_LEVEL was done on commit 239c022 (https://bitbucket.org/hudson/magic-lantern/commits/239c022451fe724d1843f265a7eaf05beeacb4c9). At that time LV_STATE cameras were not working with the crop_rec_4k branch so it didn't matter if these cameras wouldn't compile.

Quote from: reddeercity on March 06, 2018, 07:08:49 AM
I did run full mlv , & had no black level problems (green or pink cast)
And silent photo are also ok

You mean mlv_rec? It should work as long as you don't use the reduced bit depth settings. Same with silent uncompressed. What we're trying to do here is to get lossless compression working and we might be close--or not!

Quote from: andy kh on March 06, 2018, 06:00:11 AM
lossless dng is completely black

Uncompressed FRSP DNG working? Upload samples please, even if they are completely black.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 06, 2018, 04:22:12 PM
Quote from: dfort on March 06, 2018, 03:06:19 PM

Uncompressed FRSP DNG working? Upload samples please, even if they are completely black.

yes FRSP works in my quick test. here is a sample https://files.fm/u/wp7qhjnj

sample for lossless dng completely black https://files.fm/u/wsmmdjx5
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 06, 2018, 04:49:23 PM
@andy kh

Your black DNG isn't a Full-res silent lossless, it is a simple silent. Still black though.

According to exiftool your uncompressed FRSP is 5648x3720 and the lossless DNG you uploaded is 1984x1251.

I'm not saying you did anything wrong--it might be what the code is doing but please make sure you are in photo mode, LiveView on and double check your settings:

(https://farm5.staticflickr.com/4742/39759736605_7b6d5decf6.jpg) (https://flic.kr/p/23zr3u6)

@a1ex -- Instead of editing my post so it looks like I know what I'm doing I decided to show my unedited notes. I think this was a good lesson that others might learn from.

This isn't the way to do it, is it?

modules/silent/silent.c
(out_raw_info.height)+=3;
(out_raw_info.active_area.y2)+=3;


It has no affect on either FRSP or silent.

So how am I supposed to do this?

Quote from: a1ex on March 04, 2018, 08:03:06 PM
...can you save full-res lossless DNG with height greater than 3528 (https://www.magiclantern.fm/forum/index.php?topic=19300.msg196749#msg196749) on 700D/M?

Or is the answer -- no. Even though I manipulated out_raw_info.height and out_raw_info.active_area.y2 all the way to +9, the image was always saved as:

5208x3476 - according to Adobe Camera Raw (Photoshop)
5280x3537 - according to exiftool

No, I didn't go dyslexic. I checked those numbers multiple times.

Wait a minute -- Whack on the side of he head moment.

This is the first time that I double checked with exiftool -- so something is changing.

Without doing any height manipulation:

5208x3476 - according to Adobe Camera Raw (Photoshop)
5280x3528 - according to exiftool

So it looks like the answer is Yes! A FRSP lossless DNG can be saved with a height greater than 3528 but ACR will always crop it to 3476.

How about dcraw? The DNG that exiftool reported as 5280x3528 (normal - no height manipulation) will convert to a TIFF that is 5208x3476 (both ACR and exiftool agree on that). The DNG that was manipulated to +9 and reported as 5280x3537 in exiftool will convert to a TIFF that is 5208x3485. However, there's no free lunch here, the extra height is just noise on the bottom of the frame:

(https://farm5.staticflickr.com/4666/25784289357_ed5642d3c4.jpg) (https://flic.kr/p/FhtdV4)

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 06, 2018, 05:30:51 PM
@dfort: yes, exactly that was the result I was expecting. There's no additional image data (I didn't expect any); I just wanted to see whether the "rabbit" register was somehow related to the 3528 limit we have reached a while ago.

In other words, changing this register on all models is likely a good move.

Quote from: Audionut on December 07, 2017, 01:32:37 PM
The funny thing about coding for ML is that no one really knows what they are doing.

For 70D, it might be worth trying just 0xC0F37074/78, 0xC0F37300/3e8 and also the rabbit (without 0xC0F37290/94). If it still doesn't work, we may have to call this procedure in its original environment (during CR2 capture, without overriding any parameters) and start tweaking from there until things break. That's at least one order of magnitude more difficult than the current register tinkering (but I'm pretty sure it's solvable, simply because... it worked for Canon :D)

Quote from: reddeercity on March 06, 2018, 12:22:49 AM
Edit: seem incomplete , doesn't show the resources[] for compression but the over all resources etc. ... need to run a fa-capture test myself see if I can extract the info needed .
FA_CaptureTestImage does not compress anything, so the resources it uses are not relevant for lossless compression. You need to log a CR2 still image - that includes the capture process (the analog stuff followed by getting uncompressed image data into memory), lossless encoding (the one you are interested in), JPEG preview (also interesting for MJPEG), quick review (unsure if that's done by decoding the JPEG or otherwise, but this (https://www.magiclantern.fm/forum/index.php?topic=18315.msg176378#msg176378) suggests it might be the former), creating the CR2 (exif info, secondary images it may contain etc), saving the file, updating the file catalog and who knows what else.

However, FA_CaptureTestImage is a lot easier for studying (it's a lot simpler than a complete photo capture process, which I don't fully understand yet) and even works in QEMU on a few models. The full capture process doesn't work yet, but I've got the data necessary to implement it ("only" a 12MB log to interpret and match the emulation against). It's the second log from io_trace_full (https://www.magiclantern.fm/forum/index.php?topic=2388.msg197313#msg197313).

Quote from: reddeercity on March 06, 2018, 07:08:49 AM
For the 5D2 the black level is 1792

I remember it was 1024 at some point, maybe at certain settings (either x1 or x5), or was that just in photo mode? The only MLV sample with black=1024 I could find was a FRSP one from 5D2. Tried some old 500D builds, including the oldest nightly available, but couldn't re-create.

Looking through the old code, noticed the 5D2/50D had a workaround to prevent pink previews in x5 zoom, but the 500D requires it too (and works out of the box - with grayscale preview while recording in x5 mode, just like 5D2 used to be in the old days).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 06, 2018, 08:02:25 PM
Quote from: a1ex on March 06, 2018, 05:30:51 PM
For 70D, it might be worth trying just 0xC0F37074/78, 0xC0F37300/3e8 and also the rabbit (without 0xC0F37290/94).

So this is our next 70D test:

modules/silent/lossless.c
    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }


Quote from: a1ex on March 04, 2018, 01:58:12 PM
0xC0F13068 is called RABBIT (http://magiclantern.wikia.com/wiki/Register_Map#sht_mirror).

Test build on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).

@andy kh - The ball is on your side of the court.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 06, 2018, 08:11:04 PM
Quote from: dfort on March 06, 2018, 04:49:23 PM
@andy kh

Your black DNG isn't a Full-res silent lossless, it is a simple silent. Still black though.

According to exiftool your uncompressed FRSP is 5648x3720 and the lossless DNG you uploaded is 1984x1251.

I'm not saying you did anything wrong--it might be what the code is doing but please make sure you are in photo mode, LiveView on and double check your settings:


i was wrong. FRSL dont work. it doesnt save the dng in the camera. im gona do the test of your latest build now
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 06, 2018, 08:35:26 PM
with your latest build i get this FRSP https://files.fm/u/3kc5sbt if i activate mlv lite, recording button nad trast button dont response
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 06, 2018, 08:44:50 PM
That link does not contain any files.

Don't turn on mlv_lite yet. Let's concentrate on just the silent module. Turn all other modules off. Well, you could try to see if you can view the files in camera like a1ex suggested but let's try to keep the variables down to a minimum.

Are you able to get simple silent lossless DNG's?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 07, 2018, 12:51:25 AM
Took another look at the 7D and figured out where it is getting stuck:

modules/silent/silent.c
    ASSERT(out_suite->size == max_compressed_size);

    out_raw_info.frame_size = lossless_compress_raw(&out_raw_info, out_suite); // This is where the 7D is getting stuck

    if (out_raw_info.frame_size > out_suite->size)


Of course I haven't figured out how to get it unstuck.

Had some problems building the crop_rec_4k branch on the 7D. At first I thought it was a Digic 4 thing but it turned out to be an issue with the 7D.MASTER.203 platform. Made a comment on the Patch manager (https://bitbucket.org/hudson/magic-lantern/pull-requests/687/patch-manager-wip/diff#comment-58113169) pull request because that's where I was able to track down the issue.

Opening the console shows some issues but I doubt this is related to lossless compression because it also shows up if it is set to uncompressed DNG. Still, something seems to be out of whack here.

(https://farm5.staticflickr.com/4760/26791003238_da33657785.jpg) (https://flic.kr/p/GPqTSW)

This is in photo mode with just the silent module active and set to simple silent lossless DNG.

[EDIT] Tried various values here but kept getting those error messages:

src/raw.c
#if defined(LV_STATE)  /* older Digic 4 */
#define BLACK_LEVEL 1024
#endif


[EDIT 2] Did another test using the unified build. It also gave a similar error message but it was just a single message as I shot multiple simple silent DNG's.

(https://farm5.staticflickr.com/4784/39767883115_baea05c87f.jpg) (https://flic.kr/p/23A9Naa)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 07, 2018, 05:19:46 AM
i can view in the camera like i can view in my pc. lossless FRSP https://files.fm/u/66fxm8ny
now
simple silent losssles pic https://files.fm/u/cywkwa4

edit : they are all  lossless. DNG FRSP works
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 07, 2018, 06:49:18 AM
Oh Boy lot of stuff going on , Thanks @a1ex for info & direction -- there so much going on that it's very easy to get off track and go in circles .

Quote from: dfort on March 06, 2018, 03:06:19 PM
You mean mlv_rec? It should work as long as you don't use the reduced bit depth settings.
Yes , Full mlv +audio , yes I can record in 10-12 bit in 3xcrop_mode & 1:1 with some frame sync issue, but not corrupted
I was going to say as good as the old 2016/dec. build but better , Yes 3xcrop is flawless @ 10bit/24p 2144x1076
1:1 is much better and 14bit works as expected -- I'm not worried about that right now of course Lossless FRSP is the priority right now .

Quote from: a1ex on March 06, 2018, 05:30:51 PM
You need to log a CR2 still image - that includes the capture process (the analog stuff followed by getting uncompressed image data into memory), lossless encoding (the one you are interested in), JPEG preview (also interesting for MJPEG), quick review (unsure if that's done by decoding the JPEG or otherwise, but this (https://www.magiclantern.fm/forum/index.php?topic=18315.msg176378#msg176378) suggests it might be the former), creating the CR2 (exif info, secondary images it may contain etc), saving the file, updating the file catalog and who knows what else.
I did do a CR2 still with the startup log(the one's I posted a few pages back) but it doesn't print out the resoures[] like that FA_Capture_Test_log
It just give the Clock out 02E5E> ShootCaptu:ff87ffec:93:03: GetCaptureResources start.
02ED4> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
02F11> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
.......... 27 more clocksave out
0338D> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
033BA> ShootCaptu:ff880004:93:03: GetCaptureResources end.

Quote from: a1ex on March 06, 2018, 05:30:51 PM
However, FA_CaptureTestImage is a lot easier for studying (it's a lot simpler than a complete photo capture process, which I don't fully understand yet) and even works in QEMU on a few models. The full capture process doesn't work yet, but I've got the data necessary to implement it ("only" a 12MB log to interpret and match the emulation against). It's the second log from io_trace_full (https://www.magiclantern.fm/forum/index.php?topic=2388.msg197313#msg197313).
That's where I got part of the resources[] , I did run & compile the very simple io_trace for the 5D2 -- thou it did save any logs , I must of messed up some where .
Quote from: a1ex on March 06, 2018, 05:30:51 PM
I remember it was 1024 at some point, maybe at certain settings (either x1 or x5), or was that just in photo mode? The only MLV sample with black=1024 I could find was a FRSP one from 5D2. Tried some old 500D builds, including the oldest nightly available, but couldn't re-create.
Yes the original old raw from 2013  "1023" -- from the summer of 2013 3x Crop mode 2048x1024 extracted with the old raw dump of the time (raw2dng) I think  . (I have lots of old .raw file from2013 if needed , let me know. I also have the 2013 raw video build for 5d2 archive if needed .)
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/000001.dng
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/ExifTool_metadata_000001.dng.txt
Black Level                     : 1023
White Level                     : 15000


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 07, 2018, 07:04:22 AM
Quote from: reddeercity on March 07, 2018, 06:49:18 AM
it doesn't print out the resoures[] like that FA_Capture_Test_log

If you capture the two logs with the same code, it should list the resources in both cases. In any case, you will need LockEngineResources_log enabled (it is, by default, for 5D2, but not at startup). You could also enable CreateResLockEntry_log at startup, as the "lossless" resources are initialized when the camera starts.

Quote
Yes the original old raw from 2013  "1023" -- from the summer of 2013 3x Crop mode 2048x1024 extracted with the old raw dump of the time (raw2dng) I think

Right; mind running a "hg bisect" to find out when the black level changed? This trick is very useful to track all sorts of bugs, where you know it worked in some older version, but some reason is no longer working in recent ones and you have no idea where to start looking.

You could start with something like:

cd platform/5D2.212
hg bisect --reset
hg up fa84643 -C # some old changeset from 2013; still compiles cleanly here
make clean; make
# make sure the black level is about 1024 with this build
hg bisect --good

hg up unified -C # current unified
make clean; make
# make sure the black level is about 1728 in the same mode
hg bisect --bad

# test intermediate builds suggested by hg, and answer with --good or --bad
# you will need to test ~ 12 builds to find the changeset that resulted in different black level
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 07, 2018, 07:47:53 AM
thanks , sure I'll do that tomorrow .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 07, 2018, 03:24:28 PM
there is a problem with the previous website so im sharing some more simple lossless again
https://expirebox.com/filesgroup/6faf08bb7e162083a8345f5e06a34781.html
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 07, 2018, 03:59:21 PM
Quote from: andy kh on March 07, 2018, 05:19:46 AM
DNG FRSP works

First lossless DNG from the 70D--Yay!

(https://farm5.staticflickr.com/4610/25801917797_bcab194934.jpg) (https://flic.kr/p/Fj2zeF)

Well, it is a start.

It is Full-res, compressed and it looks like what we were getting out of the 6D just a few days ago. Simple silent lossless DNG's are still coming out black but the size of the file looks about right and the metadata is intact.

Image Size                      : 1984x1251


Really? Did you shoot the simple silent with the camera set in photo mode? Try switching the camera to movie mode and shoot simple silent DNG's with the Canon menu set to 1920x1080 and 1280x720. Zoom mode also works on that camera, right? Just want to check if you are getting black DNG's at all settings because on the 700D some settings would continue working even though we were pushing the height beyond what it could handle.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 07, 2018, 04:02:58 PM
The current samples do not answer one important question: is there any difference between all these 70D builds?

Same test scene and same exposure settings for all the builds you have tested, please (including the build from esas). Both simple and full-res, uncompressed and compressed => 4 samples for each build. Error message (screenshot if possible) if some particular setting with some particular build doesn't work.

Full-res silent pictures only work in photo mode, btw.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 07, 2018, 04:19:39 PM
For simple silent, maybe one of these is the solution?

src/raw.c
#if defined(CONFIG_5D3) || defined(CONFIG_100D)
    (*height)--;
#endif

#if defined(CONFIG_700D) || defined(CONFIG_650D)
    (*height)++;
#endif


Not sure which direction to go but I'm betting on subtracting instead of adding. I'll post two new test builds for the 70D. This shouldn't affect FRSP.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 07, 2018, 08:25:54 PM
all i did was in video mode not in photo mode. im not at home at the moment. i wil do all the test 2moro morning as soon as i get back home and report
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 08, 2018, 01:39:27 AM
@a1ex
Quote from: a1ex on March 07, 2018, 07:04:22 AM
Right; mind running a "hg bisect" to find out when the black level changed? This trick is very useful to track all sorts of bugs, where you know it worked in some older version, but some reason is no longer working in recent ones and you have no idea where to start looking.
You could start with something like:

cd platform/5D2.212
hg bisect --reset
hg up fa84643 -C # some old changeset from 2013; still compiles cleanly here
make clean; make
# make sure the black level is about 1024 with this build
hg bisect --good

hg up unified -C # current unified
make clean; make
# make sure the black level is about 1728 in the same mode
hg bisect --bad

# test intermediate builds suggested by hg, and answer with --good or --bad
# you will need to test ~ 12 builds to find the changeset that resulted in different black level

I run the bisect , but I may have messed up , (first time trying this )  when i run the "good bisect" nothing really came up .
On running the "bad bisect" after it did it's thing i was left with a message
QuoteThe first bad revision is:
changeset:   8123:ce6210d55f62
branch:      unified
user:        Audionut <***********@*****.com> (I blanked the email address)
date:        Fri Aug 23 04:21:39 2013 +0000
summary:     Bunch more feature.html links

Is this what you are looking for , did I do it right ?
so changeset : 8123:ce6210d55f62 on the unified on 8/23/2013 , is the problem ?
should I compile ?  if so (I never had to compile different changeset before , 1st time)
in the unified branch , something like
hg up unified changeset : 8123:ce6210d55f62
I'll try it and see what happens (I'll need to learn this sooner then later)

just to confirm what I did for the "bad bisect"
"hg up unified -C # current unified
make clean; make
# make sure the black level is about 1728 in the same mode
hg bisect --bad"

copy this on command line window on 5d2.212 platform and run it


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 08, 2018, 03:14:34 AM
Compiled the change set that was marked as bad , Could only run silent.mo mlv_rec.mo would not load , on the is old core just error out .
took 2 silent photo (simple) 1880x1250 , got two black levels (1791 & 1792) white level (11130) on both images
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/87190001.DNG
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/87190002.DNG
Magic Lantern v2.3.NEXT.2018Mar08.5D2212
Mercurial changeset: ce6210d55f62 (unified)
Built on 2018-03-08 00:57:33 by ml@ml-pc
Camera  : 5D2
Firmware: 212   -e   -e

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 08, 2018, 03:49:29 AM
That can't be right. Commit ce6210d (https://bitbucket.org/hudson/magic-lantern/commits/ce6210d55f626bebf97064197ae5f50a33b73c9f) is just a change to feature.html links.

I was thinking that this also applies to the 7D so I thought I'd run the same bisect test.

hg bisect --reset
hg up fa84643 -C


First issue was that the old changeset was using an old ARM toolchain so I had to install that. Then there were several things in debug.c that wouldn't compile so I kept throwing out functions until it compiled. Whoa--that is an old one. This was before modules? Shot a simple silent still and checked it in exiftool:

Black Level                     : 2047
White Level                     : 15000


Hum--this is way off the 1024 we're looking for on the 5D2. Looked in raw.c and saw that RAW_DEBUG_BLACK was in this build so I defined it along with RAW_DEBUG and this time the console popped up and showed black=2055 and white=15000 on the console. Ok, close enough.

hg bisect --good
hg up unified -C


@reddeercity - The "-C" option discards all the changes so all that stuff I threw out didn't mess up that changeset.

Turned on the black debug options again and whoa again--

(https://farm5.staticflickr.com/4776/25811262567_5464305dab.jpg) (https://flic.kr/p/FjRt7p)

Lots of information but where's the black level? Ok, shot a simple silent still and checked it in exiftool:

Black Level                     : 2047
White Level                     : 13947


The light conditions are changing rapidly where I'm at right now--watching the sunset. So some change is probably expected but it looks like this exercise doesn't apply to the 7D?

@reddeercity - If I were to continue on this and in fact the unified build is showing a black level of "about 1728" in your tests the next step would be:

hg bisect --bad

You should see in your terminal:

Testing changeset 11285:eb053c87b498 (4859 changesets remaining, ~12 tests)
543 files updated, 0 files merged, 319 files removed, 0 files unresolved


What hg bisect just did was to divide the changes between the two changesets (4859 changes) and update to the first changeset to test. I prefer doing this with two terminal windows, one for hg bisect and the other for building and installing ML. That way I can scroll back through the hg bisect list and see what is being tested. Of course the next thing to do is to test this changeset in the camera, mark it good or bad and hg bisect will set up the next changeset to test. Keep going and eventually it will find where things went bad.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 08, 2018, 06:02:56 AM
great thanks , yes I got that testing 12 builds etc. ... I kept pressing enter key(windows) and after the 6th there was no more change sets that are bad and then "The first bad revision is"
message come up . Now I think about it , I bet I should have been testing the changeset after each won came up . I run it again and see what happen .

Yes it's the old "stable build" . I was playing around with that old core got 10-12bit code implemented have some good success , even had a option to set bit rate to 16 in the old .raw format.
I abandoned it to work on UHD/compressed raw on the new ml backend , I did learn a lot from it .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 08, 2018, 07:12:28 AM
samples from dfort build
FRSP https://expirebox.com/download/77b127ef8cfab41b14aa3087cd6679ee.html

movie mode canon menu set to 1080P
simple lossless https://expirebox.com/download/d7379ccf0ec72b64e1086db455749b68.html

simple uncompress  DNG https://expirebox.com/download/1e66883a3eff9faac1b2341fa7fdbc4a.html

canon menu set to 720p: 1)Simple lossless https://expirebox.com/download/4c4738e227abf5fb3e78357c2118e114.html
                                     2) https://expirebox.com/download/1b45d686a6a6595be3dcba505aaa699c.html
                                     if i shoot something shining objets like lights or laptop screen, images are not black

i wil upload files from esas"s build in the evening


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 08, 2018, 07:45:00 AM
Came across some a useful windows app called https://www.impulseadventure.com/photo/jpeg-snoop.html
supports the following file format

•.JPG - JPEG Still Photo
•.THM - Thumbnail for RAW Photo / Movie Files
•.AVI* - AVI Movies (*mjpeg)
•.DNG - Digital Negative RAW Photo
•.PSD - Adobe Photoshop files
•.CRW, .CR2, .NEF, .ORF, .PEF - RAW Photo
•.MOV* - QuickTime Movies, QTVR (Virtual Reality / 360 Panoramic)
•.PDF - Adobe PDF Documents

Very good diagnostic tool  , not that I know everything about .
I using to check compression on the CR2 files and there's a lot of  info , so much there I never know there all that data  :o
What I discovered was there is 3 jpg images in the cr2
there's the thumb nail image ,
Full Res Jpeg (8bit YCbCr4:2:2) with picture profile applied
_MG_8718.CR2.export.000002.jpg
File Size                       : 1443 kB
File Modification Date/Time     : 2018:03:06 21:15:01-07:00
File Access Date/Time           : 2018:03:07 22:19:20-07:00
File Creation Date/Time         : 2018:03:07 22:19:20-07:00
File Permissions                : rw-rw-rw-
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Image Width                     : 5616
Image Height                    : 3744
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:2 (2 1)
Image Size                      : 5616x3744
Megapixels                      : 21.0


and the raw Jpg 
_MG_8718.CR2.export.000003.jpg
File Size                       : 21 MB
File Modification Date/Time     : 2018:03:06 21:15:01-07:00
File Access Date/Time           : 2018:03:07 00:25:24-07:00
File Creation Date/Time         : 2018:03:07 00:25:24-07:00
File Permissions                : rw-rw-rw-
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Image Width                     : 1448
Image Height                    : 3804
Encoding Process                : Lossless, Huffman coding
Bits Per Sample                 : 14
Color Components                : 4
Image Size                      : 1448x3804
Megapixels                      : 5.5

I extracted the jpg from the CR2 with jpeg snoop.exe  , here is the full _MG_8718.CR2.log.txt (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8718.CR2.log.txt)  some very interesting thing of note
Clipping data
*** NOTE: YCC Clipped. MCU=(   2,   0) YCC=(   -1,  127,  128) Y Underflow @ Offset 0x0000E87C.4
*** NOTE: YCC Clipped. MCU=(   2,   0) YCC=(   -1,  127,  128) Y Underflow @ Offset 0x0000E87C.4
*** NOTE: YCC Clipped. MCU=(   3,   0) YCC=(   -1,  127,  127) Y Underflow @ Offset 0x0000E87C.4
etc. ......
*** NOTE: YCC Clipped. MCU=(   9,   0) YCC=(   -1,  127,  127) Y Underflow @ Offset 0x0000E87C.4

Compression stats:
    Compression Ratio:  3.98:1
    Bits per pixel:     6.03:1

4:1 compression , interesting .

  RGB histogram in DC (before clip):
    R  component histo: [min=   -4 max=  256 avg=   93.5]
    G  component histo: [min=   -1 max=  255 avg=   81.7]
    B  component histo: [min=   -3 max=  253 avg=   62.0]
  RGB clipping in DC:
    R  component: [<0=  732] [>255=    1]
    G  component: [<0=    2] [>255=    0]
    B  component: [<0=  855] [>255=    0]
    RGB histogram in DC (after clip):
    R  component histo: [min=    0 max=  255 avg=   93.6]
    G  component histo: [min=    0 max=  255 avg=   81.7]
    B  component histo: [min=    0 max=  253 avg=   62.0]
Average Pixel Luminance (Y):
    Y=[ 82] (range: 0..255)
Brightest Pixel Search:
    YCC=[ 1018,  -26,   11] RGB=[255,255,247] @ MCU[  4,  1]

etc. .....

The reason I did this was to investigate the (5D2) full res size as different app are reporting are not the same.
from 5616x3744 to 5634x3753(UFRaw) with the same CR2 , strange
(https://thumb.ibb.co/djBzA7/UFRaw.png) (https://ibb.co/djBzA7) (https://thumb.ibb.co/ipH0Hn/UFRaw1.png) (https://ibb.co/ipH0Hn)
So is the cr2 just being crop in certain App like photoshop etc. ... but really is 5634x3753 just like the FRSP ? if so then it shouldn't be a problem save lossless at the res.

One last thing when I loaded the cr2 on ExifTool , it give me a BOAT load of info , specially about the compression   Exiftool_MG_8718.CR2.txt (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Exiftool_MG_8718.CR2.txt)
e.g.
Raw Image Segmentation  : 2 1936 1920
and this
Sensor Width                    : 5792
Sensor Height                   : 3804
Sensor Left Border              : 168
Sensor Top Border               : 56
Sensor Right Border             : 5783
Sensor Bottom Border            : 3799

Black level
Average Black Level             : 1024 1024 1024 1024
Raw Measured RGGB               : 108619 154380 158979 55163
Per Channel Black Level         : 1024 1024 1024 1023
Normal White Level              : 14977
Specular White Level            : 15489

One nice  thing about it is there a full res jpeg at 1.4 mb , would be great to be able save  that jpeg stream to image sequence *.jpg (dreaming right now)
cr2 reports black level as 1024 so , I need to find out more about the 1791 level .
Edit: the 3 exported jpg files from the cr2 with JpegSnoop
Thumbnail (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8718.CR2.export.000001.jpg)
Full Res 5616x3744 jpeg (http://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8718.CR2.export.000002.jpg)
Raw-Lossless, Huffman coding (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8718.CR2.export.000003.jpg)
original CR2 (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8718.CR2)


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on March 08, 2018, 07:51:47 AM
I noticed too the beefy sized jpg preview in CR2 files. They are the same size in sRAW as well. There is also a tif and a smaller jpg thumb in there. I use this preview in HDR workflows. You can fetch the preview with exiv2:
exiv2 -ep3 -l . *.{cr2,CR2}
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 08, 2018, 08:05:54 AM
@reddeercity - If you aren't testing each changeset how do you know whether to mark it good or bad?

I'm a little lost over what is going on with this exercise. I tried compiling that old changeset and it built fine on the 5D2 while it was a struggle on the 7D. Is there an issue with the current unified code on the 5D2? What do the "good" and "bad" silent stills DNG files look like? The ones you posted look fine to me. Everything coming out of the 7D is looking good on all the changesets I tried. Of course none of the Digic 4 cameras can do lossless compression even though we're pretty sure we found the right addresses to do this. That's what we're trying to figure out, right?

@andy kh - Ok, the simple silent isn't completely black. That's a step in the right direction. Which build worked better, height++ or height-- ?

(https://farm5.staticflickr.com/4786/40643506842_c7c8d9ea83.jpg) (https://flic.kr/p/24VwAzY)

Of course the goal is to get it to match the uncompressed as closely as possible:

(https://farm5.staticflickr.com/4796/38875128980_b20b92d1aa.jpg) (https://flic.kr/p/22egcsu)

BTW - that esas build is still here: https://bitbucket.org/es_as/magic-lantern/downloads/

As for the jpg previews in CR2 files--isn't that what the image playback is using? Don't see much point in having to do in-camera decoding unless you're manipulating the images in the camera. Maybe that's why we can't find the lossless decoding in the 5D2?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 08, 2018, 08:36:22 AM
Quote from: dfort on March 08, 2018, 08:05:54 AM
@reddeercity - If you aren't testing each changeset how do you know whether to mark it good or bad?
I mess up on the what I needed to do my bad , I'll redo it tomorrow
Quote from: dfort on March 08, 2018, 08:05:54 AM
I'm a little lost over what is going on with this exercise. I tried compiling that old changeset and it built fine on the 5D2 while it was a struggle on the 7D. Is there an issue with the current unified code on the 5D2? What do the "good" and "bad" silent stills DNG files look like?
From my understanding we are seeing where the black level changed from 1023 in (2013) to 1024 & 1791 current from from this post here
Quote from: a1ex on March 07, 2018, 07:04:22 AM
mind running a "hg bisect" to find out when the black level changed? This trick is very useful to track all sorts of bugs, where you know it worked in some older version, but some reason is no longer working in recent ones and you have no idea where to start looking.
The bad is  either pink Or green cast (black level problem)
Quote from: dfort on March 08, 2018, 08:05:54 AM
The ones you posted look fine to me. Everything coming out of the 7D is looking good on all the changesets I tried. Of course none of the Digic 4 cameras can do lossless compression even though we're pretty sure we found the right addresses to do this. That's what we're trying to figure out, right?
Yes
Quote from: dfort on March 08, 2018, 08:05:54 AM

As for the jpg previews in CR2 files--isn't that what the image playback is using? ....  Maybe that's why we can't find the lossless decoding in the 5D2?
Maybe  , There's more going on with that JPCore then we know . I wonder if on D4's it's looking to make or compile all those 3images + the History gram thumbnail
run jpeg snoop and you will see what I mean, thought the TwoIN-TwoOut  , as it's noted in the code we are only use 2 channels (one in one out) if I read the code right  say something to the effect
"we don't use these" I have a feeling it something with the driver out .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 08, 2018, 10:32:05 AM
Quote from: reddeercity on March 08, 2018, 08:36:22 AM
The bad is  either pink Or green cast (black level problem)

I'm not talking about black level issues, I'm talking about black level value ("good" = near 1024, "bad" = near 1792). This is just a convention - "hg bisect" only accepts two labels - good or bad. You can assign them otherwise if you prefer. The image will be good in both cases, as long as the black level metadata is correct. If it's not correct, you can adjust it, but your answer ("good" or "bad") should not depend on whether the metadata is correct or not. The answer should only depend on the black level value that gives correct image (or --skip if, for some reason, the test could not be run with some particular changeset).

If in doubt, check the optical black area from a (simple) silent DNG. This feature was implemented before raw recording, and wasn't changed as often as raw video, so it's more likely to get a successful test run with silent pictures, rather than raw video. Again, ignore the metadata if the image is bad; only consider the mean (or median) value of the optical black area (or simply the black level value that gives good image).

If all else fails, try reading a tutorial (http://hgbook.red-bean.com/read/finding-and-fixing-mistakes.html#x_126).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 08, 2018, 04:13:01 PM
Quote from: Which build worked better, height++ or height-- ?
/quote]


i dont remember. i wil test esas"s build now and retest your build and let you know
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 08, 2018, 04:48:04 PM
samples of esas's build

samples simple lossless 1. https://expirebox.com/download/a5d97c7b46b79a01c319d49849a65a1d.html

2.  https://expirebox.com/download/e6f035fdc9f19ce6ac4b57136842af9e.html

simple dng   https://expirebox.com/download/66e0dde45f502f0194eb3a81a59506da.html

FRSP  https://expirebox.com/download/8601c7616b80cbc76100a8d51fd40388.html
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 08, 2018, 04:56:53 PM
Great -- while you were testing I made yet another build. What happens if we include all the stuff we did with the 6D/650D to fix the problems we were having on those cameras? This probably won't work but it might be worth a try. The build is called 70D_crop_rec_4kitchen_sink and is on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 08, 2018, 05:04:52 PM
height-- is better
samples of height-- https://expirebox.com/download/77aa034865de95b3286991c66f88cf1f.html
https://expirebox.com/download/3f43abd05c91d067066639f72534574f.html

samples of height++  https://expirebox.com/download/a4c8f9cf605d703f3c7b126cf2ed3138.html

https://expirebox.com/download/860dc9b62e066ff02b63c04ea0d86452.html
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 08, 2018, 05:14:34 PM
crop rec 4k kitchen  https://expirebox.com/download/ccf6a9ad466f7e22385c0123c3da6f20.html

https://expirebox.com/download/5db28f39dfadf77202a779bb937d8d35.html
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 08, 2018, 05:22:42 PM
Well, that experiment blew up. Still it is good to know that those settings aren't the ones causing the issues.

I'm a little confused, some of your simple silent stills are uncompressed yet they are showing problems. Is simple silent on the 70D build posted on the nightly downloads page (https://builds.magiclantern.fm/70D-112.html) also having this issue?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: andy kh on March 08, 2018, 05:48:46 PM
im not sure about issues
i have a sample from the nightly build
simple dng  https://expirebox.com/download/3cc3fcf483cc7692b6963afc8d80b097.html
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 08, 2018, 11:26:16 PM
That DNG from the nightly build looks good.

Maybe we should back up to and see if there's anything else that we need to resolve the issues on the 70D:

Quote from: a1ex on March 05, 2018, 10:34:12 AM
Uncompressed full-res silent DNG, please (ideally the full set of VRAM dumps (http://www.magiclantern.fm/forum/index.php?topic=12375.0)).

For reference: 70D photo capture log (https://drive.google.com/drive/folders/0B9Mu66yg5QzRVXpjZFYwNkpzVkU) from nikfreak.

Here's a few things to track down the green/pink issues:

Quote from: a1ex on December 30, 2017, 10:00:50 PM
Re white level:

http://www.magiclantern.fm/forum/index.php?topic=16040.msg191131#msg191131
http://www.magiclantern.fm/forum/index.php?topic=16040.msg190644#msg190644
https://bitbucket.org/hudson/magic-lantern/commits/98539896ebd056e0a2128658aacd73b757a78b08
https://bitbucket.org/hudson/magic-lantern/commits/ab792b7c5137fceee4016bf9ef88767d031d27c5

These are the current 70D tweaks:

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 10, 2018, 10:53:47 PM
Added 6D and 650D lossless support to the main crop_rec_4k branch (with minor differences from dfort's version, hopefully non-functional for these models). Feel free to compile from source until the builds will appear.

70D is included too, but doesn't work well yet. Can be fixed by somebody who has the camera and some tinkering time in their hands.

Thanks dfort and other contributors!




BTW, noticed the lossless encoder doesn't really care about W*H being modulo 16 - removed this restriction and got a few extra pixels. Please double-check. I'd expect 1736x696 in 720p on 18MPix models.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 11, 2018, 01:40:42 AM
Great! Killed my old PR's and took down the test builds to reduce clutter.

I'll keep working on the 70D but need user input.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: saulbass on March 11, 2018, 05:55:44 AM
Just had a quick play with magiclantern-crop_rec_4k.2018Mar10.650D104 - on my 650D.

Looks great - it's amazing what you've managed to squeeze out of this camera.

One tiny caveat, the crop rec res in 720p & 1080p is 1920x1076. Is there a reason why we couldn't get 1080 vertical? dforts earlier builds managed a full 1920x1080.(i.e. crop_rec_4k.2018Mar05.650D104)

Anyway - major progress - haven't had time to look at anything else much - FYI the non crop rec res remains at 1736x976 as per your last builds. (i.e. crop_rec_4k.2018Mar05.650D104)

Compressed recording looks fine btw.




Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 11, 2018, 09:47:47 AM
I assume you are talking about the x5 zoom mode. This change (https://bitbucket.org/hudson/magic-lantern/commits/57614b3f26e42ae1cb834a4f9079851beff2531c) is the culprit. From the samples I have, there are 28 pixels at the top, just like in all other modes; previous builds used to skip 26 pixels in zoom mode, so the first two lines were black. Your DNG sample shows them.

The other two lines at the bottom seem valid. Sorry, missed this one.

In movie crop mode (not to be confused (https://www.magiclantern.fm/forum/index.php?topic=16054.msg181220#msg181220) with x5 zoom or crop_rec), skip_bottom was declared as 2 in previous builds. I reduced the height by 2 (for consistency with other models - height-- vs height++) and redefined skip_bottom as 0 - that should not change the active area (compared to previous build in this mode).

However, looking again at the EOS M sample I have from this mode, there are no bad lines at the bottom (height=1060 aka height++); there are just 28 black lines at the top (so there may be 2 additional lines of usable data, similar to x5 zoom). So, I don't know what was the reason for using skip_bottom = 2 for mv1080crop - were things different on other models?

The non-cropped modes had declared 4 skipped lines at the bottom. That was the reason for changing it, as 5D3 and 100D had some issues with extended resolution (they required height--, autodetected height was too much and caused hiccups / skipped frames, but 700D/650D/M used height++). I did not see a valid reason to read two additional lines (compared to other models) and then skip 4 at the bottom.

720p: there were 2 bad lines at H=726, so there must have been 3 at H=727. Reduced height by 2 -> 725 (for consistency with other models) and declared 1 skipped line (hopefully squeezed one extra good line, compared to dfort builds).
1080p: there were 2 bad lines at the bottom with H=1190 (but ML was skipping 4); reduced H by 2 for consistency and declared skip_bottom = 0 (hopefully squeezed 2 additional lines, compared to dfort builds).

Please double-check the resolution in 720p and 1080p (to make sure the additional lines are there, with valid image data). I've only counted the pixels from VRAM samples; did not test these settings on a camera.

BTW, I don't have the following VRAM samples (http://www.magiclantern.fm/forum/index.php?topic=12375.0):
- 650D movie crop mode (MVC-1080), but I have one from EOS M
- 700D (other than scattered files picked from the forum, I don't have a full set of VRAM dumps)
- 100D (same as above)
- 70D (same as above)
- 60D, 5D2, 500D, 550D, 1200D (lower priority; I have a few dumps, but not the full set)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: saulbass on March 11, 2018, 10:07:56 PM
@a1ex - thanks for the reply. I'm back stacked with work for a few days but when I get a chance will look into these tests.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 12, 2018, 10:38:35 PM
Hey you wizards! Happy to serve with my limited knowledge and a 70D at hand. With the latest crop_rec_4k.2018Mar10.70D112.zip build which I downloaded from @dfort 's bitbucket page, I get an Error message when I try to load either one of the Raw modules. Camera in Manual Photo Mode, iso 1250, shutter 1/50, Picture saving style Raw only (i.e. NOT raw +jpeg), Lens was a Sigma art 18-35,if that makes any difference.
Here's the screenshot
(https://thumb.ibb.co/doQJSn/Crop_rec_load_module_error.jpg) (https://ibb.co/doQJSn)


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 12, 2018, 11:19:09 PM
@David_Hugh - the only module that we are testing at this time is the silent module. Turn everything else off--we know that the modules you loaded don't work yet.

What we're looking for at this time is regular and lossless DNG's shot in every video mode this camera supports, mv720, mv1080 and 5x zoom and also Full-resolution silent pictures.

There is an earlier build with an incomplete set of DNG's. That build is located here:

https://bitbucket.org/es_as/magic-lantern/downloads/magiclantern-crop_rec_4k_70D.2018Jan11.70D112.zip
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 12:13:01 AM
Alright! Sorry, I thought you guys were already talking full-on video mode. Let's see if I can be a little more helpful this time around (Although, looking at the Dngs - all broken - I'm sure I probably f***** up something...) . Here are the 720/1080/5xZoom/FRSP Dngs, normal and lossless. Wasn't able to save a compressed FRSP, the error message is attached instead.

https://we.tl/OYDFhjVE3O
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 12:18:27 AM
Oh, and this was for the march 10th build by the way. If this is any help at all (in other words, if I'm not one of the inviduals too stupid to test because I dont get all the settings right - see previous post  :D  ) I'll test the January build tomorrow, it's late where I'm at.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 13, 2018, 12:33:38 AM
Results are, indeed, a bit unexpected, but the test was done correctly (exactly what I asked for). Will double-check tomorrow in QEMU.

In particular, the "card full" error is quite surprising. I'd expect this on the build from esas (before this was fixed), but I'd expect either success or some other error message on the March 10 build. Mind double-checking the build version? (Help menu, and also on silent.mo after pressing Q).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 13, 2018, 03:19:46 AM
Yes, very unexpected. I've been working with andy kh trying to get a full set of DNG files and his files from the March 10 build are very different. I've only got FRSP for that build and simple silent for the esas version but I'll keep updating this as he continues testing.

https://www.dropbox.com/sh/ba278pyafw9hlze/AACUrAyC9pWv8CnpOzXnlpqFa?dl=0

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 09:57:13 AM
Yes, you were both right. Build info in the help menu said it's esas build from January (although I'm not exactly sure how that happenend because I definitely updated the version, why else would the raw modules be "broken"). Anyway. Reinstalled the March 10 version (the right one this time - 100% sure, looked it up in help pre + post test). So now here's the Dng Set for this test, but they're not really looking any different and I still cant open them in Lightroom. On the plus side, as you predicted, I was able to save a compressed FRSP this time.

https://we.tl/GGVBUAU8kU
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 13, 2018, 10:58:51 AM
Thanks, this time it matches my expectations.

This was the same issue we had on 6D, and was solved by this block (https://bitbucket.org/hudson/magic-lantern/commits/90a58a522b39783601fc63c584bae7c8bb76dcc6#Lmodules/silent/lossless.cF275), but on 70D the registers are different.

I'd like the same test with the other register tweaks:


    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }


The March 10 build includes just the first 3.

An easier task that can be done with any recent build: full set of VRAM dumps (http://www.magiclantern.fm/forum/index.php?topic=12375.0).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 13, 2018, 03:22:01 PM
New build posted (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) - Mar13

@David_Hugh - Could you please double check this:

https://www.magiclantern.fm/forum/index.php?topic=5601.msg198375#msg198375

The instructions on how to run the test is on Reply #1579 (https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632) of the 12-bit (and 10-bit) RAW video development discussion.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 03:59:50 PM
Here's the DNG set for March 13th build.

https://we.tl/AyWvpX82yu


As for the memory test, am I correct in assuming that I need to run the allocate-raw-lv-buffer build from your download page? If that's all there is to it (plus the instructions on the 10/12 bit thread) I'll do this later in the evening (got some stuff to do now..)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 13, 2018, 04:03:44 PM
Quote from: David_Hugh on March 13, 2018, 03:59:50 PM
As for the memory test, am I correct in assuming that I need to run the allocate-raw-lv-buffer build from your download page?

Yes, that's right. The build is still posted. Thanks!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 13, 2018, 04:04:33 PM
Sorry, cannot compare with previous results, as the exposure is way different.

Please repeat with both March builds on the same test scene, if possible.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 04:19:47 PM
Ok! Will do! Here are the screenshots from the alloc-raw-lv-buffer test in the meantime!
https://we.tl/tGatqZnFOh

I did:
photo mode, right after startup
720 mode, right after startup
1080 mode, right after startup
Zoom mode (with digital zoom in the menu selected), right after startup
Zoom mode  - the "normal" way by pressing the magnyfing button, but that's obviously not how the cam starts up
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 13, 2018, 04:47:09 PM
Quote- open the console (Debug menu) and enable something that uses LiveView RAW features (raw video, raw histogram etc)

Maybe I should have written these on two different lines. Your histogram was YUV-based in all the screenshots; no RAW-based features were enabled.

RAW-based histogram looks like this (https://www.magiclantern.fm/forum/index.php?topic=12096.0), with vertical bars at each EV.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 10:22:52 PM
Right... Just for future dummy proofing this test: you can select histogram RAW based without having loaded the mlv_rec or other raw modules.

What follows: Histogram is visible, Dumbest Assumable User (in this case me) thinks - hey! histogram raw based selected, everything fine, lets do the test!

Here's the same test with raw stuff enabled:
https://we.tl/2q0izHvPBv

0=photo mode normal startup
1=photo mode live view
2=720p selected at startup
3=1080 selected at startup
4= digital zoom enabled at startup via canon menu
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 13, 2018, 10:42:48 PM
Right; there is a PR that addresses this issue (https://bitbucket.org/hudson/magic-lantern/pull-requests/730/menu-proper-handling-of-raw-jpg-h264/diff#chg-src/zebra.c), need to revisit it.

Just tried running this test in QEMU, after a very recent breakthrough (https://bitbucket.org/hudson/magic-lantern/commits/d2116b6732b982f2eada9e3acd2bf3c481fe6a25) - couple of hours ago - and got 4B328000-4D7FFFF0. First address matches :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 13, 2018, 10:54:46 PM
That's great news! Here's the repeated DNG test by the way. Same exposure settings shooting the same "target" for March1oth and 13th builds.

https://we.tl/0Qklp88OxJ
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 13, 2018, 11:04:29 PM
Looks identical to me, so probably 0xC0F37074/78 don't have any effect.

Long shot: what about enabling the 6D/650D code block for 70D? After all, today (https://en.wikipedia.org/wiki/Friday_the_13th#Tuesday_the_13th_in_Hispanic_and_Greek_culture) seems to be our lucky day (https://www.magiclantern.fm/forum/index.php?topic=21728.msg198425#msg198425)...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 14, 2018, 12:26:08 AM
I made three new test builds for the 70D and put them on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). I believe these are all combinations that we haven't tried yet. Let's see if this really is a lucky day.

modules/silent/lossless.c

test A
    /* configure the processing modules */
    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);

    if (is_camera("6D", "*") ||
        is_camera("650D", "*") ||
        is_camera("70D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */
        EngDrvOut(0xC0F37628, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3762C, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37630, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37634, 0x71000); /* 72000 on 6D and 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3763C, 0);       /* 0x1000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37640, 0);       /* 0x2000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37644, 0);       /* 0x4000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37648, 0);       /* 0x8000000 on 6D and 650D, 0 on all other D5 */
    }

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
    }
    else if (is_camera("DIGIC", "5"))
    {
        /* all other D5 models use this register instead */
        /* the hardware encoder (and other image processing modules that might be used)
         * expect slice width and slice height for these registers
         * Canon configuration: slice width = image width / 2, slice height = image height
         * our configuration:   slice width = image width, slice height = image height / 2 */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
    }

    if (is_camera("DIGIC", "5"))
    {
        /* required for full-res silent pictures? */
        /* this one doesn't seem to refer to slice size, but to total image size */
        /* sht_mirror shows 0xC0F13000 - 0xC0F13064 as RABBIT */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
    }


test B
    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);

    if (is_camera("6D", "*") ||
        is_camera("650D", "*") ||
        is_camera("70D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */
        EngDrvOut(0xC0F37628, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3762C, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37630, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37634, 0x71000); /* 72000 on 6D and 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3763C, 0);       /* 0x1000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37640, 0);       /* 0x2000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37644, 0);       /* 0x4000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37648, 0);       /* 0x8000000 on 6D and 650D, 0 on all other D5 */
    }

    else if (is_camera("DIGIC", "5"))
    {
        /* all other D5 models use this register instead */
        /* the hardware encoder (and other image processing modules that might be used)
         * expect slice width and slice height for these registers
         * Canon configuration: slice width = image width / 2, slice height = image height
         * our configuration:   slice width = image width, slice height = image height / 2 */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
    }

    if (is_camera("DIGIC", "5"))
    {
        /* required for full-res silent pictures? */
        /* this one doesn't seem to refer to slice size, but to total image size */
        /* sht_mirror shows 0xC0F13000 - 0xC0F13064 as RABBIT */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
    }


test C
    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);

    if (is_camera("6D", "*") ||
        is_camera("650D", "*") ||
        is_camera("70D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */
        EngDrvOut(0xC0F37628, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3762C, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37630, 0x71000); /* 72000 on 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F37634, 0x71000); /* 72000 on 6D and 650D, 71000 on all other D5 */
        EngDrvOut(0xC0F3763C, 0);       /* 0x1000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37640, 0);       /* 0x2000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37644, 0);       /* 0x4000000 on 6D and 650D, 0 on all other D5 */
        EngDrvOut(0xC0F37648, 0);       /* 0x8000000 on 6D and 650D, 0 on all other D5 */
    }

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F37074, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
        EngDrvOut(0xC0F37078, 0x180000);/* 0x130000 on 70D, 180000 on all other D5 */
    }
    else if (is_camera("DIGIC", "5"))
    {
        /* all other D5 models use this register instead */
        /* the hardware encoder (and other image processing modules that might be used)
         * expect slice width and slice height for these registers
         * Canon configuration: slice width = image width / 2, slice height = image height
         * our configuration:   slice width = image width, slice height = image height / 2 */
        EngDrvOut(0xC0F375B4, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F on 5D3 */
    }

    if (is_camera("DIGIC", "5"))
    {
        /* required for full-res silent pictures? */
        /* this one doesn't seem to refer to slice size, but to total image size */
        /* sht_mirror shows 0xC0F13000 - 0xC0F13064 as RABBIT */
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0xF6D171F on 5D3 */
    }
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 14, 2018, 01:29:44 AM
Out of luck for the day as it seems... none of the 3 tests shows any sign of improvment as far as I can tell (which hopefully doesnt mean much since I know 0 about how all of this works ;) )

Moreover, they dont seem any different from each other, all 3 tests, or rather all 5 tests look pretty much the same.


https://we.tl/1uKL17PkrO
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 14, 2018, 06:21:48 AM
Actually, there are significant differences between them. Now it is a matter of sorting them out and figuring out what to try out next.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 14, 2018, 06:28:53 AM
Not sure if this is going to fly
if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1)); 
        EngDrvOut(0xC0F12020, PACK32(width    - 1,  height/10 - 1));  /* 0x18A0127 */
    }



I'm finding the 5D3 113 is very similar to 5D2 , most of the 0xC0f***** below match up in the 5d2 disassembly with the same strings
I haven't try it yet , looking for some feedback -- It's late right now I'll try it tomorrow


{
        /* values for 5D3 - just for future reference */
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2  - 1));  /* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 14, 2018, 07:21:23 AM
Quote from: reddeercity on March 14, 2018, 06:28:53 AM
Not sure if this is going to fly

Well there's only one way to find out, right?

Those 0xC0f***** addresses aren't necessarily something that can be determined from the disassembly. Of course we have to use whatever tools we have available whether it be IDA, ARMu, QEMU, just a simple text editor or whatever else you have in your toolbox.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 14, 2018, 11:14:47 AM
A and C look identical to me, B is worse (has diagonal artifacts in addition to black level issues).

B is missing 0xC0F37300/4 compared to A, so these are important. The test just confirms our previous hypothesis, but doesn't give any hints on how to solve the black level issues.

To summarize:
- the artifact we have solved is this (https://www.magiclantern.fm/forum/index.php?topic=7473.msg197989#msg197989) or this (https://www.magiclantern.fm/forum/index.php?topic=15088.msg192884#msg192884) (using 0xC0F37300/4)
- the artifacts we have yet to solve are this (https://www.magiclantern.fm/forum/index.php?topic=15088.msg193294#msg193294) (vertical split, possibly pointing to slice configuration) and this (https://www.magiclantern.fm/forum/index.php?topic=15088.msg191860#msg191860) (black level issue, solved on 6D by that big block we don't know what exactly it does)
- the relevant log diff (6D vs 70D) is this (https://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-6D-vs-70D.html)

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 14, 2018, 09:23:00 PM
That was an interesting experiment that didn't pan out so I took down the ABC builds and put up a build from the current crop_rec_4k in the main repository. As usual it is on my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) but this time I'm adding the commit number in the file name.

This is the build that has been tested rather extensively by two users, David_Hugh and andy kh. I'm seeing some significant differences between the two sets of tests.

David_Hugh FRSP
(https://farm5.staticflickr.com/4781/39000698290_7e8292fd11.jpg) (https://flic.kr/p/22qmLQA)

David_Hugh FRSP lossless
(https://farm5.staticflickr.com/4782/39915724725_04b9ffc323.jpg) (https://flic.kr/p/23Pdwjr)

andy kh FRSP
(https://farm5.staticflickr.com/4776/39000811410_8ae0d73410.jpg) (https://flic.kr/p/22qnmsW)

andy kh FRSP lossless
(https://farm5.staticflickr.com/4795/40810713191_a077f26912.jpg) (https://flic.kr/p/25bizd4)

I examined the metadata with exiftool and the notable difference that I found was that David had his camera set on ISO 640 while Andy used ISO 3200.

The difference with simple silent is even more dramatic:

David_Hugh 5x zoom
(https://farm5.staticflickr.com/4782/39000931380_88314faa46.jpg) (https://flic.kr/p/22qnY8o)

David_Hugh 5x zoom lossless
(https://farm5.staticflickr.com/4780/26941302338_6769585a76.jpg) (https://flic.kr/p/H3HdyJ)

andy kh 5x zoom
(https://farm5.staticflickr.com/4778/25939465247_ba25f78634.jpg) (https://flic.kr/p/Fwbxir)

andy kh 5x zoom lossless
(https://farm5.staticflickr.com/4775/26941191288_15e423e89e.jpg) (https://flic.kr/p/H3GDy5)

Looking at the log diff (https://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-6D-vs-70D.html) there seems to be other things we can try. Any suggestions?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 14, 2018, 09:53:03 PM
Let's try just this today:

    if (is_camera("70D", "*"))
    {
        /* 70D is different */
        EngDrvOut(0xC0F37300, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
        EngDrvOut(0xC0F373E8, PACK32(width    - 1,  height/2  - 1));  /* 0xE7B0ADF on 70D */
+       EngDrvOut(0xC0F26908, 0x12369168); /* 0x1235FFAA on 70D */
    }


On my downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). You can compare your results with the crop_rec_4k.57614b3 changest build that was extensively tested.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 15, 2018, 01:11:35 AM
Do not try that!

edit: whew, it's a read channel. The address 0x12345678 was something that should only be tried in QEMU, never on real hardware (who knows what actually is at that address). If that was a write channel, you could have written into some memory region managed by other code, with completely unpredictable side effects (ranging from ¯\_(ツ)_/¯ to anything else).

edit: ok, it's overwritten later, so luckily the above change is a NOP. Will do the math later to explain what that does.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 15, 2018, 03:43:36 AM
So nobody was brave enough to try it? There was one download.

Took it from the diff log:

       6D                                                  70D
153   [0xC0F26908] <- 0x12369168: RAM address   n   153   [0xC0F26908] <- 0x1235FFAA: RAM address
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on March 15, 2018, 06:02:18 AM
A little more process , I can almost save a simple lossless dng -- much further then the frsp
I actually say saving 1 of 1 then freezes , battery pull needed still no dng save to card .

(https://preview.ibb.co/jQKnqH/IMG_2374.png) (https://ibb.co/kwBdxx)

I added

if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1)); 
        EngDrvOut(0xC0F12020, PACK32(width    - 1,  height/10 - 1));  /* 0x18A0127 */
    }

not sure if this making any difference with simple , but no difference for sure with FRSP
I also added this to raw.c as per dfort suggestion
#if defined(LV_STATE)  /* older Digic 4 */
#define BLACK_LEVEL 1024
#endif

mlv_Lite still is not being made , need to look further in to it
"EngDrvOut" was a shoot in the dark , need to do more investigation

Edit: Notice in photo mode that the simple dng always is set to 470 ISO even if I set frsp to 100 ISO , strange
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: David_Hugh on March 15, 2018, 10:09:11 AM
@dfort and a1ex

So is the build safe to test? Then I will do so as soon as I have time!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 15, 2018, 11:13:14 AM
Quote from: reddeercity on March 15, 2018, 06:02:18 AM
Notice in photo mode that the simple dng always is set to 470 ISO even if I set frsp to 100 ISO , strange

That sounds about right. Try setting the exposure time in LiveView to 1 second or longer, then compare with the exif info from the simple silent DNG.

Quote
Took it from the diff log

The diff log was made with this code (https://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987), in particular:

       .buffer = 0x12345678,


That was there because:
1) it's not trivial to get a valid address (you'd have to capture an image, this is a bit of hit and miss, depending on camera model)
2) I wanted to see only how the JPEG encoder is configured (the emulation won't go further anyway)
3) 0x12345678 is easily recognizable, so you can see what registers depend on the image buffer
4) it was the simplest way that worked on any camera model

But that 0x12345678 was not meant to be used on real hardware! There you'd have to allocate RAM for both input and output, fill the buffer with valid image data, fill the raw_info structure with correct parameters and so on.

Quote
So is the build safe to test?

This particular build is safe, but it will not make any difference (that change is overwritten later, so it's not worth testing it). Generally, with this kind of changes you should be very careful, because in our case, any task (or DMA channel) can write to any part of the memory, and Canon code reflashing things back into the ROM at shutdown is what makes this operation possibly dangerous.

Overwriting memory that you don't know what it does is a risky thing to do.
- if that memory is unused (not read after you have overwritten it), there will be no side effects at all (you may even think it's all OK, when it isn't)
- if that memory (which may contain data or code) is read after you have overwritten it, the side effects can range from:
  - temporary (e.g. some parameter changes for a split-second and is set back to the old value at the next iteration
  - noticeable (something changes on the screen, prints error, crashes etc)
  - insidious (the side effect is visible long after the overwriting, possibly on a totally unrelated feature)
  - domino (the code using the modified data gets confused and starts overwriting other stuff)
  - extreme (it's not impossible to eventually reach EraseSectorOfROM)
  - extremely insidious (modified data gets written into ROM and is noticed days/weeks/months later - this actually happened! (https://bitbucket.org/hudson/magic-lantern/pull-requests/825/prevent-canon-settings-from-being-saved/diff))

In this case, that register was the address of an EDMAC channel (which can read from or write to memory). Luckily, it was a read channel (something that reads from memory), so it's likely safe (not 100% safe - the image processor is a black box - but extremely unlikely to cause issues).

In our particular case:

[0xC0F26908] -> 0x1235FFAA: RAM address    ; somewhere in Canon code before our overrides
; our overrides (that big red block on 6D with CLOCK_ENABLE between the lines)
[0xC0F26908] <- 0x12345678: RAM address     ; after our overrides, value changed back by SetEDmac


Given that last line, any attempt to override this register has zero effects.

=> it's safe to try this particular build, but you will not notice any difference
=> it's generally not safe to try other builds with similar issues
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 15, 2018, 06:02:48 PM
I thought I was on to something because:

Quote from: a1ex on March 02, 2018, 10:09:48 PM
try overriding the highlighted registers and see if it helps.

       6D                                                  70D
153   [0xC0F26908] <- 0x12369168: RAM address   n   153   [0xC0F26908] <- 0x1235FFAA: RAM address

and:

0x12369168 != 0x12345678

but now that I take a closer look at the diff log I see that 0xC0F26908 comes up multiple times and is quite often 0x12345678. So it might be one of these cases:

Quote from: a1ex on March 15, 2018, 11:13:14 AM
  - temporary (e.g. some parameter changes for a split-second and is set back to the old value at the next iteration

This is the code that runs on the camera:

modules/silent/silent.c
static unsigned int silent_init()
{
    /* fixme in core: prop handlers should trigger when initializing, but they do not */
    prop_iso = lens_info.raw_iso;
    prop_shutter = lens_info.raw_shutter;

    silent_pic_mlv_available = (mlv_generate_guid() != 0ULL);

    if (is_camera("500D", "*") || is_camera("550D", "*") || is_camera("600D", "*"))
    {
        /* see http://www.magiclantern.fm/forum/index.php?topic=12523.msg129874#msg129874 */
        long_exposure_fix_enabled = 1;
    }

    menu_add("Shoot", silent_menu, COUNT(silent_menu));
    return 0;
}


but in order to run in QEMU it was changed to this:

static unsigned int silent_init()
{
    /* keep the old stuff */

    /* code for running only in QEMU - no error checking done, no valid input, no valid results expected */
    struct memSuite * out_suite = shoot_malloc_suite_contig(16 * 1024 * 1024);
    struct raw_info out_raw_info = {
       .buffer = 0x12345678,
       .width = 1920,
       .height = 1080,
    };
    lossless_compress_raw(&out_raw_info, out_suite);

    return 0;
}


My mistake is clearly evident so I took down yesterday's test build but maybe there is something here? If all cameras were tested with ".buffer = 0x12345678" what accounts for these differences in 0xC0F26908?

5D3
0x12347F28 - 0x12345678 = 0x028B0

100D/650D/700D/EOSM/EOSM2
0x12349EA8 - 0x12345678 = 0x04830

6D
0x12369168 - 0x12345678 = 0x23AF0

70D
0x1235FFAA - 0x12345678 = 0x1A932

Perhaps the only thing this exercise proves is that there is a group of cameras with something in common.

Back to trial and error--hopefully with a little less error. Anything we haven't tried on the 70D?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 15, 2018, 07:57:11 PM
By default, a CR2 is smaller than a full-res DNG, so Canon code must be skipping some lines and columns. From that offset, we can figure out how many. Refer to this post (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198113#msg198113) for EDMAC configurations.

6D: 0x12369168 - 0x12345678 = 146160 bytes. Full buffer width: 9744 bytes = 5568 pixels. CR2 width (dcraw -i -v): 5568. 146160 / 9744 = 15 lines skipped.

In other words, to match a full-res silent DNG with a CR2 from 6D, one has to crop 15 lines at the top.

70D: 0x1235FFAA - 0x12345678 = 108850 bytes. Full buffer width: 9884 bytes = 5648 pixels. CR2 width: 5568 pixels. 108850 / 9884 = 11, 108850 % 9884 = 126 = 72px.

In other words, to match a full-res silent DNG with a CR2 from 70D, one has to crop 11 lines at the top, 72 columns at the left side and 8 columns at the right side.

Anyone has the patience to confirm this theory by pixel peeping? If you can get two images with absolutely no movement (e.g. with a Lua script or remote trigger), that's great; otherwise, just compare the active areas and ignore the image contents.

edit: found a CR2 from 70D on some camera review site and looked at active areas.
First active pixel in CR2: 72, 38.
Last active pixel in CR2: 5567, 3707 (with some doubts about the last line).
First active pixel in DNG: 144, 48 (delta = 72, 10 - where did one line go?)
Last active pixel in DNG: 5639, 3717 (delta = 72, 10, 8 black pixels at the right)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 15, 2018, 08:57:25 PM
Sounds like a fun project for tomorrow, will check it for the 6d.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 16, 2018, 08:30:39 PM
On 6d:

Full resolution silent picture = 5496 x 3670 pixels
CR2 = 5472 x 3648 pixels

For as far as I can tell, the difference is as follows:
The FRSP has 12 pixels extra on the left side and 12 pixels extra on the right side
The FRSP has 9 pixels extra at the top and 13 extra pixels on the bottom

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 16, 2018, 08:39:28 PM
That's likely caused by difference in (declared) optical black areas (ActiveArea tag). My theory does not consider these at all (therefore, the check should be done with dcraw -E, or the equivalent setting from other software).
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 16, 2018, 08:54:55 PM
Did the 'dcraw -E' command on both pictures and got two grayscale images including black borders:

FRSP = 5568 x 3722 pixels
CR2 = 5568 x 3708 pixels

Is that what you meant ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Levas on March 16, 2018, 09:00:37 PM
The FRSP has some data in the top 'black border' across the whole width of the image and about 4 pixels in size vertical, do you know where that is used for by Canon ?

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 16, 2018, 09:23:23 PM
No idea where it's used. I might find out after emulating the photo capture process (including lossless encoding and CR2 creation).

Remember those huge logs full of registers (https://www.magiclantern.fm/forum/index.php?topic=2388.msg197313#msg197313)? Managed to replay 7 (https://www.magiclantern.fm/forum/index.php?topic=2388.msg198340#msg198340) of them :D

That was one tiny step towards emulating complex stuff like LiveView or photo capture.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: lorenzo353 on March 29, 2018, 09:25:17 PM
Hi,

you've got all info here, per camera
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_database.txt



Quote from: a1ex on March 15, 2018, 07:57:11 PM
By default, a CR2 is smaller than a full-res DNG, so Canon code must be skipping some lines and columns. From that offset, we can figure out how many. Refer to this post (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198113#msg198113) for EDMAC configurations.

6D: 0x12369168 - 0x12345678 = 146160 bytes. Full buffer width: 9744 bytes = 5568 pixels. CR2 width (dcraw -i -v): 5568. 146160 / 9744 = 15 lines skipped.

In other words, to match a full-res silent DNG with a CR2 from 6D, one has to crop 15 lines at the top.

70D: 0x1235FFAA - 0x12345678 = 108850 bytes. Full buffer width: 9884 bytes = 5648 pixels. CR2 width: 5568 pixels. 108850 / 9884 = 11, 108850 % 9884 = 126 = 72px.

In other words, to match a full-res silent DNG with a CR2 from 70D, one has to crop 11 lines at the top, 72 columns at the left side and 8 columns at the right side.

Anyone has the patience to confirm this theory by pixel peeping? If you can get two images with absolutely no movement (e.g. with a Lua script or remote trigger), that's great; otherwise, just compare the active areas and ignore the image contents.

edit: found a CR2 from 70D on some camera review site and looked at active areas.
First active pixel in CR2: 72, 38.
Last active pixel in CR2: 5567, 3707 (with some doubts about the last line).
First active pixel in DNG: 144, 48 (delta = 72, 10 - where did one line go?)
Last active pixel in DNG: 5639, 3717 (delta = 72, 10, 8 black pixels at the right)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on March 30, 2018, 05:44:23 PM
Quote from: lorenzo353 on March 29, 2018, 09:25:17 PM
you've got all info here, per camera
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_database.txt

I made a spreadsheet out of it, took out the PowerShot models, sorted by Model ID, added the name variations and highlighted the cameras supported by Magic Lantern.

https://docs.google.com/spreadsheets/d/1vMcOUh13TVOV3GY2576ke8rxiLn8qiMN9q57BEAXk-E/edit?usp=sharing

Interesting that some cameras are listed multiple times with different values. Why???
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: lorenzo353 on March 30, 2018, 07:39:48 PM
hi,
there are lines for sraw flavors
http://lclevy.free.fr/cr2/#sraw

Quote from: dfort on March 30, 2018, 05:44:23 PM
I made a spreadsheet out of it, took out the PowerShot models, sorted by Model ID, added the name variations and highlighted the cameras supported by Magic Lantern.

https://docs.google.com/spreadsheets/d/1vMcOUh13TVOV3GY2576ke8rxiLn8qiMN9q57BEAXk-E/edit?usp=sharing

Interesting that some cameras are listed multiple times with different values. Why???
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on April 03, 2018, 06:49:08 AM
5d2 disassembly , found some path stuff:
there's a lot of this "OneOut"  then go to "TwoOut" etc....  just a jpeg where as the TwoOut  raw+jpeg+thumb image ?
*"PathTwoInOneOut"
*"DevelopPath\TwoInOneOutDevelopPath.c"


*"[DEV] JpegEncodePath_ChangeCBR(ID=%#lx)"
*"[DPATH] JPG_EN START J:%#lx Y:%#lx C:%#lx"
*"[DPATH] JPG_EN PONY Enable(%d)"
*"[DPATH] JPG_EN LUCKY Enable(%d)(SEL:%d)"

Start , Pony , Lucky ?
Ok , some strange names for sure.

"***** DPU_GetJpegOrRawFullPath arraySize(%d)":
*"***** DPU_GetJpegOrRawFullPath FM_GetFilePath JPEG"
*"***** DPU_GetJpegOrRawFullPath FM_GetFilePath RAW"

kind of fits in to the "TwoInOneOut" or is that "TwoOut"

*"===sCroppedImagePath(%s)"
*"***** DPU_CalcTrimImageHeightForBorderless input data = 0"

not sure if this has anything with lossless , look interesting thou.

*"[RD ERROR] ProcessRawDecodePath failed [%#x]"
*"[RD ERROR] ProcessJpegEncodePath failed [%#x]"
*" ClearDecodeJpegPath %d"
*"TWIN StopDecWinPath Error:%d"


could this be the lossless decode path ?

Time to fire up QEMU and do some investigating
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on April 25, 2018, 12:16:12 AM
Had a look at the cr2 & frsp .dng with dcraw -i -v had some confusing info
C:\dcraw>dcraw -i -v _MG_8665.CR2

Filename: _MG_8665.CR2
Timestamp: Sun Oct 29 19:01:35 2017
Camera: Canon EOS 5D Mark II
Owner:
ISO speed: 800
Shutter: 1/49.4 sec
Aperture: f/4.0
Focal length: 24.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  5616 x 3744
Full size:   5792 x 3804
Image size:  5634 x 3753
Output size: 5634 x 3753
Raw colors: 3
Filter pattern: RG/GB
Daylight multipliers: 2.391381 0.929156 1.289254
Camera multipliers: 2153.000000 1024.000000 1817.000000 1024.000000

C:\dcraw>dcraw -i -v 86630000.DNG

Filename: 86630000.DNG
Timestamp: Thu Jan 01 00:00:00 1970
Camera: Canon EOS 5D Mark II
DNG Version: 1.3.0.0
ISO speed: 0
Shutter: 1.0 sec
Aperture: f/1.0
Focal length: 0.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:   128 x 84
Full size:   5792 x 3804
Image size:  5632 x 3752
Output size: 5632 x 3752
Raw colors: 3
Filter pattern: RG/GB
Daylight multipliers: 2.391381 0.929156 1.289254
Camera multipliers: 2.111331 1.000000 1.602564 0.000000

One of the biggest difference is the thumb image , cr2 full res (5616 x 3744) & ML writes a small thumb (128 x 84)
but that's not what confuses me , it's the "Camera multipliers" there very different
CR2
Camera multipliers: 2153.000000 1024.000000 1817.000000 1024.000000

FRSP DNG
Camera multipliers: 2.111331 1.000000 1.602564 0.000000

thou the "Daylight multipliers" are the same
2.391381 0.929156 1.289254
Is this the same on all FRSP ? are with d4 cam's only  ?
Could lossless be looking for the  CR2  "Camera multipliers" and crashes on 5d2/D4 ? or is this a result of FRSP for the gradation from top to bottom ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 04, 2018, 08:53:28 AM
I may have found  the "resources" for lossless , read this (https://www.magiclantern.fm/forum/index.php?topic=2864.msg204841#msg204841) post for the LOGs
76146> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942d70) x3 from ffa0b7b0:
76188> LiveViewMg:00096678:00:00:      1)    6000f (?)
761B9> LiveViewMg:00096678:00:00:      2)    20003 (write connection 0x3)
761E7> LiveViewMg:00096678:00:00:      3)        1 (write channel 0x1)
76234> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76272> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
762B1> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76325> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942db8) x7 from ffa0b7d4:
76366> LiveViewMg:00096678:00:00:      1)    60005 (?)
7638D> LiveViewMg:00096678:00:00:      2)    6000c (?)
763AF> LiveViewMg:00096678:00:00:      3)   1d0000 (?)
763D5> LiveViewMg:00096678:00:00:      4)   220016 (?)
763FB> LiveViewMg:00096678:00:00:      5)   220017 (?)
7641E> LiveViewMg:00096678:00:00:      6)   220027 (?)
76444> LiveViewMg:00096678:00:00:      7)   220028 (?)
76488> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
764C6> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
764F9> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76579> LiveViewMg:ffa0b3e0:99:02: ClearWbPass
765BC> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(92b440) x7 from ff8e79e0:
765FA> LiveViewMg:00096678:00:00:      1)   130009 (?)
76622> LiveViewMg:00096678:00:00:      2)   13000a (?)
76646> LiveViewMg:00096678:00:00:      3)   13000b (?)
76670> LiveViewMg:00096678:00:00:      4)    20002 (write connection 0x2)
7669A> LiveViewMg:00096678:00:00:      5)        2 (write channel 0x2)
766BA> LiveViewMg:00096678:00:00:      6)    70000 (?)
766DC> LiveViewMg:00096678:00:00:      7)    f0000 (?)
7671C> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76760> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76792> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
767BF> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76823> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(92b498) x1 from ff8e79e0:
76864> LiveViewMg:00096678:00:00:      1)    b0000 (?)
7689C> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76901> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(92b4d8) x2 from ff8e79e0:
76945> LiveViewMg:00096678:00:00:      1)    c0000 (?)
76969> LiveViewMg:00096678:00:00:      2)   120000 (?)
769A1> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
769DD> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
76A3C> **INT-50h*:ff871ad4:00:01: [PM] DisablePowerSave (Counter = 4)
76A76> **INT-50h*:ff871b44:00:01: [PM] EnablePowerSave (Counter = 3)
76AA3> **INT-50h*:FF99F344:MMIO:  [0xC022009C] -> 0x00000047
76ACB> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76AFC> LiveViewMg:ff9a4810:00:02: [ENG] UnregisterEDmacAbortCBR(8)
76B38> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76B4F> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76B87> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942bf0) x54 from ff8e79e0:
76BCC> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76BE1> LiveViewMg:00096678:00:00:      1)   100000 (?)
76C2B> **INT-36h*:000969b4:00:00: *** mpu_recv(0a 08 09 05 00 80 00 00 00), from ff99f6cc
76C6B> **INT-36h*:00096840:00:00: *** TryPostEvent(PropMgr, 0xa, 0x51a59c, 0x14), from ff85ec40
76CD2> **INT-36h*:FF99F3E0:MMIO:  [0xC022009C] -> 0x00000047
76CF0> LiveViewMg:00096678:00:00:      2)    50013 (?)
76D25> LiveViewMg:00096678:00:00:      3)    50001 (?)
76D4F> LiveViewMg:00096678:00:00:      4)    5000f (?)
76D74> LiveViewMg:00096678:00:00:      5)    5001a (?)
76D96> LiveViewMg:00096678:00:00:      6)    5000e (?)
76DB8> LiveViewMg:00096678:00:00:      7)    50019 (?)
76DDB> LiveViewMg:00096678:00:00:      8)    50007 (?)
76DFE> LiveViewMg:00096678:00:00:      9)    50009 (?)
76E20> LiveViewMg:00096678:00:00:     10)    50006 (?)
76E42> LiveViewMg:00096678:00:00:     11)    5000d (?)
76E64> LiveViewMg:00096678:00:00:     12)    50005 (?)
76E87> LiveViewMg:00096678:00:00:     13)    80000 (?)
76EAD> LiveViewMg:00096678:00:00:     14)    d0000 (?)
76ECF> LiveViewMg:00096678:00:00:     15)    e0000 (?)
76EF4> LiveViewMg:00096678:00:00:     16)   150000 (?)
76F17> LiveViewMg:00096678:00:00:     17)    60003 (?)
76F3A> LiveViewMg:00096678:00:00:     18)    6000b (?)
76F5F> LiveViewMg:00096678:00:00:     19)    60007 (?)
76F81> LiveViewMg:00096678:00:00:     20)    60009 (?)
76FA2> LiveViewMg:00096678:00:00:     21)    60010 (?)
76FC5> LiveViewMg:00096678:00:00:     22)    60014 (?)
76FEA> LiveViewMg:00096678:00:00:     23)    60013 (?)
7700D> LiveViewMg:00096678:00:00:     24)   130000 (?)
77034> LiveViewMg:00096678:00:00:     25)   130001 (?)
77059> LiveViewMg:00096678:00:00:     26)   130006 (?)
7707E> LiveViewMg:00096678:00:00:     27)   130007 (?)
770A2> LiveViewMg:00096678:00:00:     28)   130008 (?)
770C6> LiveViewMg:00096678:00:00:     29)   13000c (?)
770EC> LiveViewMg:00096678:00:00:     30)   13000d (?)
77112> LiveViewMg:00096678:00:00:     31)   13000e (?)
77136> LiveViewMg:00096678:00:00:     32)   13000f (?)
7715D> LiveViewMg:00096678:00:00:     33)   130010 (?)
77183> LiveViewMg:00096678:00:00:     34)   130016 (?)
771A6> LiveViewMg:00096678:00:00:     35)   130017 (?)
771CA> LiveViewMg:00096678:00:00:     36)   130019 (?)
771F0> LiveViewMg:00096678:00:00:     37)   13001a (?)
77216> LiveViewMg:00096678:00:00:     38)   13001b (?)
7723A> LiveViewMg:00096678:00:00:     39)   220000 (?)
7725E> LiveViewMg:00096678:00:00:     40)   220001 (?)
77281> LiveViewMg:00096678:00:00:     41)   220012 (?)
772A5> LiveViewMg:00096678:00:00:     42)   220013 (?)
772CA> LiveViewMg:00096678:00:00:     43)   220014 (?)
772F2> LiveViewMg:00096678:00:00:     44)   220015 (?)
7731D> LiveViewMg:00096678:00:00:     45)    20004 (write connection 0x4)
77349> LiveViewMg:00096678:00:00:     46)    20000 (write connection 0x0)
77372> LiveViewMg:00096678:00:00:     47)    30008 (read connection 0x8)
7739E> LiveViewMg:00096678:00:00:     48)    3000f (read connection 0xf)
773C9> LiveViewMg:00096678:00:00:     49)        9 (write channel 0x12)
773F1> LiveViewMg:00096678:00:00:     50)    10001 (read channel 0x9)
7741E> LiveViewMg:00096678:00:00:     51)    10000 (read channel 0x8)
77447> LiveViewMg:00096678:00:00:     52)    50011 (?)
7746A> LiveViewMg:00096678:00:00:     53)    5001b (?)
7748C> LiveViewMg:00096678:00:00:     54)    5001e (?)
774DA> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
  ............
77B25> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
77B55> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
77BDB> LiveViewMg:000965bc:00:00: *** UnLockEngineResources(942d04) x12 from ff8e79e0:
77C22> LiveViewMg:00096678:00:00:      1)    6000e (?)
77C4A> LiveViewMg:00096678:00:00:      2)    6000d (?)
77C6D> LiveViewMg:00096678:00:00:      3)    60015 (?)
77C90> LiveViewMg:00096678:00:00:      4)   180000 (?)
77CB3> LiveViewMg:00096678:00:00:      5)   130002 (?)
77CD6> LiveViewMg:00096678:00:00:      6)   130005 (?)
77CF9> LiveViewMg:00096678:00:00:      7)   130012 (?)
77D1B> LiveViewMg:00096678:00:00:      8)   130014 (?)
77D3E> LiveViewMg:00096678:00:00:      9)   130018 (?)
77D62> LiveViewMg:00096678:00:00:     10)   13001c (?)
77D90> LiveViewMg:00096678:00:00:     11)    20015 (write connection 0x15)
77DBE> LiveViewMg:00096678:00:00:     12)        0 (write channel 0x0)
77DF9> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
...........
77F24> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��


Also this
822AC> ShootCaptu:ff87ffec:93:03: GetCaptureResources start.
822D4> ShootCaptu:000965bc:00:00: *** LockEngineResources(680404) x38 from ffa384f8:
8230D> ShootCaptu:00096678:00:00:      1)    50001 (?)
82332> ShootCaptu:00096678:00:00:      2)    50005 (?)
82353> ShootCaptu:00096678:00:00:      3)    50007 (?)
82374> ShootCaptu:00096678:00:00:      4)    50009 (?)
82397> ShootCaptu:00096678:00:00:      5)    5000e (?)
823B7> ShootCaptu:00096678:00:00:      6)    50011 (?)
823D7> ShootCaptu:00096678:00:00:      7)    50019 (?)
823F7> ShootCaptu:00096678:00:00:      8)    5001b (?)
82419> ShootCaptu:00096678:00:00:      9)    5001e (?)
8243B> ShootCaptu:00096678:00:00:     10)    50013 (?)
8245C> ShootCaptu:00096678:00:00:     11)    50020 (?)
8247D> ShootCaptu:00096678:00:00:     12)    50021 (?)
8249F> ShootCaptu:00096678:00:00:     13)    70000 (?)
824C1> ShootCaptu:00096678:00:00:     14)    f0000 (?)
824E2> ShootCaptu:00096678:00:00:     15)    b0000 (?)
82505> ShootCaptu:00096678:00:00:     16)   200002 (?)
82528> ShootCaptu:00096678:00:00:     17)   140000 (?)
8254C> ShootCaptu:00096678:00:00:     18)   120000 (?)
82570> ShootCaptu:00096678:00:00:     19)   130009 (?)
82593> ShootCaptu:00096678:00:00:     20)   13000a (?)
825B9> ShootCaptu:00096678:00:00:     21)   13000b (?)
825DB> ShootCaptu:00096678:00:00:     22)   13000d (?)
825FE> ShootCaptu:00096678:00:00:     23)   13000e (?)
82621> ShootCaptu:00096678:00:00:     24)   13000f (?)
82645> ShootCaptu:00096678:00:00:     25)   130016 (?)
82669> ShootCaptu:00096678:00:00:     26)   22001b (?)
82695> ShootCaptu:00096678:00:00:     27)        0 (write channel 0x0)
826BC> ShootCaptu:00096678:00:00:     28)        1 (write channel 0x1)
826E0> ShootCaptu:00096678:00:00:     29)        2 (write channel 0x2)
82704> ShootCaptu:00096678:00:00:     30)        6 (write channel 0x6)
82728> ShootCaptu:00096678:00:00:     31)    10000 (read channel 0x8)
82750> ShootCaptu:00096678:00:00:     32)    10001 (read channel 0x9)
82779> ShootCaptu:00096678:00:00:     33)    20000 (write connection 0x0)
827A4> ShootCaptu:00096678:00:00:     34)    20002 (write connection 0x2)
827CD> ShootCaptu:00096678:00:00:     35)    20012 (write connection 0x12)
827F9> ShootCaptu:00096678:00:00:     36)    2001c (write connection 0x1c)
82822> ShootCaptu:00096678:00:00:     37)    30008 (read connection 0x8)
8284B> ShootCaptu:00096678:00:00:     38)    3000f (read connection 0xf)
828C1> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
82908> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
................
82DDA> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
82E08> ShootCaptu:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
82E36> ShootCaptu:ff880004:93:03: GetCaptureResources end.


Cr2 lossless ?
EEAF8>     RscMgr:ff8b65c4:80:01: Allocate MEM3 0 0
EEB1F>     RscMgr:ff8b6670:80:01: OK AllocateMEM3 0xBAA4C0(876544)(1)
EEBD1> FrontShtDe:ff888cc0:96:05: sdfExecuteMem1ToRawPath(5742)
EEC5E> FrontShtDe:ff888ea8:96:05: sdfExecuteMem1ToRawPath(5742)��(SemOK)
EEC9A> FrontShtDe:ff888ee0:96:05: ProcessTwoInTwoOutJpegPath(R) Start(5742)
EECE9> FrontShtDe:000965bc:00:00: *** LockEngineResources(72bed4) x17 from ffa59b1c:
EED33> FrontShtDe:00096678:00:00:      1)    10002 (read channel 0xa)
EED62> FrontShtDe:00096678:00:00:      2)        3 (write channel 0x3)
EED88> FrontShtDe:00096678:00:00:      3)        4 (write channel 0x4)
EEDAE> FrontShtDe:00096678:00:00:      4)    30000 (read connection 0x0)
EEDD8> FrontShtDe:00096678:00:00:      5)    20005 (write connection 0x5)
EEE03> FrontShtDe:00096678:00:00:      6)    20016 (write connection 0x16)
EEE26> FrontShtDe:00096678:00:00:      7)    50003 (?)
EEE48> FrontShtDe:00096678:00:00:      8)    5000d (?)
EEE69> FrontShtDe:00096678:00:00:      9)    5000f (?)
EEE8B> FrontShtDe:00096678:00:00:     10)    5001a (?)
EEEAC> FrontShtDe:00096678:00:00:     11)    80000 (?)
EEECE> FrontShtDe:00096678:00:00:     12)    90000 (?)
EEEEF> FrontShtDe:00096678:00:00:     13)    a0000 (?)
EEF10> FrontShtDe:00096678:00:00:     14)   160000 (?)
EEF33> FrontShtDe:00096678:00:00:     15)   130003 (?)
EEF58> FrontShtDe:00096678:00:00:     16)   130004 (?)
EEF7C> FrontShtDe:00096678:00:00:     17)   130005 (?)
EF007> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF048> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF07C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF0AE> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF0DD> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF10C> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF140> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF173> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF1A1> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF1CF> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF1FD> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF227> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF250> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF27E> FrontShtDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
EF2C1> FrontShtDe:ffa59694:16:03: [TTJ][150,5742,0] RAW(5792,3804,0,14)
EF366> FrontShtDe:00096224:00:00: *** ConnectReadEDmac(0xa, 0x0), from ffa59770
EF3C5> FrontShtDe:00096224:00:00: *** RegisterEDmacCompleteCBR(0xa, 0xffa593f8 "[TTJ][%d,%d,%d] Read1CompleteCBR", 0x0), from ffa59780
EF415> FrontShtDe:00096224:00:00: *** ConnectWriteEDmac(0x3, 0x5), from ffa597f8
EF4F6> FrontShtDe:00096224:00:00: *** RegisterEDmacCompleteCBR(0x3, 0xffa9285c "[PackMem] CompleteInterrupt In %d", 0x3), from ffa92be0
EF522> FrontShtDe:ff9a47e0:00:02: [ENG] RegisterEDmacAbortCBR(3)
EF572> FrontShtDe:00096224:00:00: *** ConnectWriteEDmac(0x4, 0x16), from ffa59830
EF5D1> FrontShtDe:00096224:00:00: *** RegisterEDmacCompleteCBR(0x4, 0xffa59340 "[TTJ][%d,%d,%d] Write2CompleteCBR", 0x0), from ffa59840
EF600> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x200807c WR2:0x1acac0f0
EF639> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x10000048 RD2:0x124d1864


DBAF3> ShootPreDe:ffa5a5ac:16:03: [FB] START WR7:0x1f50000 RD4:0x124e9de4
DC2DA> ShootPreDe:ffa5a158:16:03: [FB] STOP WR7:0x1f5ccc0 RD4:0x1f5c8a4
DFBC5> ShootPreDe:ffa5a270:16:03: [FC] STOP RD4:0x1f54b00
DFC9E> ShootPreDe:ff9a4810:00:02: [ENG] UnregisterEDmacAbortCBR(12)
DFCE8> ShootPreDe:000965bc:00:00: *** UnLockEngineResources(72a764) x5 from ffa5a4bc:
DFD27> ShootPreDe:00096678:00:00:      1)   140001 (?)
DFD55> ShootPreDe:00096678:00:00:      2)        7 (write channel 0x10)
DFD7B> ShootPreDe:00096678:00:00:      3)    10004 (read channel 0xc)
DFDA6> ShootPreDe:00096678:00:00:      4)    20017 (write connection 0x17)
DFDD2> ShootPreDe:00096678:00:00:      5)    3000e (read connection 0xe)
DFE20> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFE6C> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFEA3> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFED4> ShootPreDe:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
DFF0F> ShootPreDe:ff8852f8:95:05: Face Num : 0, Face All Num : 0
DFFA7> ShootPreDe:00096098:00:00: *** SPSState: (3) --4--> (2)          ff885408 (x=68187c z=0 t=0)
DFFFB> ShootPreDe:ff885448:95:05: spsStartProcess(5742)
E8E49> ShootPreDe:ff884154:95:05: WB RectH:(2960, 2992)
E8E97> ShootPreDe:ff884174:95:05: WB RectV:(1911, 1943)
E8EC3> ShootPreDe:ff884190:95:05: WB Limit:(H:11464) (L:1072) )
E90B5> ShootPreDe:ff884564:95:05: WB Num=2
E90E1> ShootPreDe:ff88458c:95:05: WB R =3831(1915)
E9109> ShootPreDe:ff8845b8:95:05: WB G1=2060(1030)
E912E> ShootPreDe:ff8845e4:95:05: WB G2=1919(959)
E9154> ShootPreDe:ff884610:95:05: WB B =4039(2019)
E919D> ShootPreDe:ff885aa4:95:05: ISO_DIGITAL_GAIN_1


I havn't tired to implement any of the above info in to the lossless code yet , need to exam the log's closer , anyone that's want to help
get 5D2 lossless working feel free to jump in any time .


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 05, 2018, 04:24:49 AM
For 5d2 , looks like I found the same resources as in this code from dfort , which if I'm not mistaken came from the 7D #lossless.c-274 (https://bitbucket.org/daniel_fort/magic-lantern/src/6991077b527377f56fd9130266b0dfb634b12f94/modules/silent/lossless.c?at=unified&fileviewer=file-view-default#lossless.c-274)
EED33> FrontShtDe:00096678:00:00:      1)    10002 (read channel 0xa)
EED62> FrontShtDe:00096678:00:00:      2)        3 (write channel 0x3)
EED88> FrontShtDe:00096678:00:00:      3)        4 (write channel 0x4)
EEDAE> FrontShtDe:00096678:00:00:      4)    30000 (read connection 0x0)
EEDD8> FrontShtDe:00096678:00:00:      5)    20005 (write connection 0x5)
EEE03> FrontShtDe:00096678:00:00:      6)    20016 (write connection 0x16)
EEE26> FrontShtDe:00096678:00:00:      7)    50003 (?)
EEE48> FrontShtDe:00096678:00:00:      8)    5000d (?)
EEE69> FrontShtDe:00096678:00:00:      9)    5000f (?)
EEE8B> FrontShtDe:00096678:00:00:     10)    5001a (?)
EEEAC> FrontShtDe:00096678:00:00:     11)    80000 (?)
EEECE> FrontShtDe:00096678:00:00:     12)    90000 (?)
EEEEF> FrontShtDe:00096678:00:00:     13)    a0000 (?)
EEF10> FrontShtDe:00096678:00:00:     14)   160000 (?)
EEF33> FrontShtDe:00096678:00:00:     15)   130003 (?)
EEF58> FrontShtDe:00096678:00:00:     16)   130004 (?)
EEF7C> FrontShtDe:00096678:00:00:     17)   130005 (?)


Below I have a strong believe that this could be the missing info , I was using the 7D value that did not work even on the 7D in lossless . lossless.c-253 (https://bitbucket.org/daniel_fort/magic-lantern/src/6991077b527377f56fd9130266b0dfb634b12f94/modules/silent/lossless.c?at=unified&fileviewer=file-view-default#lossless.c-253)
from the 5d3 code I know I need , 1 "read connection" & 2 "write connections"
1CB6C> FrontShtDe:00096678:00:00:      5)    30000 (read connection 0x0)
1CB95> FrontShtDe:00096678:00:00:      6)    30021 (read connection 0x21) **** I think this one could be removed totally****
1CBBD> FrontShtDe:00096678:00:00:      7)    20005 (write connection 0x5)
1CBE5> FrontShtDe:00096678:00:00:      8)    20003 (write connection 0x3)

so I'm thinking read 0x0 , write on 0x5 & 0x3 , haven't try it yet , I want to do same some investigation with  io_trace_full branch first to get more info on this .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 13, 2018, 06:58:03 AM
Volunteers needed  :) 
I'm looking for some users with digic4 cams to take (capture) a cr2 photo outside of Liveview with a special "io-trace-full" ml build for log info in to the "TwoInTwoOutLosslessPath"
and also outside of Liveview review a cr2 photo but not on the same Log as the captured cr2 . To run test , load ml io-trace-full build (link for cams are below)
you only need the "5D2_212.sym" file (your cam will be different e.g. 7d_703)  exit liveview go to the debug menu activate the Logging feature
-- exit menu so there no liveview take a cr2 photo return to debug menu and hi-light logging again to save the log . Do the same for review cr2 photo , this will help understand the Jpeg raw compression process as it's not the same as digic5 so it's a bit of a mystery and hopefully we can finally get d4 's to be supported in lossless .
io-trace-full_magiclantern-Nightly.2018Aug11.7D203.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/io-trace-full_magiclantern-Nightly.2018Aug11.7D203.zip)
io-trace-full_magiclantern-Nightly.2018Aug11.50D109.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/io-trace-full_magiclantern-Nightly.2018Aug11.50D109.zip) 
io-trace-full_magiclantern-Nightly.2018Aug11.550D109.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/io-trace-full_magiclantern-Nightly.2018Aug11.550D109.zip)
io-trace-full_magiclantern-Nightly.2018Aug11.500D111.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/io-trace-full_magiclantern-Nightly.2018Aug11.500D111.zip)
As you may have guessed I've already completed the Log's for 5D2
but for anyone that what to reproduce my results here the build for the 5D2 io_trace_full_magiclantern-Nightly.2018Aug04.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/io_trace_full_magiclantern-Nightly.2018Aug04.5D2212.zip)
if there any problems with the builds please let me know
Thanks to all who wish to push digic4 into Lossless  :D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 13, 2018, 02:41:30 PM
Here are logs from 50D: Cr2 capture (https://drive.google.com/file/d/1FHwOBrdKSGfu07FkigzkTLkQl3pDmmOd/view?usp=sharing) and Cr2 Review (https://drive.google.com/file/d/1OxUum797eu4mfsG3nRDsv5qNM6DEeQNu/view?usp=sharing)

They were taken without entering LiveView.
Hope they are good, let me know if you need others.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 13, 2018, 06:27:02 PM
They are very good, thanks a lot.

Nitpicks:
- figure out why there's a bunch of zeros at the beginning of the log (likely some bug)
- load edmac.mo during the experiment (the log contains some lines telling you to do so). It's a nitpick because the information printed is a bit redundant (I can figure it out from MMIO lines), but it's still nice to have.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 18, 2018, 02:52:26 AM
Quote from: a1ex on August 13, 2018, 06:27:02 PM
- load edmac.mo during the experiment (the log contains some lines telling you to do so). It's a nitpick because the information printed is a bit redundant (I can figure it out from MMIO lines), but it's still nice to have.

Here is the same procedure but with edmac.mo copied from nightly (as the io-trace-full build doesn't have it):
Cr2TakeEdmac (https://drive.google.com/file/d/1l47NpMrvmrTsD_eGEkH7a519oOUemd9y/view?usp=sharing) and Cr2ReviewEdmac (https://drive.google.com/file/d/1bubG8YSa6c-yU4VkSOclU9NA5IIXlBok/view?usp=sharing)

Also no bunch of zeroes int these logs... Don't know why it happened previously

Had a look to find read and write channels and the appears a bit different from reddeercity's logs
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: koljanych on August 18, 2018, 07:38:54 PM
Quote from: reddeercity on August 13, 2018, 06:58:03 AM
io-trace-full_magiclantern-Nightly.2018Aug11.550D109.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/io-trace-full_magiclantern-Nightly.2018Aug11.550D109.zip)

https://yadi.sk/d/NlOlpbms3aMHrB - capture
https://yadi.sk/d/H7dkA4SA3aMHrC - view
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 19, 2018, 02:22:51 AM
Thanks for the 550D LOG's  :)
they look very close to my 5d2 LOG's if not the same .
Looks like we my have a winner here for Lossless !
11.533.527  FrontShtDe:ff08a33c:96:05: sdfExecuteMem1ToRawPath(49091)��(SemOK)
11.533.551  FrontShtDe:ff08a374:96:05: ProcessTwoInTwoOutJpegPath(R) Start(49091)
11.533.592  FrontShtDe:ff06a018:00:00: *** LockEngineResources(8b0a14) x17:
11.533.620  FrontShtDe:00cb97fc:00:00:      1)    10002 (read channel 0xa)
11.533.636  FrontShtDe:00cb97fc:00:00:      2)        3 (write channel 0x3)
11.533.650  FrontShtDe:00cb97fc:00:00:      3)        4 (write channel 0x4)
11.533.670  FrontShtDe:00cb97fc:00:00:      4)    30000 (read connection 0x0)
11.533.689  FrontShtDe:00cb97fc:00:00:      5)    20005 (write connection 0x5)
11.533.709  FrontShtDe:00cb97fc:00:00:      6)    20016 (write connection 0x16)
11.533.721  FrontShtDe:00cb97fc:00:00:      7)    50003 (?)
11.533.734  FrontShtDe:00cb97fc:00:00:      8)    5000d (?)
11.533.748  FrontShtDe:00cb97fc:00:00:      9)    5000f (?)
11.533.760  FrontShtDe:00cb97fc:00:00:     10)    5001a (?)
11.533.772  FrontShtDe:00cb97fc:00:00:     11)    80000 (?)
11.533.784  FrontShtDe:00cb97fc:00:00:     12)    90000 (?)
11.533.797  FrontShtDe:00cb97fc:00:00:     13)    a0000 (?)
11.533.812  FrontShtDe:00cb97fc:00:00:     14)   160000 (?)
11.533.826  FrontShtDe:00cb97fc:00:00:     15)   130003 (?)
11.533.841  FrontShtDe:00cb97fc:00:00:     16)   130004 (?)
11.533.855  FrontShtDe:00cb97fc:00:00:     17)   130005 (?)


So any one with a 550D can jump in and start to dev. lossless , just look back a few page's for code on digic 4
here the code I'm working from --crop_rec_4k_Digic4_Lossless.c (https://bitbucket.org/daniel_fort/magic-lantern/src/4286b6c7b7554535273df9ab5d6ac45a764d3617/modules/silent/lossless.c?at=crop_rec_4k_Digic4&fileviewer=file-view-default) and this post  here (https://www.magiclantern.fm/forum/index.php?topic=18443.msg197977#msg197977) is where I start Lossless for 5d2 but it should apply to 550D .

Edit: a1ex , the 50D logs don't have the same level of Resources details as 5d2 , 550d is that the way it is on 50D or did I miss something when I compiled "io_trace_full" for 50D ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: scotophorus on August 19, 2018, 06:41:48 AM
Debug io-trace messages from the 7D 2.0.3, i loaded edmac.mo from nightly build and followed your instructions, i hope they are valid:

https://drive.google.com/open?id=1ZrSDF9c8gENH78rvIudZ6fjElD3nBuid
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 19, 2018, 08:02:08 AM
Thanks , looks good  ;D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on August 19, 2018, 04:46:28 PM
Quote from: reddeercity on August 19, 2018, 02:22:51 AM
Edit: a1ex , the 50D logs don't have the same level of Resources details as 5d2 , 550d is that the way it is on 50D or did I miss something when I compiled "io_trace_full" for 50D ?

Something like this should work for all cameras (in dm-spy-extra.c):


    STUB_ENTRY(CreateResLockEntry, 2 | RET, CreateResLockEntry_log),
    STUB_ENTRY(LockEngineResources, 1, LockEngineResources_log),
    STUB_ENTRY(UnLockEngineResources, 1, UnLockEngineResources_log),


Will test some more and enable it by default.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 19, 2018, 10:15:47 PM
Quote from: reddeercity on August 19, 2018, 02:22:51 AM
Edit: a1ex , the 50D logs don't have the same level of Resources details as 5d2 , 550d is that the way it is on 50D or did I miss something when I compiled "io_trace_full" for 50D ?
Quote from: a1ex on August 19, 2018, 04:46:28 PM
Something like this should work for all cameras (in dm-spy-extra.c):


    STUB_ENTRY(CreateResLockEntry, 2 | RET, CreateResLockEntry_log),
    STUB_ENTRY(LockEngineResources, 1, LockEngineResources_log),
    STUB_ENTRY(UnLockEngineResources, 1, UnLockEngineResources_log),


Will test some more and enable it by default.

9.633.183  FrontShtDe:ff882068:96:05: ->sdsPostJob InsertStageJob(32186)
9.633.233  FrontShtDe:ff8a2830:00:03: [JOB] PostNextStage (ID = 32186, Class = 150, Pos = 4)
9.633.324  FrontShtDe:ff8820dc:96:05: sdsPostJob Start(32186)
9.633.360  FrontShtDe:ff882108:96:05: sdfGetDevelopDevType(32186)(0x7000)
9.633.406  FrontShtDe:ff88217c:96:05: ENABLE(Poster:0)(Ohyear:0)
9.633.452  FrontShtDe:ff88148c:96:05: sdfAllocateMemoryDevelopSuite(32186)
9.636.780  FrontShtDe:ff882c90:96:05: sdsMem1ToRawCompression
9.638.127  FrontShtDe:ff880e4c:96:05: sdfExecuteMem1ToRawPath(32186)
9.638.227  FrontShtDe:ff881060:96:05: sdfExecuteMem1ToRawPath(32186)Å©(SemOK)
9.638.261  FrontShtDe:ff881098:96:05: ProcessTwoInTwoOutJpegPath(32186)
9.638.326  FrontShtDe:ffa3620c:00:00: *** LockEngineResources(72089c) x31:
9.638.371  FrontShtDe:00090c34:00:00:      1)    10002 (read channel 0xa)
9.638.399  FrontShtDe:00090c34:00:00:      2)        3 (write channel 0x3)
9.638.422  FrontShtDe:00090c34:00:00:      3)        4 (write channel 0x4)
9.638.454  FrontShtDe:00090c34:00:00:      4)    30000 (read connection 0x0)
9.638.488  FrontShtDe:00090c34:00:00:      5)    20005 (write connection 0x5)
9.638.519  FrontShtDe:00090c34:00:00:      6)    20016 (write connection 0x16)
9.638.543  FrontShtDe:00090c34:00:00:      7)    50003 (?)
9.638.566  FrontShtDe:00090c34:00:00:      8)    5000d (?)
9.638.588  FrontShtDe:00090c34:00:00:      9)    5000f (?)
9.638.612  FrontShtDe:00090c34:00:00:     10)    5001a (?)
9.638.639  FrontShtDe:00090c34:00:00:     11)    80000 (?)
9.638.663  FrontShtDe:00090c34:00:00:     12)    90000 (?)
9.638.685  FrontShtDe:00090c34:00:00:     13)    a0000 (?)
9.638.710  FrontShtDe:00090c34:00:00:     14)   160000 (?)
9.638.736  FrontShtDe:00090c34:00:00:     15)   130003 (?)
9.638.760  FrontShtDe:00090c34:00:00:     16)   130004 (?)
9.638.785  FrontShtDe:00090c34:00:00:     17)   130005 (?)
9.638.810  FrontShtDe:00090c34:00:00:     18)   130006 (?)
9.638.836  FrontShtDe:00090c34:00:00:     19)   130007 (?)
9.638.861  FrontShtDe:00090c34:00:00:     20)   130008 (?)
9.638.888  FrontShtDe:00090c34:00:00:     21)   13000c (?)
9.638.912  FrontShtDe:00090c34:00:00:     22)   130010 (?)
9.638.937  FrontShtDe:00090c34:00:00:     23)   130011 (?)
9.638.962  FrontShtDe:00090c34:00:00:     24)   130012 (?)
9.638.987  FrontShtDe:00090c34:00:00:     25)   130013 (?)
9.639.011  FrontShtDe:00090c34:00:00:     26)   130014 (?)
9.639.035  FrontShtDe:00090c34:00:00:     27)   130015 (?)
9.639.058  FrontShtDe:00090c34:00:00:     28)   130018 (?)
9.639.085  FrontShtDe:00090c34:00:00:     29)   13001a (?)
9.639.109  FrontShtDe:00090c34:00:00:     30)   13001b (?)
9.639.137  FrontShtDe:00090c34:00:00:     31)   13001c (?)


This looks much better for our needs :)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 20, 2018, 06:03:25 AM
Great ! can you post the full Logs some where ?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 20, 2018, 12:27:59 PM
Quote from: reddeercity on August 20, 2018, 06:03:25 AM
Great ! can you post the full Logs some where ?
Cr2TakeDetailed (https://drive.google.com/file/d/1wHKUYTst3SVSxlWWlTz50JJw8uJlvT1x/view?usp=sharing) Cr2ReviewDetailed (https://drive.google.com/file/d/1AkRh1FhCOjmhKiI2F8E-PPEAVyhh687S/view?usp=sharing)

Is this code (https://bitbucket.org/daniel_fort/magic-lantern/src/4286b6c7b7554535273df9ab5d6ac45a764d3617/modules/silent/lossless.c#lines-83) working for 5d2/7d? I got this when I tried to compile dfort's branch for 50D:

lv-img-engio.o: In function `digic_iso_step':
/Users/alex/Desktop/pullML/dfort/platform/50D.109/../../src/lv-img-engio.c:847: undefined reference to `_raw_lv_get_iso_post_gain'
make: *** [magiclantern] Error 1


Can I have rom dumps of both camera? I would like to patter match stubs for 50D
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 20, 2018, 03:24:51 PM
Quote from: aprofiti on August 20, 2018, 12:27:59 PM
Is this code (https://bitbucket.org/daniel_fort/magic-lantern/src/4286b6c7b7554535273df9ab5d6ac45a764d3617/modules/silent/lossless.c#lines-83) working for 5d2/7d?

This was from a series of failed experiments where I was trying to get lossless compression working with the silent module on the 7D. Didn't get very far with it though it does compile on the 7D. If you try merging the latest crop_rec_4k into that branch you'll notice a suggestion to add FRAME_SHUTTER_BLANKING_READ. It is defined on the 50D but not the other Digic 4 cameras. Maybe this is the key to fixing the "earthquake video" sync issues on the 10bit/12bit Digic 4 experiments? I don't know, it has been a while since playing around with this.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 20, 2018, 05:13:55 PM
Used 7D as a reference for pattern matching and come to this:

    if (is_camera("50D", "*"))
    {
        TTL_SetArgs     = (void *) 0xFF195924;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF236248;      /* called right after ProcessTwoInTwoOutJpegPath */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFFA3555C;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF0A2534;      /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFFA3652C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFFA36564;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFFA3659C;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

@dfort can you double check please? Mostly used strings to find adjacent code and then checked argument number/function code.
First one is a bit different as argument numbers passed to function, but look like it is the right one.

Code related to resources:

    else if (is_camera("50D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10002 | edmac_channel_to_index(edmac_read_chan),
            0x30000,    /* read  connection  0 */
            0x20005,    /* write connection  5 */
            0x20016,    /* write connection 16 */
            0x50003,
            0x5000d,
            0x5000f,
            0x5001a,
            0x80000,
            0x90000,
            0xa0000,
            0x160000,
            0x130003,
            0x130004,
            0x130005,
            0x130006,
            0x130007,
            0x130008,
            0x13000c,
            0x130010,
            0x130011,
            0x130012,
            0x130013,
            0x130014,
            0x130015,
            0x130018,
            0x13001a,
            0x13001b,
            0x13001c,
        };

Is the write_channel set to 0 good instead 0x10003/0x10004 from logs?

Can't find stubs for decompress_init using 7D... Im'a bit stuck... Also what are all the "EngDrvOut(...)" stuff?

@reddeercity Can you update you repository please? It should be very useful to understand what to change from the dfort's repo (won't compile also for 5d2)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 20, 2018, 07:06:46 PM
Commented "total_movie_gain *= _raw_lv_get_iso_post_gain();" in a similar way of an old reddeercity's (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198145#msg198145) post.

It show option for lossless compressed DNG, but camera freezes right after taking a lossless picture (full-res or simple).
It print overexposed areas but does't print info about compression (max_compressed_size infobox)

(https://thumb.ibb.co/k84rOK/IMAGE_2018_08_20_19_06_09.jpg) (https://ibb.co/k84rOK)(https://thumb.ibb.co/dQH8Lz/IMAGE_2018_08_20_19_06_15.jpg) (https://ibb.co/dQH8Lz)


Maybe is because it miss EngDrvOut() code for 50D? or it is not necessary?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: dfort on August 21, 2018, 12:39:17 AM
Quote from: aprofiti on August 20, 2018, 05:13:55 PM
@dfort can you double check please?

Looks good to me. Then again I don't have lossless compression working on the 7D yet so it might be that my addresses might be off.

Then there's this:

modules/silent/lossless.c
    if (is_camera("7D", "2.0.3"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xFF2E8EAC;
        Start_DecodeLosslessPath    = (void *) 0xFF2E8F5C;
        Cleanup_DecodeLosslessPath  = (void *) 0xFF2E906C;
    }


I tried finding it in the 50D using pattern matching but couldn't do it. It might be buried somewhere in the log files when shooting CR2 files.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 21, 2018, 02:54:01 AM
Good to see some actions , my plan worked  :))
@aprofiti I see you figured it out , yea that looks right freeze after a lossless , a1ex tried in qemu  I think or he still has a 5d2 and he says after the taking photo it needs to go somewhere
to be save , but it not there , maybe a wrong channel ? not sure .
Quote from: a1ex on August 07, 2018, 08:34:54 AM
Also getting closer to emulating a CR2 image capture. Emulation reaches sdfExecuteMem1ToRawPath (FrontShtDevelop),
attempts to configure JPCORE and gets stuck requesting image data from EDMAC channel #3, connection 5.
I believe that's where the firmware expects LJ92 data from the lossless encoder.

One thing is for sure it does capture the lossless , just need to understand why it not saving  , the jpeg raw compression slices could be wrong etc. ....
I think I did some mods there I'll look at my source . Look here (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198487#msg198487) I play around with slices and I got the compressed sized as seen in that photo but did not save ,
so yea I think "EngDrvOut" is needed with out I get no compressed size on screen just lockup/freezes after pressing the shutter for photo .


if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1)); 
        EngDrvOut(0xC0F12020, PACK32(width    - 1,  height/10 - 1));  /* 0x18A0127 */
    }



Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 21, 2018, 08:00:16 AM
Here my sources code for Lossless on digic 4/5d2 I'm working from eXperimental4kLossless_magic-lantern.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/eXperimental4kLossless_magic-lantern.zip)
also here is the build for Lossless on 5d2 , doesn't work but if someone what to try and see what happen ......
magiclantern-eXperimental4kLossless.2018Aug20.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-eXperimental4kLossless.2018Aug20.5D2212.zip)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 21, 2018, 01:27:36 PM
Quote from: reddeercity on August 21, 2018, 08:00:16 AM
Here my sources code for Lossless on digic 4/5d2 I'm working from eXperimental4kLossless_magic-lantern.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/eXperimental4kLossless_magic-lantern.zip)
also here is the build for Lossless on 5d2 , doesn't work but if someone what to try and see what happen ......
magiclantern-eXperimental4kLossless.2018Aug20.5D2212.zip (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/magiclantern-eXperimental4kLossless.2018Aug20.5D2212.zip)
Thank you reddeercity, this is what i noticed:

    if (is_camera("5D2", "2.1.2"))
    {
        /* ProcessTwoInTwoOutJpegPath, 5D2 2.1.2 */
        TTL_SetArgs     = (void *) 0xFF1BEB18;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF259B58;      /* called right after ProcessTwoInTwoOutJpegPath */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW/SRAW/MRAW */
        TTL_RegisterCBR = (void *) 0xFF258E70;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) 0xFF0AA9D4;      /* called next, with PictureType as arguments */
        TTL_Start       = (void *) 0xFF259E3C;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF259E74;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF259EAC;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

Cross-checked with 50D and 7D from me and dfort and they appears to be good, they appears point out at the same functions.
This mean that is less likely to be bad but can't be sure because we don't have lossless working on these camera...

Quote from: dfort on August 21, 2018, 12:39:17 AM
I tried finding it in the 50D using pattern matching but couldn't do it. It might be buried somewhere in the log files when shooting CR2 files.
Same here, as it's a "ProcessTwoInTwoOutJpegPath" camera instead of "ProcessTwoInTwoOutLosslessPath", it I tried looking for string regarding Jpeg printed by "ImgPlayDrv" from logs:

    if (is_camera("50D", "*"))
    {
        Setup_DecodeLosslessRawPath = (void *) 0xff962c60; //<<<<<< ImagePlayDriverTask ProcessID = 6 - not sure if need to start from here or if the bottom one are better, but this is in the same address range like the one from D5 cameras
    //    Setup_DecodeLosslessRawPath = (void *) 0xffa17290; //DEC RequestDecodeJpeg(mode:1) 193
    //    Setup_DecodeLosslessRawPath = (void *) 0xffa16e68; //DEC SetParameterDecodeJpeg 148
        Start_DecodeLosslessPath    = (void *) 0xFF960AE8; //StartJpegDecoding 614
        Cleanup_DecodeLosslessPath  = (void *) 0xff963e2c; //ClearDecodeJpegPath 414
    }

Maybe a better idea is to sneak peek address from a working d5 camera using ProcessTwoInTwoOutJpegPath (700D, 650D, M, 100D).
What do you think about @dfort?

Tried this with 50D with no results, but I wasn't expecting to works:

    if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1));/* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }

@reddeercity can you explain how you got these address and values? Is due to this procedure (https://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987)?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on August 21, 2018, 01:46:22 PM
Yes. Or logs done by running io-trace-full build on a real camera.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 22, 2018, 04:41:22 AM
Quote from: ArcziPL on August 21, 2018, 01:46:22 PM
Yes. Or logs done by running io-trace-full build on a real camera.
Yes , real hardware/Camera
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 22, 2018, 06:20:59 AM
Quote from: aprofiti on August 21, 2018, 01:27:36 PM
Thank you reddeercity, ..........
Tried this with 50D with no results, but I wasn't expecting to works:

    if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F12010,        width    - 1                 );  /* 0xB8F     */
        EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1));/* 0xF6D0B8F */
        EngDrvOut(0xC0F1201C,        width/10 - 1                 );  /* 0x127     */
        EngDrvOut(0xC0F12020, PACK32(width/10 - 1,  height/20 - 1));  /* 0x18A0127 */
    }

@reddeercity can you explain how you got these address and values? Is due to this procedure (https://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987)?
Mainly a guess , but being the 5d2 & the 5D3 are very similar , (e.g. c0f06084 D4 , c0f06804 D5 they do the same thing on both camera) it's usually 1 or 2 reg numbers different .
Case in point -- all my 4k/UHD dev. is base of the first 3k 5d3 preset and work backwards from there to where I nearly have it working (Oh so close now  :D , it shouldn't be too long from now )

But those  "EngDrvOut" address are in my decompiled 5d2 rom and also in my dm-0001_io-trace-full_reveiwing_cr2_outside_liveview Log so I figured it couldn't hurt .

**dm-0001_io-trace-full_reveiwing_cr2_outside_liveview**
4.098.439  FrontShtDe:ff9a5630:MMIO : [0xC0F12010] <- 0x0E9F078F
4.098.441  FrontShtDe:ff9a5630:MMIO : [0xC0F12014] <- 0x0E9F077F
4.098.445  FrontShtDe:ff9a5630:MMIO : [0xC0F1201C] <- 0x00E90078
4.098.446  FrontShtDe:ff9a5630:MMIO : [0xC0F12020] <- 0x00E90077


and from 5d2.212.dis -- there 3 address for c0f12010
ff4dfd6c: c0f12010 rscsgt r2, r1, r0, lsl r0
ff4e0298: c0f12010 rscsgt r2, r1, r0, lsl r0
ff4e0838: c0f12010 rscsgt r2, r1, r0, lsl r0


from 5d2.212.dis -- there 3 address for c0f12014
ff4dfd74: c0f12014 rscsgt r2, r1, r4, lsl r0
ff4e02a0: c0f12014 rscsgt r2, r1, r4, lsl r0
ff4e0840: c0f12014 rscsgt r2, r1, r4, lsl r0


from 5d2.212.dis -- there 3 address for c0f1201C
ff4dfd84: c0f1201c rscsgt r2, r1, r12, lsl r0
ff4e02b0: c0f1201c rscsgt r2, r1, r12, lsl r0
ff4e0850: c0f1201c rscsgt r2, r1, r12, lsl r0


from 5d2.212.dis -- there 3 address for c0f12020
ff4dfd8c: c0f12020 rscsgt r2, r1, r0, lsr #32
ff4e02b8: c0f12020 rscsgt r2, r1, r0, lsr #32
ff4e0858: c0f12020 rscsgt r2, r1, r0, lsr #32


Not sure what the strings are doing , I still have a very had time understand strings fully .
After the "Pack32" stuff I have no idea if that code is right for d4 cam's ,
I think it has to do with the slices (d4/5d2 are 1936 & 1920) Understanding What is stored in a Canon RAW .CR2 file (http://lclevy.free.fr/cr2/)

Found d4/5d2 old compressed raw test from back in march of this year
had the "EngDrvOut" stuff , I did something else too to the code can't remember right now , I'll have to look in one of my old compressed raw source code experiments .
(https://image.ibb.co/jei6Az/IMG_2364_small.png) (https://imgbb.com/)


@a1ex , I notice something in the cr2 compressed between the 5d2 & the 5d3 in Understanding What is stored in a Canon RAW .CR2 file (http://lclevy.free.fr/cr2/) the 5D3 uses  "jpeg comp" "2" where the 5d2 uses "4
Quotethe Full Raw Jpeg is encoded with 14 bits, in 2 colors (prior to the 50D and the Digic IV), then using 4 colors with the 50D up to 1100D. Since 1D X and up to 6D, Canon is back to 2 components
(https://preview.ibb.co/niAs0z/Raw_Data.png) (https://ibb.co/kAZKfz)

Could this be a problem ? or does this even apply ? mind you I don't total understand this .
Does this have something to do with slices ? 
width/10 - 1,  height/20 - 1
etc. ....
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 22, 2018, 06:51:54 AM
Quick check of the dm-50D-cr2Take.Log for "C0f12010 , C0f12014 , C0f1201C & C0f12020"
there are present

3.418.899  FrontShtDe:ff97d91c:MMIO : [0xC0F12010] <- 0x0C5F064F
3.418.900  FrontShtDe:ff97d91c:MMIO : [0xC0F12014] <- 0x0C5F063F
3.418.904  FrontShtDe:ff97d91c:MMIO : [0xC0F1201C] <- 0x018B00C9
3.418.906  FrontShtDe:ff97d91c:MMIO : [0xC0F12020] <- 0x018B00C7

if they do anything is another matter
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 23, 2018, 02:47:41 AM
To my understanding, to make lossless works we need to "tell" to camera via registry values what is the size of the image to process and others possible values to fix image;
Also image is processed into two "slices" from LossLess routine. Are they calculated from sensor size, image size or jpeg size?

Using dfort's spreadsheet (https://docs.google.com/spreadsheets/d/1vMcOUh13TVOV3GY2576ke8rxiLn8qiMN9q57BEAXk-E/edit#gid=487482970) as a reference for sensor size, some possible values found in logs and used to make it works could be:

Camera 50D:
Sensor Width:  4832  -> 0x12E0
Sensor Height: 3228  -> 0xC9C
Image Width:   4752  -> 0x1290
Image Height:  3168  -> 0xC60

PACK32(Sensor Width - 1, Sensor Height -1)      -> 0000 1100 1001 1011 (0xC9B) 0001 0010 1101 1111 (0x12DF)   -> 0xC9B12DF
PACK32(Sensor Width - 1, Sensor Height/2 -1)    -> 0000 0110 0100 1101 (0x64D) 0001 0010 1101 1111 (0x12DF)   -> 0x64D12DF


Quote from: ArcziPL on August 21, 2018, 01:46:22 PM
Yes. Or logs done by running io-trace-full build on a real camera.
Unfortunately looking into logs for RABBIT's address range 0xc0f1*, I can find only those lines:

15.702.354     CtrlSrv:ff85de78:82:01: ImgDDev Reg c0f140e0 01b00000
15.702.389     CtrlSrv:ff85de78:82:01: ImgDDev Reg c0f140e4 01b005a0


@a1ex Is again logging 50D a bit different from others camera and need to be taken care of it? (500D logs appears to have it)
How does full-io-trace's logs looks like from D5 ProcessTwoInTwoOutJpegPath cameras (ex. 700D)?

Edit:
Found pack32 sensor size at a different address than expected:

2.505.534  ShootCaptu:ff97d91c:MMIO : [0xC0F080B0] <- 0x0C9B12DF
2.743.185  FrontShtDe:ff97d91c:MMIO : [0xC0F13068] <- 0x0C9B12DF
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 23, 2018, 06:26:13 AM
Slices are as far as I know are a set size in the encoder for each camera .
Looking in to those reg's again

  [0xC0F12010] <- 0x0E9F078F
  [0xC0F12014] <- 0x0E9F077F
  [0xC0F1201C] <- 0x00E90078
  [0xC0F12020] <- 0x00E90077


If you do the math on the last reg's  0x00E90078 = 15270008 , 0x00E90077 = 15270007 
then add the Decimal number together you get 30540015 very close to the (see post #417) compressed size for 5d2 5632x3752 = 30842880
if I did my math right . Maybe I'll try "c0f1201c & c0f12020 and see if it works .
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: aprofiti on August 23, 2018, 09:36:37 PM
I was using wrong log to look for 0xC0F13*** and found the correct one, here are MMIO traces regarding RABBIT:

2.743.052  FrontShtDe:ff97d91c:MMIO : [0xC0F13060] <- 0x80000000
2.743.183  FrontShtDe:ff97d91c:MMIO : [0xC0F13064] <- 0x00000E08
2.743.185  FrontShtDe:ff97d91c:MMIO : [0xC0F13068] <- 0x0C9B12DF
2.743.187  FrontShtDe:ff97d91c:MMIO : [0xC0F13050] <- 0x00000014
2.743.189  FrontShtDe:ff97d91c:MMIO : [0xC0F13054] <- 0x00000000
2.743.190  FrontShtDe:ff97d91c:MMIO : [0xC0F13058] <- 0x00000000
2.743.922  FrontShtDe:ff97d91c:MMIO : [0xC0F13048] <- 0x00000003
2.743.924  FrontShtDe:ff97d91c:MMIO : [0xC0F13060] <- 0x00000000
2.743.929  FrontShtDe:ff97d7d0:MMIO : [0xC0F13000] <- 0x00000000
2.743.931  FrontShtDe:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000003
2.743.934  FrontShtDe:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
2.743.936  FrontShtDe:ff97d7d0:MMIO : [0xC0F13004] <- 0x00000001
2.866.854  FrontShtDe:ff97d91c:MMIO : [0xC0F13060] <- 0x80000000
2.872.860  FrontShtDe:ff97d91c:MMIO : [0xC0F13000] <- 0x80000000
2.873.155  FrontShtDe:ff97d91c:MMIO : [0xC0F13008] <- 0x00000101
2.873.156  FrontShtDe:ff97d91c:MMIO : [0xC0F1300C] <- 0x00000880
2.873.158  FrontShtDe:ff97d91c:MMIO : [0xC0F13010] <- 0x00000CC0
2.873.160  FrontShtDe:ff97d91c:MMIO : [0xC0F13014] <- 0x00000008
2.873.162  FrontShtDe:ff97d91c:MMIO : [0xC0F13018] <- 0x0C5F020F
2.873.163  FrontShtDe:ff97d91c:MMIO : [0xC0F1301C] <- 0x0000020F
2.873.165  FrontShtDe:ff97d91c:MMIO : [0xC0F13020] <- 0x00000001
2.873.167  FrontShtDe:ff97d91c:MMIO : [0xC0F13050] <- 0x00000000
2.873.169  FrontShtDe:ff97d91c:MMIO : [0xC0F13054] <- 0x00000000
2.873.170  FrontShtDe:ff97d91c:MMIO : [0xC0F13058] <- 0x00000000
2.876.725  FrontShtDe:ff97d91c:MMIO : [0xC0F13000] <- 0x00000000
2.876.729  FrontShtDe:ff97d7d0:MMIO : [0xC0F13000] <- 0x00000000
2.876.732  FrontShtDe:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000003
2.876.735  FrontShtDe:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
2.876.737  FrontShtDe:ff97d7d0:MMIO : [0xC0F13004] <- 0x00000001
2.991.160  FrontShtDe:ff97d91c:MMIO : [0xC0F13000] <- 0x80000000
2.995.403  RearShtDev:ff97d91c:MMIO : [0xC0F13000] <- 0x80000000
2.995.483  RearShtDev:ff97d91c:MMIO : [0xC0F13008] <- 0x00000101
2.995.484  RearShtDev:ff97d91c:MMIO : [0xC0F1300C] <- 0x00000280
2.995.486  RearShtDev:ff97d91c:MMIO : [0xC0F13010] <- 0x000003C0
2.995.488  RearShtDev:ff97d91c:MMIO : [0xC0F13014] <- 0x00000000
2.995.490  RearShtDev:ff97d91c:MMIO : [0xC0F13018] <- 0x0077009F
2.995.491  RearShtDev:ff97d91c:MMIO : [0xC0F1301C] <- 0x00000000
2.995.493  RearShtDev:ff97d91c:MMIO : [0xC0F13020] <- 0x00000001
2.995.495  RearShtDev:ff97d91c:MMIO : [0xC0F13050] <- 0x00000000
2.995.497  RearShtDev:ff97d91c:MMIO : [0xC0F13054] <- 0x00000000
2.995.499  RearShtDev:ff97d91c:MMIO : [0xC0F13058] <- 0x00000000
2.995.547  RearShtDev:ff97d91c:MMIO : [0xC0F13000] <- 0x00000000
2.995.551  RearShtDev:ff97d7d0:MMIO : [0xC0F13000] <- 0x00000000
2.995.554  RearShtDev:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000003
2.995.556  RearShtDev:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
2.995.559  RearShtDev:ff97d7d0:MMIO : [0xC0F13004] <- 0x00000001
2.997.414  RearShtDev:ff97d91c:MMIO : [0xC0F13000] <- 0x80000000
3.261.519  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13000] <- 0x00000000
3.261.522  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000030
3.261.524  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000001
3.261.527  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13004] <- 0x00000001
3.262.055  **INT-64h*:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000000
3.262.057  **INT-64h*:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
3.262.060  **INT-64h*:ff97d7d0:MMIO : [0xC0F13000] <- 0x80000000
3.265.204  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000000
3.265.205  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
3.265.209  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13000] <- 0x80000000
3.268.690  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13000] <- 0x00000000
3.268.693  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000030
3.268.694  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000001
3.268.698  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13004] <- 0x00000001
3.269.408  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13000] <- 0x80000000
3.269.411  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13008] <- 0x00000111
3.269.415  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1300C] <- 0x00000E00
3.269.417  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13010] <- 0x00001500
3.269.420  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13014] <- 0x0000001D
3.269.422  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13018] <- 0x0C5F009F
3.269.424  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1301C] <- 0x0000006F
3.269.426  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13050] <- 0x00000000
3.269.861  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13000] <- 0x00000000
3.269.863  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000030
3.269.865  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000001
3.269.868  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13004] <- 0x00000001
3.373.401  **INT-5Bh*:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000000
3.373.403  **INT-5Bh*:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
3.373.406  **INT-5Bh*:ff97d7d0:MMIO : [0xC0F13000] <- 0x80000000
3.373.616  **INT-6Eh*:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
3.373.864  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13048] <- 0x00000000
3.373.866  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F1304C] <- 0x00000000
3.373.869  ImgPlayDrv:ff97d7d0:MMIO : [0xC0F13000] <- 0x80000000

Something to notice about apart the PACK32(Sensor Width - 1, Sensor Height -1)?

I tried EngDrvOut with these values:

      EngDrvOut(0xC0F2E02C, 0x064D0C9B); // here is PACK32(Sensor Width - 1, Sensor Height/2 -1) in canon configuration. Replaced with ML configuration for "slices"
      EngDrvOut(0xC0F13068, 0x0C9B12DF); // FrontShutter PACK32(Sensor Width - 1, Sensor Height -1)
      EngDrvOut(0xC0F13064, 0x00000E08);
      EngDrvOut(0xC0F13050, 0x00000014);
      EngDrvOut(0xC0F13054, 0x00000000);
      EngDrvOut(0xC0F13058, 0x00000000);
      EngDrvOut(0xC0F13060, 0x80000000);
      EngDrvOut(0xC0F13000, 0x80000000);


Unfortunately camera still freezes after taking a silent picture and I'm still trying to figure out what informations to extract from MMIO traces

Quote from: reddeercity on August 23, 2018, 06:26:13 AM
Slices are as far as I know are a set size in the encoder for each camera .
Looking in to those reg's again

  [0xC0F12010] <- 0x0E9F078F
  [0xC0F12014] <- 0x0E9F077F
  [0xC0F1201C] <- 0x00E90078
  [0xC0F12020] <- 0x00E90077


If you do the math on the last reg's  0x00E90078 = 15270008 , 0x00E90077 = 15270007 
then add the Decimal number together you get 30540015 very close to the (see post #417) compressed size for 5d2 5632x3752 = 30842880
if I did my math right . Maybe I'll try "c0f1201c & c0f12020 and see if it works .


2.743.197  FrontShtDe:ff97d91c:MMIO : [0xC0F12010] <- 0x0C5F064F  // PACK32(? , Image Height - 1)
2.743.199  FrontShtDe:ff97d91c:MMIO : [0xC0F12014] <- 0x0C5F063F  // PACK32(? , Image Height - 1)
2.743.203  FrontShtDe:ff97d91c:MMIO : [0xC0F1201C] <- 0x018B00C9 // PACK32(close to image width * 12 , ?)
2.743.204  FrontShtDe:ff97d91c:MMIO : [0xC0F12020] <- 0x018B00C7 // PACK32(close to image width * 12 , ?)

They appears to be used also on 50D. Their values are close to sensor/image size
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on August 27, 2018, 02:10:21 AM
Playing around with different " EngDrvOut" code .
Tried if (is_camera("5D2", "*"))
    {
        EngDrvOut(0xC0F13068, PACK32(width    - 1,  height    - 1));  /* 0x0EDB169F on 5D2 */ 
        //EngDrvOut(0xC0F12010,        width    - 1                 );   /* 0x0E9F078F *//
        //EngDrvOut(0xC0F12014, PACK32(width    - 1,  height/2    - 1)); /* 0x0E9F077F *//
        //EngDrvOut(0xC0F1201C,        width - 1                 );  /* 0x00E90078 *//
        //EngDrvOut(0xC0F12020, PACK32(width - 1,  height - 1));  /* 0x00E90077 *//
    }

Those values came from my cr2 review log outside liveview io-trace-full build , didn't work but something interesting
happen when I tried burst mode in lossless , started to save frames around 5-7 then said out of memory this printed out a error on screen
and made a "assert log"
ML ASSERT:
tmp_suite
at silent.c:1042 (silent_pic_take_lv), task shoot_task
lv:1 mode:3

shoot_task stack: 18ffa8 [190168-18e168]
0xUNKNOWN  @ ff8773e0:190160
0x0009D668 @ 7c7b4:190100
0xUNKNOWN  @ 9d6bc:1900e8
0x009A5444 @ 9a655c:1900d0
0x0004EADC @ 9a5cd4:18ffd8
0x0004E378 @ 4eb38:18ffa8

Magic Lantern version : eXperimental4kLossless.2018Aug26.5D2212
Mercurial changeset   : 4286b6c7b755+ (crop_rec_4k_Digic4)
Built on 2018-08-26 22:28:53 UTC by david@reddeercity.
Free Memory  : 166K + 3698K

not sure if it helpful , there is some "unknow" there should there be a variable there ?
on to the next change and see would happen.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on October 12, 2018, 08:21:44 AM
have another go at lossless on 5D2/D4
Did a dm-spy log in sRaw2 (small raw2 - 2800x1872)
Hoping to find something how is encodes for full frame 5768 to 2808
[TTJ][150,5768,0] SRAW(2808,1872,0,15)
      ..........
DA738> FrontShtDe:ffa59888:16:03: [TTJ] START WR1:0x31d00b8 WR2:0x2e040c4
DA76C> FrontShtDe:ffa598b4:16:03: [TTJ] START RD1:0x100882fa RD2:0x124d1864

I'm see some different address in the recourse's then full frame , going thought the log now .
Also did a dm-spy with sRaw (3820x2560) I think , haven't looked at yet .
I'll update if I find anything useful .
   
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on November 04, 2018, 05:55:38 AM
I think we may have a encoder size/resolution  miss match on the 5d2/d4 ,
The size we capture is 5632x3752 see image in post (https://www.magiclantern.fm/forum/index.php?topic=18443.msg205333#msg205333)

Looking at 5d2 cr2 with exiftool
ensor Width                    : 5792
Sensor Height                   : 3804
Sensor Left Border              : 168
Sensor Top Border               : 56
Sensor Right Border             : 5783
Sensor Bottom Border            : 3799

So the Canon active area is
5624x3748
and magic lantern active area is
5632x3752
I would think the encoder has stick rules for input & export and by changing input size It could cause a locking up in the encoder .
I wonder if we change the active area to what canon is for the encoder .

From what I tell all problem start with the encoder on export , that when it lockup the cam & need a battery pull
I need to look at the silent picture module code and see if I can change the capture area to the same as Canon
and see if there still a issue.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on November 04, 2018, 08:59:57 AM
Quote from: reddeercity on November 04, 2018, 05:55:38 AM
I think we may have a encoder size/resolution  miss match on the 5d2/d4 ,

Encoder is always configured from full raw buffer size. Active area is just how these pixels are interpreted by ML or post-processing software; the lossless encoder does not know which pixels are active and which ones are not; it simply doesn't care about that. The lossless routine does not use the active area parameters in any way.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on November 05, 2018, 05:13:53 AM
Ok , understood -- Is there any way to see or log what the encoder is doing ?
Is it that the expected data is not where the encoder needs it to be ?
other words the address mem1 is not right ? 
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 05, 2019, 11:47:20 PM
Quote from: a1ex on March 02, 2018, 10:09:48 PM
lossless-1080-qemu-5D3-vs-6D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-5D3-vs-6D.html)
lossless-1080-qemu-5D3-vs-70D.html (http://a1ex.magiclantern.fm/bleeding-edge/raw/lossless/lossless-1080-qemu-5D3-vs-70D.html)

Hi there. There's some progress on a lossless RAW compression on 70D. Not as far, as I'd wish but still.

The starting point was a recent revision of crop_rec_4k_mlv_snd, where the stubs and two image size registers were already set. However, the compressed image shown heavy deviations from a reference non-compressed one. It was similar to the initial distortions seen on 6D and 650D.

Uncompressed FRSP vs. compressed FRSP:
(https://i.postimg.cc/7b4VkrzX/00-starting-point-uncompressed-fullres.png) (https://postimg.cc/7b4VkrzX)    (https://i.postimg.cc/hQKbbVnN/01-starting-point-lossless-fullres.png) (https://postimg.cc/hQKbbVnN)


I've found two solutions leading to a working losslessly compressed FRSP ("Full-res" option in Silent Picture sub-menu) and losslessly compressed LV buffer ("Simple" option).


1. Overriding register 0xC0F373B4

Whetever it really does, a value of 0 in register 0xC0F373B4 results in a valid compressed DNG.
EngDrvOut(0xC0F373B4, 0);

Interesting fact: originally the camera sets this register to 0x11. A similar change of another register, also from a value 0x11 to 0 is needed for 6D and 650D.
    if (is_camera("6D", "*") ||
        is_camera("650D", "*"))
    {
        /* not sure what exactly these do, but they seem to be required to get correct image
         * taken from register log diffs: http://www.magiclantern.fm/forum/index.php?topic=18443.msg197987#msg197987 */
        EngDrvOut(0xC0F37610, 0);       /* 0x11 on 650D, 0 on all other D5 (not sure if needed) */



2. Second, independent solution is an overriding of register 0xC0F373F4

This one is really interesting.

Original log from the 70D:
[0xC0F373F4] <- 0x7FFF7FFF: ???
[0xC0F373F8] <- 0x1111BBBB: ???


The first register contains two 16-bit words, which seem to define horizontal and vertical boundaries of a slice, which should get somehow processed.

By lowering the least significant word ([0xC0F373F4] <- 0x7FFF1111) we get such a picture:
(https://i.postimg.cc/9RdY7J6G/03-0x-C0-F373-F4-0x7-FFF1111.png) (https://postimg.cc/9RdY7J6G)

By lowering also the most significant word ([0xC0F373F4] <- 0x01000A00) it looks like this:
(https://i.postimg.cc/V5SYm2cr/05-0x01000a00.png) (https://postimg.cc/V5SYm2cr)

What it really is used for, I've no idea.

Changing any (or both) of the words to 0x0000 will give us a nice, valid, losslessly compressed DNG. E.g:

EngDrvOut(0xC0F373F4, 0x7FFF0000);

I wasn't able to find the meaning of the second mentioned register. There was no visible difference when changing it. Maybe it refers to another processing mode of the compressor module or maybe to a different write channel?

Interesting fact: similar values can be seen other cameras; here 5D3:
177 [0xC0F375B8] <- 0x7FFF7FFF: ???
178 [0xC0F375BC] <- 0x7FFF7FFF: ???
179 [0xC0F375C0] <- 0x7FFF7FFF: ???
180 [0xC0F375C4] <- 0x7FFF7FFF: ???
181 [0xC0F375C8] <- 0x7FFF7FFF: ???
182 [0xC0F375CC] <- 0x7FFF7FFF: ???
183 [0xC0F375D0] <- 0x7FFF0000: ???
184 [0xC0F375D4] <- 0x7FFF0000: ???
185 [0xC0F375D8] <- 0x7FFF7FFF: ???
186 [0xC0F375DC] <- 0x7FFF7FFF: ???
187 [0xC0F375E0] <- 0x7FFF7FFF: ???



The miniature picture embedded in DNG is unfortunately all black in the compressed files (despite being fine on the uncompressed FRSP). But I don't see it as an issue now, as it's same on 700D.


Now the sad part. Despite silent pictures are working very well with these changes, the compressed MLV still doesn't. The symptoms are still like in the past:

Quote from: andy kh on March 05, 2018, 04:55:19 AM
#dfort  tried installing your latest crop rec4k for 70D. after installing  i turn On crop rec and mlv lite. restart my camera. no buttons would work after that. i tried reinstalling your build 2 times but no luck
Quote from: andy kh on March 06, 2018, 08:35:26 PM
with your latest build i get this FRSP https://files.fm/u/3kc5sbt if i activate mlv lite, recording button nad trast button dont response

Describing it a bit more detailed:

If mlv_lite is activated and configured to 14-bit uncompressed, the camera will normally record MLV.

If being already in the movie mode, opening ML menu, changing bit-depth to 14-bit lossless: the ML menu will freeze immediately just after highlighting this option (without clicking SET button). No reaction to menu navigation, ML menu can't be closed. However, the cam still works, it can be switched to photo mode and will work normally showing "Raw detect error" for a while.

When being in photo mode, you can select 14-bit lossless in RAW video menu. Apparently there is no initialization of buffers/compressor executed at that point. When switching the camera to movie mode the buttons themself are working properly. ML console shows even "REC key pressed" (when pressed) and "Starting raw recording..." but the recording doesn't start. You can enter ML menu but it will freeze as soon as you change the tab to Movie. To unfreeze the menu: change mode to photo.


I have a feeling it will unfortunately not be enough to play with some more registers in lossless.c to fix the RAW video. I've "simulated the 650D" behaviour with wrongly configured compressor by applying wrong values on 700D.
(https://i.postimg.cc/zbZkv9cz/emulating-650-D-on-700-D.png) (https://postimg.cc/zbZkv9cz)
Result: silent picuteres had distorted colors but the lossless MLV were still working (with distorted colors as well).

Could it rather be related to some buffer allocation, conflicting resources or waiting for flags/semaphores? How can it be further debugged?

1. What is exactly executed as soon as the 14bit lossless option is only highlighted in the ML menu when being in the video mode?
2. What and how to debug? I have already compiled with #define RAW_DEBUG but I don't know how to store the content of the console to a file O_o
3. How does compression differ in photo mode ("simple" option should be the LV buffer?) from movie mode?
4. Can these be conflicticting/wrong EDMAC resources in video mode? I am reluctant to play around with it on my own, as I have no idea how it works but it's clear for me, that a wrong change might results in "random" writes into any memory area.
5. Any other ideas...?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 06, 2019, 12:10:40 AM
Very nice!

1. ML is attempting to compress the LiveView frame every 2 seconds, to find out the compression ratio. This happens in measure_compression_ratio -> forwarded to compress_task -> it compresses one frame as usual.

2. Hard to answer... I'd probably get some MMIO logs (io_trace_full branch), and... from there I'll probably end up staring at them.

3. In full-res mode, ML is operating outside LiveView (only the mirror is still up). In other words, all the image processing resources are available. In LiveView, some of them may be used. In movie mode, the configuration performed by Canon might be slightly different.

Need to read more carefully, but if the simple silent picture, saved as lossless DNG, works in movie mode, I see no reason why MLV compression wouldn't work. The former would be a lot easier to debug, too.

I'm starting to think we were really lucky on most DIGIC 5 models, where the resources used by lossless compression weren't in conflict with resources used by LiveView (well, there were a few conflicts that we knew how to solve). I had many attempts to get it working on DIGIC 4, but didn't document them, because... it just didn't work in LiveView.

4. EDMAC resources: will check. You could experiment with edmac.mo and see what it prints - screenshots, free channels, logs. These should help refreshing my memory. General idea: you can pick any unused channels (not active in the overview screen, and reported as "free" when you run the "Find free EDMAC channels" test). By default, the ones used for edmac_memcpy (i.e. uncompressed raw video) are reused, so I don't expect any trouble here.

Other ideas... not sure, maybe near the end of the week.

"The miniature picture embedded in DNG is unfortunately all black in the compressed files" -> expected; it's not implemented.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 06, 2019, 10:24:51 AM
Quote from: a1ex on March 06, 2019, 12:10:40 AM
Need to read more carefully, but if the simple silent picture, saved as lossless DNG, works in movie mode, I see no reason why MLV compression wouldn't work

Hi Alex, that's a great point! I tried silent pictures in photo mode only, as the hint in the menu was showing that full-res silent pics are not working in movie mode. But you're fully right, simple silent pictures can be also done in movie mode.

So, here comes more:

1. Only silent.mo loaded: simple compressed DNG in movie mode works
2. Only silent.mo + mlv_rec.mo loaded, RAW video set to off: simple compressed DNG in movie mode works
3. Only silent.mo + mlv_rec.mo loaded, RAW video set to on: simple compressed DNG in movie mode works, raw video recording is also working (uncompressed of course)
4. Only silent.mo + mlv_lite.mo loaded, RAW video set to off: simple compressed DNG in movie mode works
5. Only silent.mo + mlv_lite.mo loaded, RAW video set to on, bit depth to 14bit uncompressed: simple compressed DNG in movie mode doesn't work!

In case 5 it stucks with "Preparing..." on the screen. The camera is still responsive. Just the text stays there until mode is changed to photo.
(https://i.postimg.cc/7Jngv1MS/70-silent-in-video-mode-stuck.png) (https://postimg.cc/7Jngv1MS)

For a start I made some screenshots of EDMAC channels.

Photo mode:
(https://i.postimg.cc/JHrQzdhp/70d-edmac-photo-a.png) (https://postimg.cc/JHrQzdhp) (https://i.postimg.cc/NLBDDLxR/70d-edmac-photo-b.png) (https://postimg.cc/NLBDDLxR)


Video mode:
(https://i.postimg.cc/8jnbT4gb/70d-edmac-video-a.png) (https://postimg.cc/8jnbT4gb) (https://i.postimg.cc/zVfjmM6B/70d-edmac-video-b.png) (https://postimg.cc/zVfjmM6B)

I'm a total noob here but they seem for me nearly the same.

BTW: no in-cam screenshots for the last case, as trying to take them in movie mode results in a memory error. Just checked on 700D and it's same.

(https://i.postimg.cc/RqdRD9Py/70d-mem-err-screenshot-video-mode.png) (https://postimg.cc/RqdRD9Py)
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 06, 2019, 10:48:59 AM
With mlv_lite's RAW Video enabled, the entire memory is preallocated during stanby. I'm seriously thinking to revert this change, just haven't managed to code anything in this direction. Been focusing on new ports + QEMU lately (as my main camera is defective anyway).

I forgot one important detail - in crop_rec_4k, the silent picture module will go out of LiveView to save images, even simple ones. That change was done for being able to show a preview, but... the image processing resources are also free in this "paused LV" mode; no wonder it worked :D

Try undoing out this change (https://bitbucket.org/hudson/magic-lantern/commits/bc85e86ecb); will it still work? The goal is compressing one frame while LiveView is still running.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 06, 2019, 11:30:13 AM
Quote from: a1ex on March 06, 2019, 10:48:59 AM
With mlv_lite's RAW Video enabled, the entire memory is preallocated during stanby.
Sounds not that "lite" at all. ;)

Quote from: a1ex on March 06, 2019, 10:48:59 AMI forgot one important detail - in crop_rec_4k, the silent picture module will go out of LiveView to save images, even simple ones. That change was done for being able to show a preview, but... the image processing resources are also free in this "paused LV" mode; no wonder it worked :D

Try undoing out this change (https://bitbucket.org/hudson/magic-lantern/commits/bc85e86ecb); will it still work? The goal is compressing one frame while LiveView is still running.
Changes in silent.c undone. Result:

silent.mo + mlv_rec loaded, RAW video on: simple silent pictures can be taken (both compressed and uncompressed)

silent.mo + mlv_lite loaded, RAW video on (14-bit selected): simple silent pictures can't be taken (both compressed and uncompressed) -> stuck at "Preparing..."

EDIT:
I've just double checked. Before reverting the change in silent.c, I was also not able to save uncompressed simple silent pictures with mlv_lite loaded and activated.
I also don't see any perceptual difference in behaviour between these two versions. Shall there be any? (to be sure it worked as expected, i.e. simple pictures are taken without pausing LV...)

EDIT #2:
700D behaves the same. The simple silent pictures don't work in movie mode with mlv_lite activated. However, lossless MLV videos do work... Looks like a non-related (?) issue.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 06, 2019, 02:14:04 PM
You don't have to try saving pictures with mlv_lite active. It just won't work, because the silent picture module will not be able to allocate memory.

The goal is compressing one frame while Canon's LiveView is running (in both photo and movie mode).

You can tell whether the picture is saved outside LiveView or not, by looking at the screen. Does the image pause for a moment (with a short flicker), or does it stay real-time while the image is saved?

You could call the following from 'don't click me' (run_test in debug.c):

msleep(2000);
PauseLiveView();
ResumeLiveView();


You will notice a difference. The same "flicker" (or stuttering or whatever) happens on crop_rec_4k with simple silent picture - it pauses LiveView for a moment. On the "unified" codebase, a simple silent picture is saved right away, without pausing LiveView, so there will be no "flicker" or stuttering.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 06, 2019, 06:23:42 PM
Thank you a1ex!

My previous post was garbage. The silent.mo didn't compile after my manually entered changes and I didn't notice immediately... There were also leftovers of the previous build on the card...

silent.c: In function 'silent_pic_take_lv':
silent.c:1250:14: error: too many arguments to function 'silent_pic_save_file'
         ok = silent_pic_save_file(&local_raw_info, 0);
              ^
silent.c:560:12: note: declared here
static int silent_pic_save_file(struct raw_info * raw_info)
            ^
../../Makefile.filerules:31: recipe for target 'silent.o' failed
make[5]: *** [silent.o] Error 1


Now it's fixed, I can confirm there is no visible "flicker" when taking a simple silent photo anymore and no preview is shown afterwards, so we can be sure LV is not paused anymore.

Result with silent.mo + mlv_rec loaded, RAW video on (just in case it matters):

Simple silent pictures do work, both compressed and uncompressed, in photo and in video mode! The compressed ones look identical to uncompressed and are 48% smaller, so all as it should be.  8)

Quote from: a1ex on March 06, 2019, 02:14:04 PMYou could call the following from 'don't click me' (run_test in debug.c):
Thanks for reminding where to find it. I will definitely need it sooner or later.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 06, 2019, 07:40:24 PM
Sounds good!

In this case, debugging should be "just" a matter of adding printf's in mlv_lite, possibly in compress_task, to find out where it gets stuck. Just a random thought - does that task even start? Most of ML code does not check whether a task could be successfully started; need to fix it.

Otherwise, lossless.c has its own share of debug messages - enable them with verbose=1 or verbose=2 (just hardcode the value). These are also useful, to see whether the encoder is called with the right arguments, whether it starts, whether it finishes, how long it takes and so on. These are very useful hints if things don't work.

I was afraid of something worse, i.e. one of these cases where only a MMIO trace could help - assuming it could get saved to card before the camera locked up completely. That's harder to debug, but usually not impossible.

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 06, 2019, 10:45:32 PM
Let me say it again: my previous post was garbage. >:(

Implementing a lot of printf() in
mlv_lite.c -> compress_task()
lossless.c -> lossless_compress_raw_rectangle()

let me find out, that the camera stucks at

lossless_compress_raw_rectangle():
    ...
    /* configure the processing modules */
    TTL_Prepare(TTL_ResLock, &TTL_Args);


I checked it closer to find out, that it stucks also in photo mode (sic!) when saving simple silent photos with L-DNG option (with the modified simple.c, so LV is not paused)  :-[

Previously I reported that it was working, because the file indeed got saved. Unfortunately this is what happens:
- entering LV
- pressing half-shutter
- lossless_compress_raw_rectangle() is started but stucks at the above mentioned line
- I did exit LV at that point...
- and only then the execution continued and file was finally saved.

If LV is not exited by user, you can't trigger second silent picture as the first one is still hanging and waiting.

Shit, so we're worse than thought.

I don't believe this log helps but posting anyway:
Camera start in photo mode -> LV entry -> simple silent photo (which stucks) -> exit LV -> card door open
Scanning modules...
Load modules...
  [i] load: SILENT.MO
Linking..
.text: 0
.data: 0
.config_vars: 0
.module_strings: 0
.text: 0
.data: 0
.config_vars: 0
.module_strings: 0
Register modules...
Load configs...
Init modules...
  [i] Init: 'silent'
  [i] prop 0x80000007
updating Movie Tweaks -> Movie Logging
  [i] prop 0x80000005
  [i] cbr 'CBR_CUSTOM_PICTURE_TAKING' -> 000BE738C
  [i] cbr 'CBR_SHOOT_TASK' -> 000BE8380
  [i] cbr 'CBR_VSYNC' -> 000BE685C
  [i] cbr 'CBR_DISPLAY_FILTER' -> 000BE66FC
Updating symbols...
  [i] 404: edmac_format_size 44d810
  [i] 404: edmac_format_size 44fba0
  [i] 404: edmac_format_size 459780
  [i] 404: edmac_format_size 45b3c0
  [i] 404: dual_iso_get_recovery_iso 4648c0
  [i] 404: dual_iso_is_active 4648c0
  [i] 404: auto_ettr_intervalometer_wait 46f7b0
  [i] 404: auto_ettr_intervalometer_warning 46f7b0
  [i] 404: auto_ettr_export_correction 47b880
  [i] 404: dual_iso_get_dr_improvement 488530
  [i] 404: dual_iso_get_recovery_iso 488530
  [i] 404: edmac_format_size 48d140
Modules loaded
Default raw buffer OK for 1984x761 (2.5MB).
raw update from shoot_task
LV raw buffer: b328000 (1984x761)
Skip left:144 right:8 top:28 bottom:0
Resolution changed: 0x0 -> 1984x761
Subsampling mode: 3B0Sx1B2S (5472x3648 1.62)
LV raw dimensions changed
raw update from shoot_task
LV raw buffer: b328000 (1984x761)
Skip left:144 right:8 top:28 bottom:0
Subsampling mode: 3B0Sx1B2S (5472x3648 1.62)
active area: x=144..1976, y=28..761
lv2raw sx:2565 sy:1529 tx:158 ty:36
raw2lv test: (-6,-6) - (725,485)
  should be: (0,0) - (720,480)
raw2bm test: (-6,-6) - (725,485)
  should be: (0,0) - (720,480)
bm2raw test: (158,36) - (1961,752)
  should be: (144,28) - (1976,761)
Black check 1/5: 2047...5.37, ref 2047...5.55, delta=0
Black check 2/5: 2047...5.21, ref 2047...5.55, delta=0
Black check 3/5: 2048...5.39, ref 2047...5.55, delta=1
Black check 4/5: 2047...5.86, ref 2047...5.55, delta=0
Black check 5/5: 2047...5.36, ref 2047...5.55, delta=0
Black check 1/5: 2047...5.78, ref 2047...5.55, delta=0
Black check 2/5: 2047...5.56, ref 2047...5.55, delta=0
Black check 3/5: 2047...5.79, ref 2047...5.55, delta=0
Black check 4/5: 2047...5.44, ref 2047...5.55, delta=0
Black check 5/5: 2047...5.68, ref 2047...5.55, delta=0
Black level: 2047
dynamic range: 10.70 EV (iso=200)
black=2047 white=10461
raw update from shoot_task
LV raw buffer: b328000 (1984x761)
Skip left:144 right:8 top:28 bottom:0
Subsampling mode: 3B0Sx1B2S (5472x3648 1.62)
Black check 1/5: 2047...5.80, ref 2047...5.64, delta=0
Black check 2/5: 2047...5.71, ref 2047...5.64, delta=0
Black check 3/5: 2047...5.40, ref 2047...5.64, delta=0
Black check 4/5: 2047...5.98, ref 2047...5.64, delta=0
Black check 5/5: 2047...5.56, ref 2047...5.64, delta=0
Black check 1/5: 2047...5.87, ref 2047...5.64, delta=0
Black check 2/5: 2047...5.36, ref 2047...5.64, delta=0
Black check 3/5: 2047...5.55, ref 2047...5.64, delta=0
Black check 4/5: 2047...5.44, ref 2047...5.64, delta=0
Black check 5/5: 2047...5.14, ref 2047...5.64, delta=0
dynamic range: 10.70 EV (iso=200)
black=2047 white=10461
srm_malloc_suite(1)...
[SRM] alloc all buffers
UILock: 00000000 -> 41000001 => 41000001
[SRM] buffer 5ca00064
[SRM] buffer 42000064
[SRM] buffer 44400064
UILock: 41000001 -> 41000000 => 41000000
[SRM] not using 5ca00064
srm_malloc_suite => ff5e8
Freeing LV raw buffer 4b328000.
ArcziPL: starting lossless_compress_raw_rectangle()
ArcziPL: lossless_compress_raw_rectangle, stage 1.
ArcziPL: lossless_compress_raw_rectangle, stage 2.
ArcziPL: lossless_compress_raw_rectangle, stage 3.
ArcziPL: lossless_compress_raw_rectangle, stage 4.
ArcziPL: lossless_compress_raw_rectangle, stage 5.
ArcziPL: lossless_compress_raw_rectangle, stage 6.
ArcziPL: lossless_compress_raw_rectangle, stage 7. <- printed just before TTL_Prepare(TTL_ResLock, &TTL_Args); the camera hangs here until I quit LV
ArcziPL: lossless_compress_raw_rectangle, stage 8. <- printed just after TTL_Prepare(TTL_ResLock, &TTL_Args);
ArcziPL: lossless_compress_raw_rectangle, stage 9.
ArcziPL: lossless_compress_raw_rectangle, stage 10.
ArcziPL: lossless_compress_raw_rectangle, stage 11.
ArcziPL: lossless_compress_raw_rectangle, stage 12.
[TTL] 1984x761 14bpp
WR1: 46800064 EDMAC#17<45> (113428 2.0MB)
WR2: 0 EDMAC#2<22>
RD1: 4400080 EDMAC#25<1>
RD2: 0 EDMAC#25<0>
ArcziPL: lossless_compress_raw_rectangle, stage 13.
ArcziPL: lossless_compress_raw_rectangle, stage 14.
ArcziPL: lossless_compress_raw_rectangle, stage 15.
ArcziPL: lossless_compress_raw_rectangle, stage 16.
ArcziPL: lossless_compress_raw_rectangle, stage 17.
[TTL] Elapsed time: 30483 us
ArcziPL: lossless_compress_raw_rectangle, stage 18.
ArcziPL: lossless_compress_raw_rectangle, stage 19.
[TTL] Output size : 1.3MB (51.25%)
ArcziPL: lossless_compress_raw_rectangle, stage 20.
srm_free_suite(ff5e8)
[SRM] free all buffers
UILock: 41000000 -> 41000000 => 41000000
Save configs...
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 06, 2019, 11:23:55 PM
Alright - it locks up while trying to get exclusive access to some image processing resources (https://www.magiclantern.fm/forum/index.php?topic=6740). This is because something else from Canon's LiveView is using some of them.

I'd say let's find out what resources are causing the lockup (by trial and error, commenting out some of them to narrow down). Once that is done, we can attempt to find what other code attempts to lock the same resources, and maybe - if lucky - we can find a way to disable it somehow.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 07, 2019, 10:13:28 AM
Quote from: a1ex on March 06, 2019, 11:23:55 PMI'd say let's find out what resources are causing the lockup (by trial and error, commenting out some of them to narrow down)

I guess this would mean removing some lines from
    else if (is_camera("5D3", "*") || is_camera("6D", "*") || is_camera("70D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            0x30001,    /* Read connection 1 (uncompressed input) */
            0x2002d,    /* Write connection 45 (compressed output) */
          //0x20016,    /* Write connection 22 (for WR2 - not used) */
            0x50034,
            0x5002d,
            0x50010,
            0x90001,
            0x230000,
            0x160000,
            0x260000,
            0x260001,
            0x260002,
            0x260003,
        };


Is it right?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: a1ex on March 07, 2019, 12:09:15 PM
Yes, that's right. If you comment out all of them, it should go past TTL_Prepare, but... encoding is not going to work.

Besides, if you test this outside LiveView, it may work even without locking some of the resources. For example, IIRC EDMAC channels/connections don't *have* to be locked in order to use them; the locks are just a way to make sure each resource will be used by one task at a time. However, other resources seem to require locking; these routines will also perform some changes to clock enable bits (messages about ClockSave in the log file).

If lucky, some resources might be always locked during LiveView, but they might be used only during mode changes or autofocus or something like that. In this case, encoding might work with these entries commented out.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: ArcziPL on March 07, 2019, 09:28:28 PM
Hell yeah, first 14-bit lossless MLV from 70D! 8)

(https://i.postimg.cc/qtTkyZR4/70d-first-lossless-mlv.png) (https://postimg.cc/qtTkyZR4)

It's amazing what you know about and can do with closed internals of these cams!

The one to be commented out is
//0x5002d,

If too much was commented out (all but first four lines: EDMAC channels and read/write connections) it was still stucking at the same point and additionally locking the camera. LV couldn't be exited. After changing mode between photo/video it thrown Err80.

With only this one line removed I haven't noticed any side effects yet but wasn't focusing on it. I'll test it in the next days/weeks, report back and submit a pull request if all seems to be fine.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: theBilalFakhouri on March 07, 2019, 09:35:57 PM
This is the right time to switch to 70D  :D
Great work!
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Danne on March 07, 2019, 09:38:09 PM
Thumbs up! Great job.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: buro341 on April 11, 2019, 09:00:26 PM
awesome, is there much difference between ML for 70D and 77D?
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: Walter Schulz on April 11, 2019, 09:02:20 PM
Yes, there is a working ML for 70D but not for 77D.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: 2blackbar on June 25, 2019, 05:18:38 PM
Great work ! ArcziPL Do You think 5D2 needs some code to be commented out to prevent camera lockup like 70D ?
Quote from: ArcziPL on March 07, 2019, 09:28:28 PM
Hell yeah, first 14-bit lossless MLV from 70D! 8)

(https://i.postimg.cc/qtTkyZR4/70d-first-lossless-mlv.png) (https://postimg.cc/qtTkyZR4)

It's amazing what you know about and can do with closed internals of these cams!

The one to be commented out is
//0x5002d,

If too much was commented out (all but first four lines: EDMAC channels and read/write connections) it was still stucking at the same point and additionally locking the camera. LV couldn't be exited. After changing mode between photo/video it thrown Err80.

With only this one line removed I haven't noticed any side effects yet but wasn't focusing on it. I'll test it in the next days/weeks, report back and submit a pull request if all seems to be fine.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on July 02, 2020, 12:37:23 AM
Starting to dig in this again for d4 cam's , found something very interesting

        EosCompressLevel :
         Unknown = 0xF,
        JpegUncompressed = 0,
        JpegCompression1 = 1,
        Normal = 2,
        Fine = 3,
        Lossless = 4,
        SuperFine = 5,
        JpegCompression6 = 6,
        JpegCompression7 = 7,
        JpegCompression8 = 8,
        JpegCompression9 = 9,
        JpegCompression10 = 10,


"Lossless = 4" , seems there more compression available then just Lossless  :o

Edit:
found this string
"Primary Image: Size <{0}>, Format <{1}>, CompressLevel <{2}>\n"
+ "Secondary Image: Size <{3}>, Format <{4}>, CompressLevel <{5}>"


So this is what Canon is doing in the background 

1: PrimaryImageSize
2: PrimaryImageFormat
3: PrimaryCompressLevel
4: SecondaryImageSize
5: SecondaryImageFormat
6: SecondaryCompressLevel

There's 2 levels of compression it looks like , input compression ? (from sensor ?) and then export compression ,Lossless ,Uncompressed , etc. ...
Interesting !


Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: names_are_hard on July 02, 2020, 01:42:58 AM
Primary vs secondary could be full size image vs preview?  I'm not sure what to make of Lossless vs JpegUncompressed, unless only the first is Raw?

Definitely looks interesting and worth some experiments.
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on July 02, 2020, 06:51:21 AM
Quote from: names_are_hard on July 02, 2020, 01:42:58 AM
Primary vs secondary could be full size image vs preview?
Yes i thinks so the Log files from my 5d2 supports this plus it looks like the JP62 chip (to my knowledge the d4 encoder chip for lossless/cr2/h264/etc. ...)
is producing 4 different image streams from the sensor .
5616x3744
4080x2720
3861x2574
2784x1856


9.707.568     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(5616) keta(4)
9.708.482     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(3744) keta(4)
.......
9.700.451      CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(4080) keta(4)
9.701.108     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(2720) keta(4)
........
9.711.244     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(3861) keta(4)
9.712.166     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(2574) keta(4)
......
9.703.646     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(2784) keta(4)
9.704.562     CtrlSrv:fface2f4:83:03: DlgShootOlcInfo.c CreateCompoBmpFromNumber(1856) keta(4)


Log for HD resizing

7.264.373  ImgPlayDrv:ffa3d77c:1a:02: RSZ Input  Add:0x40D00000 Basew:2106 x:0 y:0 w:2106 h:1404
7.264.406  ImgPlayDrv:ffa3d7a8:1a:02: RSZ Output Add:0x41B07800 Basew:720 x:0 y:52 w:720 h:428
7.264.426  ImgPlayDrv:ffa3d7d4:1a:02: RSZ pfil:2 pres:2 xxa:117 xxb:80 xya:351 xyb:214
7.264.433  ImgPlayDrv:ffa3e9e4:1a:02: RSZ ColMode:1
7.264.449  ImgPlayDrv:ffa3e624:1a:02: RSZ SetFixedEngineParameter(0,2106)
7.264.460  ImgPlayDrv:ffa3e0e0:1a:02: RSZ ConvertParameterOfInputAndOutput
7.264.495  ImgPlayDrv:ffa3e550:1a:02: RSZ ConvertParameterOfInputAndOutput (Err:0x0)
7.264.512  ImgPlayDrv:ffa3e570:1a:02: RSZ x:0 y:0 w:2106 h:1404
7.264.534  ImgPlayDrv:ffa3e5a4:1a:02: RSZ pfil:2 xx:80/117 pres:2 xy:214/351
7.264.554  ImgPlayDrv:ffa3e5cc:1a:02: RSZ In  xa:0 xb:2106 xn:0 ya:0 yb:1404 yn:0
7.264.573  ImgPlayDrv:ffa3e5f4:1a:02: RSZ Out xa:0 xb:720 xn:0 ya:0 yb:428 yn:0
7.264.597  ImgPlayDrv:ffa3dd80:1a:02: RSZ CorrectPositionAndSize(0,2106)


preview & position i think "RSZ pfil:2 xx:80/117 pres:2 xy:214/351"
I know this is lcd liveview  "RSZ Output Add:0x41B07800 Basew:720 x:0 y:52 w:720 h:428"


Quote from: names_are_hard on July 02, 2020, 01:42:58 AM
I'm not sure what to make of Lossless vs JpegUncompressed, unless only the first is Raw?
Being there are using a Jpeg JP62 chip for encoding image compression , Jpeg must have a uncompressed setting .
from this chart below link , Jpeg2000 has uncompressed & Lossless/Lossy: Wavelet @ 12ibt (38 bit)
http://preservationtutorial.library.cornell.edu/presentation/table7-1.html
looks like its Jpeg2000
https://en.wikipedia.org/wiki/Lossless_JPEG

All this makes sense , since the h264 is Motion Jpeg (Mjpeg)
Log from 5D2 playing a H264.mov file on the LCD screen on camera .
And It's Motion Jpeg !  :o
My dream is still to have MJpeg @ native sensor size (5.6k)
All ml cam would benefit from that , specially the one with slow card write speed.
4.447.781     CtrlSrv:ff84a7b0:2c:03: StartResizeMotionJpeg(0,0x40d00000,0,0x5c007800)
4.447.797     CtrlSrv:ff98ddf4:1a:03: -------------------------------------501
4.447.823     CtrlSrv:ffa4e818:1a:03: [H264Resize] pInputAddress  = 0x40D00000
4.447.844     CtrlSrv:ffa4e82c:1a:03: [H264Resize] pOutputAddress = 0x5C007800
4.447.849     CtrlSrv:ffa4e83c:1a:03: [H264Resize Req]
  4.447.883     CtrlSrv:ffa3f6cc:1a:02: RSZ RequestResizeYuv RdAdr:0x40d00000, WrAdr:0x5c007800
4.447.902     CtrlSrv:ffa3f6f0:00:00: *** LockEngineResources(861d20) x20:
4.447.915     CtrlSrv:000970ec:00:00:      1)    6000f (?)
4.447.926     CtrlSrv:000970ec:00:00:      2)    60009 (?)
4.447.937     CtrlSrv:000970ec:00:00:      3)    6000e (?)
4.447.948     CtrlSrv:000970ec:00:00:      4)    60006 (?)
4.447.960     CtrlSrv:000970ec:00:00:      5)    6000d (?)
4.447.971     CtrlSrv:000970ec:00:00:      6)    6000a (?)
4.447.982     CtrlSrv:000970ec:00:00:      7)    60002 (?)
4.447.994     CtrlSrv:000970ec:00:00:      8)    60001 (?)
4.448.006     CtrlSrv:000970ec:00:00:      9)   130001 (?)
4.448.019     CtrlSrv:000970ec:00:00:     10)   130002 (?)
4.448.032     CtrlSrv:000970ec:00:00:     11)   130010 (?)
4.448.045     CtrlSrv:000970ec:00:00:     12)   130011 (?)
4.448.058     CtrlSrv:000970ec:00:00:     13)   130017 (?)
4.448.071     CtrlSrv:000970ec:00:00:     14)   130018 (?)
4.448.084     CtrlSrv:000970ec:00:00:     15)   130019 (?)
4.448.097     CtrlSrv:000970ec:00:00:     16)   13001a (?)
4.448.110     CtrlSrv:000970ec:00:00:     17)   13001b (?)
4.448.123     CtrlSrv:000970ec:00:00:     18)   13001c (?)
4.448.142     CtrlSrv:000970ec:00:00:     19)    10003 (read channel 0xb)
4.448.160     CtrlSrv:000970ec:00:00:     20)        3 (write channel 0x3)


Edit:
Did a little google searching and found out that digital still camera's (not sure if all )are using JPEG-LS Standard compression
http://dicom.nema.org/medical/dicom/2017a/output/chtml/part05/sect_F.2.html
Quote
The lossless and lossy (near-lossless) coding is based on a predictive scheme with statistical modeling,
in which differences between pixels and their surround are computed and their context modeled prior to coding,
with a run-length escape mechanism.
This scheme achieves consistently better compression in lossless mode than the lossless processes of JPEG defined in ISO 10918-1, with less complexity.
..........
A single JPEG-LS process is used for bit depths up to 16 bits.

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on July 06, 2020, 08:28:35 AM
Taking a another run at Lossless on 5d2 --
One thing i have notice between Lossless dng & Uncompressed dng is when uncompressed dng is being save there's a pause & flicker in liveview
before it save the dng , but in lossless on 5d2 it just freezes liveview , no pause or flicker .

I see in a previous post by a1ex  https://www.magiclantern.fm/forum/index.php?topic=18443.msg212992#msg212992 about this .
So tried to comment out part of the ttl resources to see if this made a differences , No still don't save , just freeze liveview .

At one point i disable all the resources & had a "ERROR 70" flash at bootup , needed a quick battery pull
Here part of the log file with the error
1269:  3309.178 [STARTUP] ERROR ASSERT : Memory\Memory.c, Task = module_task
1270:  3309.268 [STARTUP] ERROR ASSERT : Line 184
1271:  3309.299 [STARTUP] ERROR ASSERT : FALSE
1274:  3309.909 [ENG] CreateResLockEntry cannot allocate pLockEntry->pResource
1277:  3317.781 [STARTUP] ERROR ASSERT : KernelDry\KerSem.c, Task = audio_common_task
1278:  3317.859 [STARTUP] ERROR ASSERT : Line 297
1279:  3317.890 [STARTUP] ERROR ASSERT : 0
1280:  3318.028 [STARTUP] startupErrorRequestChangeCBR : OverWrite (0x1d => 0x1d)
1298:  3398.663 [MC] Disp Error : 70


This line
1274:  3309.909 [ENG] CreateResLockEntry cannot allocate pLockEntry->pResource
Seems to me before i commend out all the resource i must have had Res Lock but I think it couldn't find the path where to save it  ?
Here the log file 5d2_photo-mode-error70-lossless-log000.log (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/5d2_photo-mode-error70-lossless-log000.log)

I need to get my head around the ResLock stuff (https://www.magiclantern.fm/forum/index.php?PHPSESSID=rqaqomnmfv79is923coa6o57u2&topic=6740.msg54663#msg54663) , there's more going on here then i can see .
But the good thing there it saved a total Log file in Photo mode , gave me some nice info
Here is those 4 raw streams being produce again . Be nice to get inside the JP62 chip and see the gears work
1003:  1092.749 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(5616) keta(4)
1004:  1093.519 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(3744) keta(4)
1005:  1094.247 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()
1006:  1095.904 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(4080) keta(4)
1007:  1096.543 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(2720) keta(4)
1008:  1097.312 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()
1009:  1098.747 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(2784) keta(4)
1010:  1099.563 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(1856) keta(4)
1011:  1100.323 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()
1015:  1105.096 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(3861) keta(4)
1016:  1105.905 [GUI] DlgShootOlcInfo.c CreateCompoBmpFromNumber(2574) keta(4)
1017:  1106.825 [GUI] DlgShootOlcInfo.c CreateCompoBmpForImageSize()



   
Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: reddeercity on July 07, 2020, 05:01:56 AM
Found a very good document about Lossless compression from a canon cr2 file ,
breaks down how lossless data is aranged on a image .
Canon_CR2_file_format_ raw bit coder/decode.txt (https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/Canon_CR2_file_format.txt)

QuoteThe forth IFD is the sweet spot of the CR2 file.  It is a lossless JPEG
* compressed image of dimensions the size of the camera's photo sensor.  Canon
* adds three proprietary TIFF tags
Quotethe compressed data is organized as a concatenation of a Huffmann
* code (i.e. HufCode or codeword) followed by a variable length difference code
* (i.e. Diff) bit stream
The RAW image is divided into 4 equal sized
* The finished image inteleaves quadrants row segments 0/1 and interleaves
* quandrant rows segments 2/3 while combined quandrants 0/1 is located to the
* left and combined quadrants 2/3 are concatenated to the right:
*     +------------------------+------------------------+ Finished Image
*     |      Quad0.Row0        |      Quad2.Row0 |
*     |      Quad1.Row0        |      Quad3.Row0 |
*     |      Quad0.Row1        |      Quad2.Row1 |
*     |      Quad1.Row1        |      Quad3.Row1 |
*     | etc        |   etc |
*     |      Quad0.RowN        |      Quad2.RowN |
*     |      Quad1.RowN        |      Quad3.RowN |
*     +------------------------+------------------------+
* So it can be said that Quads 0 and 2 contain the same two BAYER grid colors
* and Quads 1 and 3 contain the other two BAYER grid colors.


etc. ....
Interesting to see how lossless data is organized , since this is what we use for lossless  .

Title: Re: ProcessTwoInTwoOutLosslessPath
Post by: theBilalFakhouri on July 04, 2022, 06:16:24 AM
On 700D there are 16 used resources for TTL_ResLock, I did some experiments and seems using only 5 of them is enough to get valid lossless compression in both RAW video and Silent picture:

    if (is_camera("700D", "*") || is_camera("650D", "*") || is_camera("EOSM", "*") || is_camera("100D", "*"))
    {
        uint32_t resources[] = {
            0x00000 | edmac_channel_to_index(edmac_write_chan),
            0x10000 | edmac_channel_to_index(edmac_read_chan),
            //0x20005,//no effect
            //0x20016,//no effect
            //0x30002,//no effect
            //0x50034,//no effect
            //0x5002d,//no effect
            0x50010,//affect silent pic
            0x90001,//affect silent pic + raw video
            //0x90000,//no effect
            0xa0000,//camera lock up without it
            //0x160000,//no effect
            //0x260000,//no effect
            //0x260001,//no effect
            //0x260002,//no effect
            //0x260003,//no effect
        };


Also, if we want to throw out Silent picture, only using 4 resources would work fine for RAW video which are:
0x00000 | edmac_channel_to_index(edmac_write_chan),
0x10000 | edmac_channel_to_index(edmac_read_chan),
0x90001,
0xa0000,


if we want to make Silent picture works it become:
0x00000 | edmac_channel_to_index(edmac_write_chan),
0x10000 | edmac_channel_to_index(edmac_read_chan),
0x50010, ---> Added this
0x90001,
0xa0000,


-Why I am mentioning this?

I have hope that DIGIC 4 models would have similar thing, by looking back at DIGIC 4 experiments, we can see notes about there are conflicts among some TTL_ResLock resources and LiveView:
Greg's experiment on 500D (https://www.magiclantern.fm/forum/index.php?topic=18443.msg184360#msg184360), a1ex's experiment on 60D (https://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)

-TTL_ResLock resources from logs:

-500D  (https://www.magiclantern.fm/forum/index.php?topic=18443.msg183665#msg183665)from Greg
-7D (https://www.magiclantern.fm/forum/index.php?topic=18443.msg183741#msg183741) from nikfreak
-5D2  (https://www.magiclantern.fm/forum/index.php?topic=18443.msg204882#msg204882)from reddeercity
-550D  (https://www.magiclantern.fm/forum/index.php?topic=18443.msg205252#msg205252)from reddeercity
-50D  (https://www.magiclantern.fm/forum/index.php?topic=18443.msg205270#msg205270)from aprofiti

So the shared resources among DIGIC 4:

1)    10002 (read channel 0xa)
2)        3 (write channel 0x3)
3)        4 (write channel 0x4)
4)    30000 (read connection 0x0)
5)    20005 (write connection 0x5)
6)    20016 (write connection 0x16)
7)    50003 (?)
8)    5000d (?)
9)    5000f (?)
10)    5001a (?)
11)    80000 (?)
12)    90000 (?)
13)    a0000 (?)
14)   160000 (?)
15)   130003 (?)
16)   130004 (?)
17)   130005 (?)


+ These which presented only on 500D and 50D:
18)   130006 (?)
19)   130007 (?)
20)   130008 (?)
21)   13000c (?)
22)   130010 (?)
23)   130011 (?)
24)   130012 (?)
25)   130013 (?)
26)   130014 (?)
27)   130015 (?)
28)   130018 (?)
29)   13001a (?)
30)   13001b (?)
31)   13001c (?)



-Comparing DIGIC 4 resources to 700D:

For first two, resources are for read and write EDMAC channels, we shouldn't worry about them.
For 0xa0000, it's shared among all models DIGIC 4 and DIGIC 5, this seems the most important one, camera locks up without it on 700D
Last one for 700D it's 0x90001, I don't know which one is equivalent on DIGIC 4, could be 0x80000 or 0x90000? , but 0x80000 (https://www.magiclantern.fm/forum/index.php?topic=18443.msg184360#msg184360) was conflicting with LiveView.

For silent pic, 0x50010 was required on 700D, could be 0x5001a on DIGIC 4, or 0x5000X ? but 0x5001a was conflicting with LiveView, Greg's notes (https://www.magiclantern.fm/forum/index.php?topic=18443.msg184360#msg184360)

-Of course I could be wrong, maybe other resources are required too on DIGIC 4, well, maybe not all of them are required, knowing this thing may make things easier.

-I can't test these, no DIGIC 4 models around me, but might provide some tests (not in near future).
-I read that the newer DIGIC 4 models like 60D, 600D and 1100D are the easiest D4 models (https://www.magiclantern.fm/forum/index.php?topic=18443.msg198151#msg198151) to start digging into it because of their similarities to DIGIC 5.
-Probably I need to check what the minimum required resources on 100D (could be same as 700D) and 5D3 and see if the theory applies there.