ProcessTwoInTwoOutLosslessPath

Started by a1ex, December 18, 2016, 09:06:41 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

janvkem

Quote from: a1ex on April 01, 2017, 05:02:05 PM
Can you share your settings directory, so I can try to reproduce?

Sorry, my bad. Checked again with a completely fresh card (other test was just with deleting the ML folder and autoexec.bin file) and issue appears to be gone. I had 2 additional files on the card but cannot recall the names (removed them before I saw your post unfortunately)... My apologies for the incorrect information.


a1ex

Yes; unfortunately, it doesn't give me a clue about what happened...

Would be best if you can find a way to reproduce.

hyalinejim

I should note that on my first recording I had modules mlv_rec, _play and _lite loaded: I got the grayscale preview for a very brief time and then the LCD switched to some kind of report log that I haven't seen before.

2nd recording I switched off all modules except mlv_lite: I got greyscale preview continuously but recording stopped automagically after around 4 minutes.

3rd recording lowered resolution to 2880 x 1320 and very quickly got the onscreen log again:

https://s17.postimg.org/6fokginq7/20170401_170314.jpg

Here are my original settings (first recording) if it helps:

https://www.dropbox.com/s/d07hnipovcsrpf4/SETTINGS.zip?dl=0



eNnvi

So how can we help to Port this on other cameras?

a1ex

Quote from: hyalinejim on April 01, 2017, 05:04:10 PM
[ERROR] Invalid block size at position 0x1981e600

Diagnosed the above issue (it was related to variable frame sizes) and hopefully fixed it. Tested by filling the card a couple of times and checking the validity of the recorded clip with mlv_play (it doesn't play the video yet, but from the error message it's possible to tell whether the file structure is valid or not).

New experimental build available.

Will explain how it works later.

hyalinejim

Nice one!

EDIT: Checked out new build and I'm happily recording at 3072 x 1286. Around 37 minutes into that take now and temperature is showing as 55 degrees. Previously I'd never really seen it go above around 45.

lostfeliz

37 minutes at 3072 x 1286?! Wow. How many GB was the whole clip?

ilia3101

hyalinejim is that losslessly compressed video you were able to record for 37 minutes?

hyalinejim

Quote from: lostfeliz on April 01, 2017, 10:17:02 PM
37 minutes at 3072 x 1286?! Wow. How many GB was the whole clip?

Just checked the card: 208GB.

Quote from: Ilia3101 on April 01, 2017, 10:40:41 PM
hyalinejim is that losslessly compressed video you were able to record for 37 minutes?

Yes, indeed. By comparison, at the same resolution on my card I got around seven seconds in 10 bit. To be clear, I stopped the lossless 14bit recording at 37 minutes. It's only limited by the size of your recording media.

Danne

Lossless 14bit files 3k :o
I need my camera already.

beauchampy

Hey a1ex! Really nice job! I'm still wrapping my head around all these incredible developments.

Questions about the lossless compression - it says compressed with Canon lossless JPEG, but this is still raw, right? How does the information differ to the normal 14-bit raw files?

Markus

Still can't believe all this incredible progress isn't just a big april fools joke.

a1ex

Quote from: beauchampy on April 02, 2017, 03:06:07 AM
Questions about the lossless compression - it says compressed with Canon lossless JPEG, but this is still raw, right? How does the information differ to the normal 14-bit raw files?

Lossless JPEG is what you have in regular CR2 files and compressed DNGs. It's not the 8-bit JPEG you are used with.

I was also fooled by this for a long time - the routines used to compress the CR2, on most camera models, had JPEG in their name, so I didn't look into them, thinking they were for regular 8-bit JPEG compression...

https://en.wikipedia.org/wiki/Lossless_JPEG
http://lclevy.free.fr/cr2/#lossless
https://bitbucket.org/baldand/mlrawviewer/src/e7abaaf4cf9be66f46e0c8844297be0e7d88c288/LJ92.py
https://bitbucket.org/hudson/magic-lantern/commits/83ced42a1009db90ff33ceae5e4e198fe5943b47

nikfreak

[size=8pt]70D.112 & 100D.101[/size]

bouncyball

@a1ex: Happiest April 1st ever! :D

Quote from: hyalinejim on April 01, 2017, 11:49:00 PM
Just checked the card: 208GB.

Yes, indeed. By comparison, at the same resolution on my card I got around seven seconds in 10 bit. To be clear, I stopped the lossless 14bit recording at 37 minutes. It's only limited by the size of your recording media.
@hyalinejim: Hmm... it's about 93mb/sec. Which CF card did you use?

Levas

YEs, Canon standard CR2 compression for RAW video  :D

How hard is this to port to other cams, is it just a few lines of routines to be altered or is it a pain in the :o to port  :P ?

I hear compression ratios of 58%, is that true ? Because in that case it would be even better then 10 bit (about 70% smaller compared to 14 bit)

hyalinejim

Quote from: bouncyball on April 02, 2017, 10:04:02 AM
@hyalinejim: Hmm... it's about 93mb/sec. Which CF card did you use?

OK, so the long file I recorded is 218,855,342 KB and it's 55,874 frames at 3072 x 1286 resolution.

Card is Komputerbay 256GB 1066x

Levas

That's about 57% compression ratio  :o

eNnvi

@levas no, as Alex tested a few weeks ago the compression would be the same as a 10bit stream would give the same results of a 14 bit as it's used as a 10bit with leading zeros. So it's not worth to use a 10bit stream with this method.

Just I'd like to know how I can help to Port this on other cameras and if this would get rid of focus pixels on many cams like 700D. Would be nice to know if it's the same routine to compress and delete focus pixels :D

Levas

QuoteOK, so the long file I recorded is 218,855,342 KB and it's 55,874 frames at 3072 x 1286 resolution.

The above gives about 4Mb frames which should be about 7Mb uncompressed, that's 57% isn't it?

Edit: I know that it doesn't work with 10 bit stream, which would be even more great.
But to me it looks like 14 bit compressed is more efficient  then 10 bit.
57% vs 71%

bouncyball

Quote from: eNnvi on April 02, 2017, 01:21:08 PM
Just I'd like to know how I can help to Port this on other cameras and if this would get rid of focus pixels on many cams like 700D. Would be nice to know if it's the same routine to compress and delete focus pixels :D
The focus pixel issue has no relation to canon HW compression at all.

eyeland

This is amazing :D
Has anyone played around with cranking frame-rates up instead of resolution?
Daybreak broke me loose and brought me back...

COMMANDES

How soon will it be possible to test a compressed RAW at 650D?
Canon 650D, EOSM 2.02, M50 1.1.0