Author Topic: ProcessTwoInTwoOutLosslessPath  (Read 54978 times)

janvkem

  • New to the forum
  • *
  • Posts: 10
Re: ProcessTwoInTwoOutLosslessPath
« Reply #100 on: April 01, 2017, 05:28:25 PM »
Can you share your settings directory, so I can try to reproduce?

Sorry, my bad. Checked again with a completely fresh card (other test was just with deleting the ML folder and autoexec.bin file) and issue appears to be gone. I had 2 additional files on the card but cannot recall the names (removed them before I saw your post unfortunately)... My apologies for the incorrect information.

hyalinejim

  • Member
  • ***
  • Posts: 103
Re: ProcessTwoInTwoOutLosslessPath
« Reply #101 on: April 01, 2017, 05:39:11 PM »

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9933
  • 5D Mark Free
Re: ProcessTwoInTwoOutLosslessPath
« Reply #102 on: April 01, 2017, 05:43:31 PM »
Yes; unfortunately, it doesn't give me a clue about what happened...

Would be best if you can find a way to reproduce.

hyalinejim

  • Member
  • ***
  • Posts: 103
Re: ProcessTwoInTwoOutLosslessPath
« Reply #103 on: April 01, 2017, 06:08:56 PM »
I should note that on my first recording I had modules mlv_rec, _play and _lite loaded: I got the grayscale preview for a very brief time and then the LCD switched to some kind of report log that I haven't seen before.

2nd recording I switched off all modules except mlv_lite: I got greyscale preview continuously but recording stopped automagically after around 4 minutes.

3rd recording lowered resolution to 2880 x 1320 and very quickly got the onscreen log again:

https://s17.postimg.org/6fokginq7/20170401_170314.jpg

Here are my original settings (first recording) if it helps:

https://www.dropbox.com/s/d07hnipovcsrpf4/SETTINGS.zip?dl=0


marekk

  • Senior
  • ****
  • Posts: 331
  • www.oddajmisie.com

eNnvi

  • New to the forum
  • *
  • Posts: 39
  • 700D
Re: ProcessTwoInTwoOutLosslessPath
« Reply #105 on: April 01, 2017, 08:25:29 PM »
So how can we help to Port this on other cameras?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9933
  • 5D Mark Free
Re: ProcessTwoInTwoOutLosslessPath
« Reply #106 on: April 01, 2017, 08:36:31 PM »
[ERROR] Invalid block size at position 0x1981e600

Diagnosed the above issue (it was related to variable frame sizes) and hopefully fixed it. Tested by filling the card a couple of times and checking the validity of the recorded clip with mlv_play (it doesn't play the video yet, but from the error message it's possible to tell whether the file structure is valid or not).

New experimental build available.

Will explain how it works later.

hyalinejim

  • Member
  • ***
  • Posts: 103
Re: ProcessTwoInTwoOutLosslessPath
« Reply #107 on: April 01, 2017, 09:11:58 PM »
Nice one!

EDIT: Checked out new build and I'm happily recording at 3072 x 1286. Around 37 minutes into that take now and temperature is showing as 55 degrees. Previously I'd never really seen it go above around 45.

lostfeliz

  • New to the forum
  • *
  • Posts: 43
  • 5diii. 60d.
Re: ProcessTwoInTwoOutLosslessPath
« Reply #108 on: April 01, 2017, 10:17:02 PM »
37 minutes at 3072 x 1286?! Wow. How many GB was the whole clip?

Ilia3101

  • Member
  • ***
  • Posts: 112
Re: ProcessTwoInTwoOutLosslessPath
« Reply #109 on: April 01, 2017, 10:40:41 PM »
hyalinejim is that losslessly compressed video you were able to record for 37 minutes?
5D2

hyalinejim

  • Member
  • ***
  • Posts: 103
Re: ProcessTwoInTwoOutLosslessPath
« Reply #110 on: April 01, 2017, 11:49:00 PM »
37 minutes at 3072 x 1286?! Wow. How many GB was the whole clip?

Just checked the card: 208GB.

hyalinejim is that losslessly compressed video you were able to record for 37 minutes?

Yes, indeed. By comparison, at the same resolution on my card I got around seven seconds in 10 bit. To be clear, I stopped the lossless 14bit recording at 37 minutes. It's only limited by the size of your recording media.

Danne

  • Hero Member
  • *****
  • Posts: 2946
Re: ProcessTwoInTwoOutLosslessPath
« Reply #111 on: April 01, 2017, 11:53:24 PM »
Lossless 14bit files 3k :o
I need my camera already.

beauchampy

  • Senior
  • ****
  • Posts: 289
Re: ProcessTwoInTwoOutLosslessPath
« Reply #112 on: April 02, 2017, 03:06:07 AM »
Hey a1ex! Really nice job! I'm still wrapping my head around all these incredible developments.

Questions about the lossless compression - it says compressed with Canon lossless JPEG, but this is still raw, right? How does the information differ to the normal 14-bit raw files?

Markus

  • Senior
  • ****
  • Posts: 310
Re: ProcessTwoInTwoOutLosslessPath
« Reply #113 on: April 02, 2017, 06:44:10 AM »
Still can't believe all this incredible progress isn't just a big april fools joke.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9933
  • 5D Mark Free
Re: ProcessTwoInTwoOutLosslessPath
« Reply #114 on: April 02, 2017, 08:12:08 AM »
Questions about the lossless compression - it says compressed with Canon lossless JPEG, but this is still raw, right? How does the information differ to the normal 14-bit raw files?

Lossless JPEG is what you have in regular CR2 files and compressed DNGs. It's not the 8-bit JPEG you are used with.

I was also fooled by this for a long time - the routines used to compress the CR2, on most camera models, had JPEG in their name, so I didn't look into them, thinking they were for regular 8-bit JPEG compression...

https://en.wikipedia.org/wiki/Lossless_JPEG
http://lclevy.free.fr/cr2/#lossless
https://bitbucket.org/baldand/mlrawviewer/src/e7abaaf4cf9be66f46e0c8844297be0e7d88c288/LJ92.py
https://bitbucket.org/hudson/magic-lantern/commits/83ced42a1009db90ff33ceae5e4e198fe5943b47

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 946
Re: ProcessTwoInTwoOutLosslessPath
« Reply #115 on: April 02, 2017, 09:25:22 AM »
Have to ask it:only 14/15bit?
70D.112 & 100D.101

bouncyball

  • Member
  • ***
  • Posts: 199
Re: ProcessTwoInTwoOutLosslessPath
« Reply #116 on: April 02, 2017, 10:04:02 AM »
@a1ex: Happiest April 1st ever! :D

Just checked the card: 208GB.

Yes, indeed. By comparison, at the same resolution on my card I got around seven seconds in 10 bit. To be clear, I stopped the lossless 14bit recording at 37 minutes. It's only limited by the size of your recording media.
@hyalinejim: Hmm... it's about 93mb/sec. Which CF card did you use?

Levas

  • Hero Member
  • *****
  • Posts: 863
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #117 on: April 02, 2017, 11:14:13 AM »
YEs, Canon standard CR2 compression for RAW video  :D

How hard is this to port to other cams, is it just a few lines of routines to be altered or is it a pain in the :o to port  :P ?

I hear compression ratios of 58%, is that true ? Because in that case it would be even better then 10 bit (about 70% smaller compared to 14 bit)

hyalinejim

  • Member
  • ***
  • Posts: 103
Re: ProcessTwoInTwoOutLosslessPath
« Reply #118 on: April 02, 2017, 11:24:09 AM »
@hyalinejim: Hmm... it's about 93mb/sec. Which CF card did you use?

OK, so the long file I recorded is 218,855,342 KB and it's 55,874 frames at 3072 x 1286 resolution.

Card is Komputerbay 256GB 1066x

Levas

  • Hero Member
  • *****
  • Posts: 863
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #119 on: April 02, 2017, 11:27:29 AM »
That's about 57% compression ratio  :o

eNnvi

  • New to the forum
  • *
  • Posts: 39
  • 700D
Re: ProcessTwoInTwoOutLosslessPath
« Reply #120 on: April 02, 2017, 01:21:08 PM »
@levas no, as Alex tested a few weeks ago the compression would be the same as a 10bit stream would give the same results of a 14 bit as it's used as a 10bit with leading zeros. So it's not worth to use a 10bit stream with this method.

Just I'd like to know how I can help to Port this on other cameras and if this would get rid of focus pixels on many cams like 700D. Would be nice to know if it's the same routine to compress and delete focus pixels :D

Levas

  • Hero Member
  • *****
  • Posts: 863
  • 6d - Nightly build user
Re: ProcessTwoInTwoOutLosslessPath
« Reply #121 on: April 02, 2017, 01:34:37 PM »
Quote
OK, so the long file I recorded is 218,855,342 KB and it's 55,874 frames at 3072 x 1286 resolution.

The above gives about 4Mb frames which should be about 7Mb uncompressed, that's 57% isn't it?

Edit: I know that it doesn't work with 10 bit stream, which would be even more great.
But to me it looks like 14 bit compressed is more efficient  then 10 bit.
57% vs 71%

bouncyball

  • Member
  • ***
  • Posts: 199
Re: ProcessTwoInTwoOutLosslessPath
« Reply #122 on: April 02, 2017, 02:36:00 PM »
Just I'd like to know how I can help to Port this on other cameras and if this would get rid of focus pixels on many cams like 700D. Would be nice to know if it's the same routine to compress and delete focus pixels :D
The focus pixel issue has no relation to canon HW compression at all.

eyeland

  • Member
  • ***
  • Posts: 153
  • 5D Mark Spree
Re: ProcessTwoInTwoOutLosslessPath
« Reply #123 on: April 04, 2017, 01:59:30 AM »
This is amazing :D
Has anyone played around with cranking frame-rates up instead of resolution?
Daybreak broke me loose and brought me back...

COMMANDES

  • New to the forum
  • *
  • Posts: 20
Re: ProcessTwoInTwoOutLosslessPath
« Reply #124 on: April 05, 2017, 01:14:49 PM »
How soon will it be possible to test a compressed RAW at 650D?
Canon 650D, Canon EF-S 18-55mm f/3.5-5.6, Canon EF 40mm f/2.8 STM, Tamron EF 28-75mm f/2.8, Canon EF 70-200mm f/4L USM, Helios 44M 58mm f/2