ProcessTwoInTwoOutLosslessPath

Started by a1ex, December 18, 2016, 09:06:41 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

vorob

Hi, can anyone explain what's going on here? Why do we need this? CR2 files are compressed? Lost smth? Not 14bit? Thank you and sorry for my lack of knowledge.

Walter Schulz

CR2 out of the cam (with and without ML): Lossless compressed
DNG/MLV/RAW out of ML (produced by some ML features, not all of them): Uncompressed.

This thread is about enabling ML to produce lossless compressed output, too.
And compression means less bandwidth required writing to card.

vorob

QuoteCR2 out of the cam (with and without ML): Lossless compressed
DNG/MLV/RAW out of ML (produced by some ML features, not all of them): Uncompressed.

This thread is about enabling ML to produce lossless compressed output, too.
And compression means less bandwidth required writing to card.

QuoteFiles are 15-20% smaller and image previews load faster in lightroom. The disadvantage is some metadata is discarded and dng files don't play well with some non-adobe software. There's also an argument that dng files are more future proof, but since old dng files are different to new dng files, I don't buy that argument; I'm sure I'll still be able to access/convert cr2 files in 20 years if it becomes and issue.

Basically if you never intend to move outside of Adobe's ecosystem then it's a nice choice for saving some hard drive space. With modern hardware the speed advantage will be minimal for most people, and there's the risk if you move to capture one or dxo (say) in the future, you'll have issues with camera profiles and lens correction not working accurately (despite the format being semi-supported).

Well, i still don't get it... :(

a1ex

Ignore the second quote, as it does not apply to this thread at all.

What's researched here does not provide any reason to replace the CR2 format for regular still pictures.

- Better compression? In our DNGs, compression is not any better or worse than CR2; it is identical.
- Saving a conversion step (if you convert to DNG anyway)? Not a good reason for justifying the development effort IMO.
- Additional EXIF info? Maybe, but XMP works and is minimally invasive to Canon code.
- "Quality loss from converting your Canon RAW files into DNGs"? There's none. Once uncompressed, raw data is 100% identical (check with e.g. dcraw -4 -E after converting a CR2 with Adobe DNG Converter).

It's all about the files saved by ML features (silent pictures, possibly raw video). Maybe also opening the door to MJPEG.

Walter Schulz

@vorob: Your second quote comes from Canonrumours.
Lesson to be learnt: Don't trust anything written about ML outside ML forum. They tend to mix up things badly. And almost never you will find a corrections section.

g3gg0

well, one could ask before posting news.
and the best was to do that is... asking on the forum, either as thread or if necessary via PM to us core devs.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Walter Schulz

May I ask if such an incident ever happened? I'm willing to bet it hasn't in the past and won't happen in the future.
If I'm loosing the bet I will also win because it will tell me about a reliable news site.

Maximo

Hi guys,
How likely will we have 4K lossless video DNG files on the 5D mark IV?  Many thanks.

Walter Schulz


Licaon_Kter

@Maximo: Each new 5DIV camera that @a1ex receives for free in order to develop ML for, will increase chances by 5%!

Wayne H

Really excited by this find and development and where it could potentially lead.

Motion JPeg, or dng at 50-60% compressed of the original file size and higher resolutions would be just unbelievable... maybe a april 1st announcement :D, fingers crossed.

Massive thank you to the whole magic lantern community for this amazing software and a special thanks to the core developers.

Any news on how things are progressing?

a1ex

After fiddling a bit with these routines, I have both good and bad news:

- good: 700D has routines very similar to 5D3 (named TwoInTwoOutJpegPath; not tested)
- good: 5D3 has two sets of routines for lossless compression (TwoInTwoOutLosslessPath and TwoInTwoOutJpegPath); both are working (second one requires some fiddling)
- bad: the routines are a bit different on 60D (not very LiveView-friendly)
- ugly: I wasn't able to decode the compressed images from 60D (just corrupted images), but didn't try too hard

- good: output bit depth (0xC0E00084) can be overriden from 14 to 15 bits
- info: a file with 15 bits is just 3 bytes (!) higher than a 14-bits one (headers show 15 bits and contents is totally different for same input image)
- bad: other bit depths (8, 10, 12) did not work at all
- bad: bit depth overriding did not work on 5D3 at all (maybe I'm missing something)

- good: the routines can accept raw image streams of arbitrary bit depths (10/12/14/16, configured with 0xC0F371FC)
- bad: the above doesn't appear to improve compression ratio at all
- ugly: compressing a zeroed out image gives a ratio of... 35.7%! (very large encoder overhead?)

What happens:
- output levels do not depend on input bit depth
- instead, 10-bit streams are padded with zero LSBs; the end result is the same as zeroing the lower bits.

From the 14/15-bit experiment, dividing the input data by some constant value (e.g. dividing by 16 for 14->10 bit conversion) probably has the same impact on file size as if we were able to reconfigure the encoder for a lower bit depth (which we can't). That's my current understanding, but I have no hard proof on that (as I don't even know how the encoder works).

Some numbers:


Note: the identical checksum confirms that compressing a 10-bit raw image gives the same result as converting a 14-bit image with 4 LSBs zeroed out (therefore, the unpacker module probably normalizes the values internally by adding LSBs).

Bottom line: pushing the lossless compression ratio below 50% (relative to 14-bit uncompressed raw size) is hard. Of course, it depends on noise levels and scene contents, unlike uncompressed raw, but the variations appear to be small.




(M)JPEG is, indeed, done with the same routines. The main difference is the type of data accepted as input (8-bit debayered, probably hardcoded to YUV422).

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

DeafEyeJedi

This is all mostly good news to me and excellent work as always @a1ex!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

a1ex

Quote from: g3gg0 on December 25, 2016, 10:28:49 PM
-> MAYBE on april 1st ;)

Good prediction. It's working - experimental build posted.



To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).

Baldand's LJ92 decoder from mlv_rec_lj92 branch almost decodes the payload, but the image data is rendered incorrectly for some reason. Didn't try to debug it yet, as I'm far from understanding how lossless JPEG compression works.

g3gg0

finally! in-camera compressed MLV files. yay!
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!


andy kh

5D Mark III - 70D

hyalinejim

Amazing! Had my fingers quietly crossed for this.

janvkem

Great to see this getting along! Just tested and the DNG files are ~2.3 MB each whereas they normally are ~3.5 MB (1920x1080), so ~65% compression! It seems that there is still some bug inside however: When I record it crashes after a couple of seconds. There is an MLV file created on the card. Inspecting it on my pc showed roughly a double size of the MLV file compared to the extracted DNG files (819 MB for the MLV file and 387 MB for the extracted DNG files).

a1ex

Can you share your settings directory, so I can try to reproduce?

hyalinejim

Ran camera for 11 minutes at 3168 x 1320 24p. First frame of the recording looks good to me:

https://www.dropbox.com/s/8trg0exly76nx9a/m01-1531_000000.dng?dl=0

However mlv dump gave up after 101 frames with this message:

Unknown Block: )Nh,, skipping
[ERROR] Invalid block size at position 0x1981e600
Processed 101 video frames


a1ex

Output of "mlv_dump file.mlv -v" ?

The longest clip I've tried was a 3GB 1920x1080 - 1867 frames.

edit: 3168 x 1320 24p (24.006 in x5 zoom), with grayscale preview, ran for 2 minutes, 3 chunks (MLV, M00, M01), 2908 frames. All seems OK. I've used the build from the experimental page for 1.2.3 (not a local compilation).

MitchLally

Wow. This is incredible.

Will there be a custom MLV_DUMP for mac? :)

MCAN