Author Topic: ProcessTwoInTwoOutLosslessPath  (Read 37726 times)

vorob

  • New to the forum
  • *
  • Posts: 7
Re: ProcessTwoInTwoOutLosslessPath
« Reply #75 on: January 10, 2017, 09:09:08 AM »
Hi, can anyone explain what's going on here? Why do we need this? CR2 files are compressed? Lost smth? Not 14bit? Thank you and sorry for my lack of knowledge.

Walter Schulz

  • Hero Member
  • *****
  • Posts: 5313
Re: ProcessTwoInTwoOutLosslessPath
« Reply #76 on: January 10, 2017, 09:16:40 AM »
CR2 out of the cam (with and without ML): Lossless compressed
DNG/MLV/RAW out of ML (produced by some ML features, not all of them): Uncompressed.

This thread is about enabling ML to produce lossless compressed output, too.
And compression means less bandwidth required writing to card.

vorob

  • New to the forum
  • *
  • Posts: 7
Re: ProcessTwoInTwoOutLosslessPath
« Reply #77 on: January 10, 2017, 09:20:19 AM »
Quote
CR2 out of the cam (with and without ML): Lossless compressed
DNG/MLV/RAW out of ML (produced by some ML features, not all of them): Uncompressed.

This thread is about enabling ML to produce lossless compressed output, too.
And compression means less bandwidth required writing to card.

Quote
Files are 15-20% smaller and image previews load faster in lightroom. The disadvantage is some metadata is discarded and dng files don't play well with some non-adobe software. There's also an argument that dng files are more future proof, but since old dng files are different to new dng files, I don't buy that argument; I'm sure I'll still be able to access/convert cr2 files in 20 years if it becomes and issue.

Basically if you never intend to move outside of Adobe's ecosystem then it's a nice choice for saving some hard drive space. With modern hardware the speed advantage will be minimal for most people, and there's the risk if you move to capture one or dxo (say) in the future, you'll have issues with camera profiles and lens correction not working accurately (despite the format being semi-supported).

Well, i still don't get it... :(

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9574
  • 5D Mark Free
Re: ProcessTwoInTwoOutLosslessPath
« Reply #78 on: January 10, 2017, 10:12:20 AM »
Ignore the second quote, as it does not apply to this thread at all.

What's researched here does not provide any reason to replace the CR2 format for regular still pictures.

- Better compression? In our DNGs, compression is not any better or worse than CR2; it is identical.
- Saving a conversion step (if you convert to DNG anyway)? Not a good reason for justifying the development effort IMO.
- Additional EXIF info? Maybe, but XMP works and is minimally invasive to Canon code.
- "Quality loss from converting your Canon RAW files into DNGs"? There's none. Once uncompressed, raw data is 100% identical (check with e.g. dcraw -4 -E after converting a CR2 with Adobe DNG Converter).

It's all about the files saved by ML features (silent pictures, possibly raw video). Maybe also opening the door to MJPEG.

Walter Schulz

  • Hero Member
  • *****
  • Posts: 5313
Re: ProcessTwoInTwoOutLosslessPath
« Reply #79 on: January 10, 2017, 10:37:29 AM »
@vorob: Your second quote comes from Canonrumours.
Lesson to be learnt: Don't trust anything written about ML outside ML forum. They tend to mix up things badly. And almost never you will find a corrections section.


Walter Schulz

  • Hero Member
  • *****
  • Posts: 5313
Re: ProcessTwoInTwoOutLosslessPath
« Reply #81 on: January 10, 2017, 01:31:18 PM »
May I ask if such an incident ever happened? I'm willing to bet it hasn't in the past and won't happen in the future.
If I'm loosing the bet I will also win because it will tell me about a reliable news site.

Maximo

  • New to the forum
  • *
  • Posts: 10
Re: ProcessTwoInTwoOutLosslessPath
« Reply #82 on: January 27, 2017, 02:54:38 PM »
Hi guys,
How likely will we have 4K lossless video DNG files on the 5D mark IV?  Many thanks.

Walter Schulz

  • Hero Member
  • *****
  • Posts: 5313
Re: ProcessTwoInTwoOutLosslessPath
« Reply #83 on: January 27, 2017, 02:58:59 PM »
58.12699378 %

Licaon_Kter

  • Senior
  • ****
  • Posts: 494
  • M
Re: ProcessTwoInTwoOutLosslessPath
« Reply #84 on: January 27, 2017, 03:24:36 PM »
@Maximo: Each new 5DIV camera that @a1ex receives for free in order to develop ML for, will increase chances by 5%!

Wayne H

  • Just arrived
  • *
  • Posts: 1
Re: ProcessTwoInTwoOutLosslessPath
« Reply #85 on: January 27, 2017, 04:41:29 PM »
Really excited by this find and development and where it could potentially lead.

Motion JPeg, or dng at 50-60% compressed of the original file size and higher resolutions would be just unbelievable... maybe a april 1st announcement :D, fingers crossed.

Massive thank you to the whole magic lantern community for this amazing software and a special thanks to the core developers.

Any news on how things are progressing?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9574
  • 5D Mark Free
Re: ProcessTwoInTwoOutLosslessPath
« Reply #86 on: March 20, 2017, 05:44:26 PM »
After fiddling a bit with these routines, I have both good and bad news:

- good: 700D has routines very similar to 5D3 (named TwoInTwoOutJpegPath; not tested)
- good: 5D3 has two sets of routines for lossless compression (TwoInTwoOutLosslessPath and TwoInTwoOutJpegPath); both are working (second one requires some fiddling)
- bad: the routines are a bit different on 60D (not very LiveView-friendly)
- ugly: I wasn't able to decode the compressed images from 60D (just corrupted images), but didn't try too hard

- good: output bit depth (0xC0E00084) can be overriden from 14 to 15 bits
- info: a file with 15 bits is just 3 bytes (!) higher than a 14-bits one (headers show 15 bits and contents is totally different for same input image)
- bad: other bit depths (8, 10, 12) did not work at all
- bad: bit depth overriding did not work on 5D3 at all (maybe I'm missing something)

- good: the routines can accept raw image streams of arbitrary bit depths (10/12/14/16, configured with 0xC0F371FC)
- bad: the above doesn't appear to improve compression ratio at all
- ugly: compressing a zeroed out image gives a ratio of... 35.7%! (very large encoder overhead?)

What happens:
- output levels do not depend on input bit depth
- instead, 10-bit streams are padded with zero LSBs; the end result is the same as zeroing the lower bits.

From the 14/15-bit experiment, dividing the input data by some constant value (e.g. dividing by 16 for 14->10 bit conversion) probably has the same impact on file size as if we were able to reconfigure the encoder for a lower bit depth (which we can't). That's my current understanding, but I have no hard proof on that (as I don't even know how the encoder works).

Some numbers:


Note: the identical checksum confirms that compressing a 10-bit raw image gives the same result as converting a 14-bit image with 4 LSBs zeroed out (therefore, the unpacker module probably normalizes the values internally by adding LSBs).

Bottom line: pushing the lossless compression ratio below 50% (relative to 14-bit uncompressed raw size) is hard. Of course, it depends on noise levels and scene contents, unlike uncompressed raw, but the variations appear to be small.



(M)JPEG is, indeed, done with the same routines. The main difference is the type of data accepted as input (8-bit debayered, probably hardcoded to YUV422).


DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2771
  • 5D3 / M1 / 7D / 70D / SL1
Re: ProcessTwoInTwoOutLosslessPath
« Reply #88 on: March 20, 2017, 10:17:31 PM »
This is all mostly good news to me and excellent work as always @a1ex!
5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101