dual_iso: Option to embed original cr2 in converted dng

[Marsu42]

The dng spec has the option to embed a raw file like cr2 in the converted dng for safekeeping if it should be required later on, for whatever reason. The embedded can be extracted or deleted later on.

Since the cr2hdr process is lossy and a newer version might have better results, I'm currently keeping all old DUAL cr2 around, resulting in quite a messy archival and tagging situation. It would be great if cr2hdr would have the option to embed the original cr2 in the dng so both are kept in the same place, though of course at the cost of a larger file - but if the original file is to be kept anyway this doesn't matter.

[Audionut]

I had a quick look through the exiftool docs and couldn't see a way to do it.

I assume it could be done via chdk-dng and implemented as a command line option in cr2hdr. 
For re-processing the CR2s (as cr2hdr improves), how would this be handled?  Extract CR2, delete the original DNG and output new DNG(CR2)?

Ideally, this would only be needed until cr2hdr progresses to the point where keeping the original CR2s becomes pointless.

[Marsu42]

I assume it could be done via chdk-dng and implemented as a command line option in cr2hdr. 

Any possible way is a good way as long as it works :-) and this option imho could raise the acceptance to adopt dual_iso, it surely would for me.

For re-processing the CR2s (as cr2hdr improves), how would this be handled?  Extract CR2, delete the original DNG and output new DNG(CR2)?

This surely would be "nice to have", but even just the option to keep the cr2 around in the same place would be enough, it could be left to the user to extract and re-process it with a newer cr2hdr.

Ideally, this would only be needed until cr2hdr progresses to the point where keeping the original CR2s becomes pointless.

Just like other lossy postprocessing like denoising the question is: When will that point be? It might be possible that 5 years from now someone figures out to convert dual_iso images much better than thought possible now, you never know...

[Audionut]

This surely would be "nice to have", but even just the option to keep the cr2 around in the same place would be enough, it could be left to the user to extract and re-process it with a newer cr2hdr.

I honestly do not understand this at all. 

If you have the CR2 embeded in the DNG.  What should happen to the CR2 and original DNG when being processed again?  Describe it in detail please as,

Any possible way is a good way as long as it works :-)

is clearly dependent on each users interpretation of the idea of working.  As far as I am concerned, it works fine just the way it is now, for instance. 

File management in underrated.

[Marsu42]

I honestly do not understand this at all.  If you have the CR2 embeded in the DNG.  What should happen to the CR2 and original DNG when being processed again?  Describe it in detail please

Now I don't understand you - processed again by what? If you further postprocess the dng in for example lightroom the embedded (dual_iso) cr2 isn't touched at all but just carried along. If another, better cr2hdr is released, you can extract the cr2 and convert it again, it's really just like a filesystem backup but with the convenience of having one file in the same place and you just have to tag one file.

File management in underrated.

Again, this a personal approach, and it is very commendable if you're up to the task - but Adobe who do have some clue about software usage scenarios created the "raw in dng" spec for a reason, keeping the files at the same place is one good and fool-proof way of keeping the original even if it's not the only one or your approach for this matter, and again there is no reason why any approach should be the only one and invalidate the other.

[Audionut]

If another, better cr2hdr is released, you can extract the cr2 and convert it again, it's really just like a filesystem backup but with the convenience of having one file in the same place and you just have to tag one file.

Sure you can extract the CR2 and process again.  But, wouldn't it be better to automatically have cr2hdr extract the CR2, delete the old DNG, process the CR2 again, and embed the CR2 in the new DNG?

Else you're just going to end up with DNGs all over the place that require user control.  And to make matters worse, these DNGs could have the CR2s still embedded in them (increased filesize).

but Adobe who do have some clue about software usage scenarios

Adobe have become to large to care about the users needs beyond the ability of those needs to generate profit.  IMO!  But we're getting off-topic.  My fault again 

[Marsu42]

But, wouldn't it be better to automatically have cr2hdr extract the CR2, delete the old DNG, process the CR2 again, and embed the CR2 in the new DNG?

Of course, but I don't want to make the wish-list too large and hinder any implementation chance, so for me personally just chucking the cr2 into the dng as a first step would be enough - I use ACR, so maybe there's just an existing dcraw command line, I didn't research this (yet) since I want to discuss the general idea first.

The best way certainly would be your suggestion - automatically re-process an embedded cr2 and substitute the dng, this would need a cr2hdr version tag somewhere in the dng.

Adobe have become to large to care about the users needs beyond the ability of those needs to generate profit.

I only use LR, but in this case I have to say Adobe is observing user's requests, they really implement features from their forum voted wishlist (see LR4->LR5). And I'm happy they did the dng spec, next to raw in dng it's really good (tif->dng = great compression, cr2->dng = also smaller files, option for better-than-jpeg lossy compression, hdr, in-file tagging w/o sidecar, ...)

[l_d_allan]

Ideally, this would only be needed until cr2hdr progresses to the point where keeping the original CR2s becomes pointless.

I'm wondering if cr2hdr.exe has become "stable" in the sense that newer versions don't change/improve the image quality. It becomes an issue of whether it might be worth reprocessing older .cr2's with the latest/greatest cr2hdr.exe.

Or does this not apply? Is it more or less the equivalent of Adobe coming out with an improved ProcessVersion (like 2003 -> 2010 -> 2012) which can justify converting older image files to the latest/greatest process version?

If the image quality is improving, then my speculation is that the .cr2 is the actual "master". Rather than trying to accomplish embedding the original .cr2 within the .dng, just keep the .cr2 and regenerate the .dng when a latest/greatest cr2hdr.exe is developed that actually makes a difference to image quality.

Something like a speed improvement for cr2hdr.exe wouldn't warrant reprocessing the original .cr2's to make new .dng's.

Or not? Do I have a flawed understanding of what cr2hdr.exe does?

[Marsu42]

I'm wondering if cr2hdr.exe has become "stable" in the sense that newer versions don't change/improve the image quality. It becomes an issue of whether it might be worth reprocessing older .cr2's with the latest/greatest cr2hdr.exe.

Read from here on: http://www.magiclantern.fm/forum/index.php?topic=7139.msg79947#msg79947

However, with cr2hdr being a lossy process, I feel much more comfortable with keeping the original cr2 of good dual_iso shots around, just in case there are going to be further improvements even after the current bugs are ruled out.

The in-place update probably wouldn't be justified alone because there are weekly improved cr2hdr to be expected, but at least for me keeping cr2+dng in one file would be very comfortable - I could then just remove the cr2 from the dng I feel are converted fine or not important enough, but be unburdened from redundant archival/tagging for the rest.

Anyway, it doesn't seem this is likely to happen because unlike the closed source Adobe DNG converter, dcraw doesn't seem to have an easy option to embed cr2 in dng to it'd have to be custom-made.

[Audionut]

I'm wondering if cr2hdr.exe has become "stable" in the sense that newer versions don't change/improve the image quality. It becomes an issue of whether it might be worth reprocessing older .cr2's with the latest/greatest cr2hdr.exe.

That's for the individual to decide.

For me personally, if the converted DNG looks fine I don't see the point in keeping the original CR2's.
An updated cr2hdr might net me slightly better shadows, but I'm not going to process 200 images again for a client who probably (most likely) isn't even going to notice the slight increase in shadow detail anyway.
Where you have images with highlight/shadow artifacts, then it makes much more sense to keep the originals.  For images that I would like to print large in the future, I will keep the original CR2 to use an updated cr2hdr before finally sending the image off to print.
To each their own.

[a1ex]

Just found the exiftool command that does this.

https://bitbucket.org/hudson/magic-lantern/commits/26b7a08d2540b62c72bce3f40ad929e4e14367a5

(back then I tried with -OriginalRawImage, and exiftool said the tag is not writable; the writable one is -OriginalRawFileData)

[Marsu42]

Just found the exiftool command that does this.

Terrific, thanks, I never managed to read through the exiftool doc page :_p ...

1. There is a warning after I try to re-process some random dual_iso file - link to sample: https://bitbucket.org/Marsu42/ml-pull/downloads/DUAL2668.CR2

Code: [Select]

Input file      : DUAL2668.DNG
DUAL2668.CR2    : extracting from DUAL2668.DNG
Warning: Bad compressed RAW image - DUAL2668.DNG

2. Deleting files w/o explicit user request seems like a bad idea, imho it would be advisable to add an option for this like --move-original or an additional --delete-original, whatever you like. Reason: Sometimes I process dual_iso images with different options (soft film curves) right away and then delete the ones I don't like in Lightroom.

[kichetof]

Just found the exiftool command that does this.

King a1ex !
This is a fantastic find!!!

But, as Marsu42 said, don't delete the original file by default or add an arguments to keep the file. I'll have some problem with lightroom if you delete the file (there are no API in LR to delete an image from the catalog, but you can delete the file... but it remain into the catalog without orignal file)

[Walter Schulz]

there are no API in LR to delete an image from the catalog, but you can delete the file... but it remain into the catalog without orignal file)

What about marking as rejected? Is there an API call for "Delete rejected photos"?

Ciao
Walter

[Marsu42]

there are no API in LR to delete an image from the catalog, but you can delete the file... but it remain into the catalog without orignal file

Well, you can always do a "find missing" and then manually remove it, if you didn't know this option.

[a1ex]

1. There is a warning after I try to re-process some random dual_iso file

I get this too; probably it's best to ask the exiftool guys about it.

2. Deleting files w/o explicit user request seems like a bad idea

It is with explicit user request (the help for --embed-original mentions that original will be deleted). Otherwise, you will end up with two originals (one in the CR2 and another in the DNG), so, for me, the expected behavior is to move the original and embed it into the DNG.

Reason: Sometimes I process dual_iso images with different options (soft film curves) right away and then delete the ones I don't like in Lightroom.

Without this option (the classic way), if you want to process the same file with different options, you will need to either make some copies of the input file, or rename the output files.

With this option (CR2 moved into DNG), you have 3 choices:
- make a few copies of the CR2, and process the copies (easiest IMO)
- process the CR2 once, it will be embedded in the DNG, then make a few copies of that DNG and reprocess them with different options. So, the process is pretty similar, and it's mostly a matter of adjusting the scripts.
- process the CR2 once, it will be embedded in the DNG, then extract it from the DNG with exiftool and re-process (this would be the failsafe route, if for some reason you need the original CR2 back).

Another reason for moving the original is that, if you mix dual and non-dual shots, it will be obvious what file you should develop (there will be a single file, the DNG). If the original is kept, you may have trouble with batch processing (you can't just select all and process). With my postprocessing script I've solved it by developing the DNG only, if there are two files with the same name, but I imagine it's not that straightforward with most other workflows.

I'll have some problem with lightroom if you delete the file

In this case, I suggest doing the postprocessing before importing the file in Lightroom (but probably hard to do with your plugin).

I can add an option to --keep-original, but in my opinion, the default behavior should be to delete the original.

[Marsu42]

so, for me, the expected behavior is to move the original and embed it into the DNG.

Yes, you're correct, but I couldn't help but comment that from my programing and IT experience extra-super-high care and explicitness for "deleting" files from apps is advisable.

so, for me, the With this option (CR2 moved into DNG), you have 3 choices: - make a few copies of the CR2, and process the copies (easiest IMO)

Sure, creates some overhead because I copy a file and then delete it, but of course no problem there, fortunately I don't need a gui for cr2hdr :-p

I can add an option to --keep-original, but in my opinion, the default behavior should be to delete the original.

+1 from me

[a1ex]

What about --embed-original-move and --embed-original-copy?

Another thing I didn't think about: if you keep the CR2, and you also embed it in the DNG, you won't be able to reprocess the DNG (since it will try to extract the CR2 and it will fail, because the file is already there).

[kichetof]

In this case, I suggest doing the postprocessing before importing the file in Lightroom (but probably hard to do with your plugin).

I can add an option to --keep-original, but in my opinion, the default behavior should be to delete the original.

I can't. The plugin works only into LR with images in LR.

Well, you can always do a "find missing" and then manually remove it, if you didn't know this option.

Yes I can but I can't remove it from the plugin but, I can set the marker as rejected for the original and add a keyword so no problem

[Marsu42]

What about --embed-original-move and --embed-original-copy?

Yup, sounds very reasonable.

Another thing I didn't think about: if you keep the CR2, and you also embed it in the DNG, you won't be able to reprocess the DNG (since it will try to extract the CR2 and it will fail, because the file is already there).

Well, then check if the cr2 is already there and then don't extract it from the dng :-p as it is bound to be the same as the embedded one? Or extract the cr2 to a temp directory and then process it from there, I already mentioned a temp working directory for all the exiftool stuff would be nice and not to dump the temp files wherever the original is (network drive, cf/sd card, external hd...).

[a1ex]

--embed-original-copy done, test it out!

It now also tries to preserve some metadata if you reprocess the DNG. Also, after embedding the CR2, it will try to read it back and will compare it with the original file before deleting (to make sure the original data is safe).