DNG file sizes are BIG, why?

[Dante]

Hi guys,

I'm running Mac and have tried both the standalone app CR2HDR.app (2.0 beta) and the LR plugin to convert dual ISO CR2 files to DNG. (5DMKIII V.123)
(dcraw 9.19 installed).

With the app it's just drag and drop and out comes a DNG, with LR I have tried changing the plugin options.
My problem is that my DNG files comes out huge. I run a (normal sized 27Mb) CR2 and the DNG converted is 44Mb.
In LR (CR2HDR-20 bit version)the option to embed original CR2 is disabled and everything else is also set to default. The .app has no options but the result is the same.

I have dl a couple of examples from others (both CR2 and DNG) and their converted DNGs are always smaller than the CR2
If i try to convert their CR2 my DNG is double the size.

What am I missing?

[dmilligan]

CR2 is compressed (lossless), DNG is not.

[Dante]

It doesn't matter if i set DNG-compression in the LR-plugin to no-compress, -compress - compress-lossy.

The resulting DNGs all have exact the same size. 44Mb

If I just do an normal LR export to DNG (of course it won't cook the dual ISO) the file size of the resulting DNG is just 22Mb.

So I still don't understand...

[dmilligan]

cr2hdr uses the adobe dng converter to compress DNGs. You probably don't have it installed (or where cr2dng is expecting it to be).

Code Select Expand


static const char* search_path[] = {
    "./",                                                       /* working dir */
    ".\\",                                                      /* working dir (Windows) */
    "/Applications/Adobe DNG Converter.app/Contents/MacOS/",    /* Mac */
    "C:\\Program Files\\Adobe\\",                               /* Windows */
    "C:\\Program Files (x86)\\Adobe\\",                         /* Some 64-bit Windows */
};

static const char* find_adobe_dng_converter()
{
    static char adobe_dng_path[1000] = "";
   
    /* valid path from previous attempt? just return it */
    if (adobe_dng_path[0])
        return adobe_dng_path;
   
    /* let's try some usual paths */
    int i;
    for (i = 0; i < COUNT(search_path); i++)
    {
        int is_win_path = (search_path[i][strlen(search_path[i])-1] == '\\');
        if (is_win_path)
        {
#if defined(WIN32) || defined(_WIN32)
            snprintf(adobe_dng_path, sizeof(adobe_dng_path), "%s%s", search_path[i], "Adobe DNG Converter.exe");
            if (can_exec(adobe_dng_path))
                return adobe_dng_path;
#else
            /* let's try Wine */
            snprintf(adobe_dng_path, sizeof(adobe_dng_path), "winepath \"%s%s\"", search_path[i], "Adobe DNG Converter.exe");
            FILE* f = popen(adobe_dng_path, "r");
            if (f)
            {
                int ok = fgets(adobe_dng_path, sizeof(adobe_dng_path), f) != 0;
                pclose(f);
               
                if (ok)
                {
                    strtok(adobe_dng_path, "\n");
                    if (can_exec(adobe_dng_path))
                        return adobe_dng_path;
                }
            }
#endif
        }
        else
        {
#if !defined(WIN32) && !defined(_WIN32)
            snprintf(adobe_dng_path, sizeof(adobe_dng_path), "%s%s", search_path[i], "Adobe DNG Converter");
            if (can_exec(adobe_dng_path))
                return adobe_dng_path;
#endif
        }
    }
   
    /* nothing found */
    adobe_dng_path[0] = 0;
    return 0;
}


[Dante]

Thanks for helping, but I'm stuck.

/Applications/Adobe DNG Converter.app
Is where my Adobe Converter sits.

I get no error msg from LR or the standalone CR2HDR app that something is missing. So I can't really tell where they are looking for the Adobe DNG converter. (If they don't find the converter, how can they still be making DNG files from the CR2?)

The LR workflow should be this right?

1. Export to CR2HDR - Get 22Mb something DNG file (depending on the options in CR2HDR plugin)
2. Do adjustments in LR to the new DNG
3. Update DNG preview and metadata (File is still 22Mb)


But this is what's happening:

1. Export to CR2HDR - Get 44Mb!!! something DNG file (regardless of settings in CR2HDR plugin)
2. Do adjustments in LR to the new DNG
3. Update DNG preview and metadata File is now 22Mb!!!!  (I did nothing to it but update meta and thumb after adjustments)

Or is this how it should behave? 
If so, then I don't know what the settings for CR2HDR do since they don't do much with the output..

[budafilms]

It's normal.

[Dante]

What is normal?

Settings in LR-plugin doing noting and file sizes being 44Mb+ and being reduced to half, just because I write new meta to them doesn't seem very normal..

[Walter Schulz]

CR2:
Lossless compression. -> Files originating from the same camera may have different file sizes.

DNG created by CR2HDR:
Uncompressed. -> All files have same sizes. See upper part of pic below.


DNG exported by LR:
Lossless compression. -> Different file sizes again. See lower part of pic below.

[Dante]

Thanks Walter, (no pictures included?!).

So I guess the scenario I'm seeing is "Normal" then..
That files are being compressed lossless, when I write the adjustment/meta to the DNG with LR.
Even though I'm not doing any "DNG-export", just writing meta to the file?
At least that is my understanding what "Update DNG preview and metadata" command in LR will do (=update meta) or is this equal to "export"?

Also what I still don't understand is why there are settings in the C2RHDR-plugin (regarding compression) that have no effect on the outcome?

[Walter Schulz]

Thanks Walter, (no pictures included?!).

Sorry?

That files are being compressed lossless, when I write the adjustment/meta to the DNG with LR.
Even though I'm not doing any "DNG-export", just writing meta to the file?
At least that is my understanding what "Update DNG preview and metadata" command in LR will do (=update meta) or is this equal to "export"?

Looks like if LR is changing file contents it will use lossless compression.

Also what I still don't understand is why there are settings in the C2RHDR-plugin (regarding compression) that have no effect on the outcome?

Will test this one.

Ciao
Walter

[Dante]

Yes it seems through testing that LR is in fact re-saving/exporting the DNG again when "Update DNG preview and metadata, with the lossless option.
Although I have found no documentation from Adobe saying that's whats happening and there is nowhere to change the compression option for the output.
Only thing that can be changed is the size of the preview thumb, in "preferences/File Handling". This setting will also effect the "Update DNG preview and metadata".

What is left in the dark is why the settings made in CR2HDR-Plugin change nothing in the output...

(Also noted that the plugin itself can not be placed in a folder on the Mac that contain any "-" (in my case "/work-adobe/LR/").
The plugin was able to load in LR and I can see it in Plugin Manager but it doesn't show up in right click context menu nor under "Presets" in the "File/Export menu" if the plugin is located in a folder with "-". Just put it in another folder and it will work fine. Hope it will help someone else with the same problem.

Please let me know when you had time to test the settings-not-doing-anything-problem.

Thanks!

[dmilligan]

This is an issue with the LR plugin:
http://www.magiclantern.fm/forum/index.php?topic=11056.msg119792#msg119792

It appears the author has not had time to fix the issue yet:

Unfortunately I don't have too much time at the moment.

[DFM]

Yes it seems through testing that LR is in fact re-saving/exporting the DNG again when "Update DNG preview and metadata, with the lossless option.
Although I have found no documentation from Adobe saying that's whats happening and there is nowhere to change the compression option for the output.

Lightroom is non-destructive. You don't ever "re-save" a file, you export a new copy. Of course you can choose to write that over the original, but it's still a different concept to the way something like Photoshop does it.

Lightroom's default for exporting DNGs is lossless compression. You can choose DNG lossy compression via the Export dialog. The compression ratio is roughly equivalent to the 100% option for JPEGs. You cannot change the compression ratio in Lightoom or DNG Converter.

Adobe DNG Converter can save to all four DNG formats, depending on the commandline parameter you set:

-c = compressed (lossless)
-u = uncompressed
-lossy = lossy compression
-l = linear DNGs (stores debayered pixels, usually RGB channels).

when you use -lossy, you can also specify either:

-side PPP = maximum pixel width or height on the long edge
-count PPP = desired kilopixel count  (1024 = 1 megapixel, etc.)

e.g.  -lossy -count 4096 will save a lossy DNG with 4 megapixels resolution at the original aspect ratio.

[Walter Schulz]

Lightroom is non-destructive. You don't ever "re-save" a file, you export a new copy. Of course you can choose to write that over the original, but it's still a different concept to the way something like Photoshop does it.

You are missing the point. Please follow these steps:
1. Download my testfile. http://filehorst.de/d/bphrlvAm
    It is a DNG created by CR2HDR.EXE. Extracted Filesize is 37,994,110 Bytes
2. Import this DNG into LR. My LR version is 5.6
3. Add a Keyword Tag. Mine was "Alpen".
4. Open context menu -> Metadata -> Update DNG preview and metadata
    File size now is 21,738,104

Please verify or falsify this one.

Ciao
Walter

[DFM]

Please verify or falsify this one.

I followed your steps:

• Import DNG to LR5.6 using 'Add'
• Add another keyword
• Photo > Update DNG Preview & Metadata

No reduction in file size on disc. Screenshots at http://i.imgur.com/DgYF07F.gif

[Walter Schulz]

Thanks!
Is there a method to compare our LR settings?

[DFM]

Thanks!
Is there a method to compare our LR settings?

In theory it shouldn't matter; nothing in the application or cat prefs changes the code used to update the IPTC/EXIF block. I'll run some more tests and try to make it break.

[Walter Schulz]

Just installed "Windows 7 Enterprise 32-bit, german" on a virtual machine (VMware Player). Added all essential MS patches. Downloaded LR 5.6 installation file from Adobe's site and installed it (german again).
*No* settings changed. Run import to copy my test file to C:\Bilder. Added a keyword, opened context menu and selected "Metadaten" (= Metadata in en) and "DNG-Vorschau und Metadaten aktualisieren" (= "Update DNG preview and metadata" in en).
Filesize after that: 21,736,646 Bytes. Shrunk again.
I can upload the VM if needed.

Ciao
Walter

[philmoz]

Just installed "Windows 7 Enterprise 32-bit, german" on a virtual machine (VMware Player). Added all essential MS patches. Downloaded LR 5.6 installation file from Adobe's site and installed it (german again).
*No* settings changed. Run import to copy my test file to C:\Bilder. Added a keyword, opened context menu and selected "Metadaten" (= Metadata in en) and "DNG-Vorschau und Metadaten aktualisieren" (= "Update DNG preview and metadata" in en).
Filesize after that: 21,736,646 Bytes. Shrunk again.
I can upload the VM if needed.

Ciao
Walter

That's what I see on OS/X - and I would expect it to be the same on all platforms.

LR saves DNG files with lossless compression, unless you enabled the lossy compression setting (Preferences --> File Handling).

Your original DNG file is not compressed, so when LR updates the preview and re-saves the DNG file it compresses the main image data.

Phil.

[Walter Schulz]

Thanks for your reply. Was just about asking a Mac user to test this. ;-)
Now I'm wondering why this doesn't happen to DFM.

[walter_schulz]

@DFM: Any new info about this item?
There are Windows and OS X users to back my claim about LR 5.6 changing DNGs into lossless compressed if metadata are added.