Author Topic: [RESOLVED] cr2hdr Dual ISO conversion issues  (Read 10556 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
[RESOLVED] cr2hdr Dual ISO conversion issues
« on: November 13, 2015, 07:45:14 AM »
Here's the issue, cr2hdr seems to do fine when converting one dual_iso dng at a time but it sometimes seg faults when doing a batch.

[EDIT: It doesn't have anything to do with batch processing. I had an instance where cr2hdr failed when converting a single dng then succeeded when re-running cr2dng on the same dng.]

Code: [Select]
Horizontal stripe fix...
Segmentation fault: 11

So the failure seemed to be happening on "Full-res reconstruction..."

I've come across this every once in a while but now I have a dual_iso mlv file that is causing this consistently on both my system and Danne's.
(BTW--this came up while testing his new MLPP workflow for OS-X, which is coming along very nicely.)

Ok--this is how to reproduce the issue. Download this mlv file from my dropbox account:

https://www.dropbox.com/sh/mfouwwdch8uauas/AACMTxQwzTUvGwmcgWYGMNGla?dl=0

It is 648 MB so I'd like to delete from there it as soon as enough people have had a chance to try it out. (That shot won't win any awards but it seems to consistently reproduce the issue.)

Next convert it to dng using mlv_dump. I included Mac and Windows versions of mlv_dump and cr2hdr in my bitbucket download area so you don't have to go hunting for them. (Use the testing versions.)

Here's a simple way to do it with mlv_dump:
Code: [Select]
mlv_dump --dng M23-1338.MLV
Finally, batch convert these through cr2hdr you can try:
Code: [Select]
cr2hdr *
Or
Code: [Select]
for i in *.dng; do cr2hdr $i; done
Or whatever method you use to batch convert. You can even import the files into Lightroom, select all of them and run them as a batch using kichetof's cr2hdr plugin. Everything I have tried has caused cr2hdr to seg fault on random frames.

So far this has only been tested on OS-X but it would be interesting to see if it also affects Windows and Linux.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3217
  • 60Da / 1100D / EOSM
Re: cr2hdr batch conversion issues
« Reply #1 on: November 13, 2015, 05:48:04 PM »
Curious, does MLVFS crash on this file if you use it's dual iso processing?

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr batch conversion issues
« Reply #2 on: November 13, 2015, 06:26:40 PM »
No, MLVFS doesn't have any problems converting all the dual_iso frames--the problem is with cr2hdr. I was suggesting using MLVFS to make the dng's without the dual_iso option then processing them with cr2hdr. I'll update the instructions to make that clear.

There is also a problem with cr2hdr not finding the correct camera model. I filed a bug report on that. Again, no problem with MLVFS on this issue.

Looks like cr2hdr could use a little of your dmilligan magic.  :D

[EDIT: I just tried running cr2hdr on dng's copied out of the MLVFS virtual drive and even though it seemed like everything was working as planned, including the cr2hdr seg fault, the processed dng's couldn't be opened with Adobe Camera Raw so I just took out the suggesting of using MLVFS to create the dng's.]
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11472
  • 5D Mark Free
Re: cr2hdr Dual ISO conversion issues
« Reply #3 on: November 14, 2015, 07:16:45 PM »
I had an instance where cr2hdr failed when converting a single dng then succeeded when re-running cr2dng on the same dng.

In this case, you should be able to find the bug with valgrind - simply run "valgrind cr2hdr buggy.dng".

If that doesn't work, can you try to reproduce the issue on a subset of those DNGs? On my current network connection, the traffic is limited, so I prefer to use that half-GB for something else.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #4 on: November 14, 2015, 08:14:58 PM »
Thanks for the tip. I'll try that soon but today I'm actually going to get out and shoot something. Also thanks for the fix on the camera name issue I posted builds of the latest on my bitbucket download area if anyone wants to test your latest changes. Exiftool is up to 10.05 so I included that along with dcraw 9.26.

https://bitbucket.org/daniel_fort/magic-lantern/downloads
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #5 on: November 18, 2015, 02:22:24 AM »
Tried valgrind for the first time. I followed the instructions and compiled with the -g option but I keep getting:

Code: [Select]
valgrind: /usr/local/bin/cr2hdr: cannot execute binary file
Doesn't work with mlv_dump or raw2dng either.

Seems to work on other binaries just fine:

Code: [Select]
valgrind --tool=memcheck dcraw
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #6 on: November 18, 2015, 02:32:31 AM »
If that doesn't work, can you try to reproduce the issue on a subset of those DNGs? On my current network connection, the traffic is limited, so I prefer to use that half-GB for something else.

I ran a loop with a counter to see how many times it needs to go through all 400 frames before all the dng's are processed and I got results from 1 (no errors) to 16 times. There doesn't seem to be one problem file, it is random.

Danne has been aware of this problem, he runs cr2hdr six times in his workflow to insure all files get processed. I just happen to have some files that are more than a half-dozen degrees of stubborn.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

Danne

  • Hero Member
  • *****
  • Posts: 4251
Re: cr2hdr Dual ISO conversion issues
« Reply #7 on: November 18, 2015, 03:36:51 AM »
The extra runnings was more useful in older builds. I think your specific file is the only one that cr2hdr didn,t manage to convert for the last six months I have been using dual iso. Your loop can be useful as a settings alternative in cr2hdr-r and folder_prores(MLPP) workflow.

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3217
  • 60Da / 1100D / EOSM
Re: cr2hdr Dual ISO conversion issues
« Reply #8 on: November 18, 2015, 03:39:05 AM »
I don't really have time to run cr2hdr over and over again to get the error to happen (no errors during the 15 mins it took to write this post), but if you say the error happens right after "Horizontal Stripe Fix" then it must be somewhere in the following code (lines 2711 - 2753):
Code: [Select]
    if (use_stripe_fix)
    {
        printf("Horizontal stripe fix...\n");
        int* delta = malloc(w * sizeof(delta[0]));

        /* adjust dark lines to match the bright ones */
        for (int y = 0; y < h; y ++)
        {
            /* apply a constant offset (estimated from unclipped areas) */
            int delta_num = 0;
            for (int x = raw_info.active_area.x1; x < raw_info.active_area.x2; x ++)
            {
                int b = bright[x + y*w];
                int d = dark[x + y*w];
                if (MAX(b,d) < white_darkened)
                {
                    delta[delta_num++] = b - d;
                }
            }

            /* compute median difference */
            int med_delta = median_int_wirth(delta, delta_num);
           
            if (delta_num < 200)
            {
                //~ printf("%d: too few points (%d)\n", y, delta_num);
                continue;
            }

            if (ABS(med_delta) > 200*16)
            {
                printf("%d: offset too large (%d)\n", y, med_delta);
                continue;
            }

            /* shift the dark lines */
            for (int x = 0; x < w; x ++)
            {
                dark[x + y*w] = COERCE(dark[x + y*w] + med_delta, 0, 0xFFFFF);
            }
        }
        free(delta);
    }

The most likely cause of a seg fault is bad pointer arithmetic -> dereferencing a pointer to an invalid location (outside the bounds of valid data). At first glance the thing that stands out is the second for loop (the inner one). If for some reason raw_info.active_area.x1 and x2 are not valid (either x1 < 0 or x2 > width), then you could potentially go out of bounds in those following array lookups. I would suggest validating them and see if that is in fact the problem.

Put this at the beginning of that code block:
Code: [Select]
    if(raw_info.active_area.x1 < 0 || raw_info.active_area.x2 > w)
    {
        printf("Invalid active area!\n");
        goto err;
    }

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #9 on: November 18, 2015, 05:29:15 PM »
Put this at the beginning of that code block:

I tried that but keep getting the same error.

Code: [Select]
Hot pixels      : 121
AMaZE interpolation ...
Amaze took 0.09 s
Edge-directed interpolation...
Semi-overexposed: 95.90%
Deep shadows    : 0.00%
Horizontal stripe fix...
Segmentation fault: 11

I believe I followed your instructions:

Code: [Select]
        /* adjust dark lines to match the bright ones */
        for (int y = 0; y < h; y ++)
        {
            /* apply a constant offset (estimated from unclipped areas) */
            int delta_num = 0;
            if(raw_info.active_area.x1 < 0 || raw_info.active_area.x2 > w)
            {
                printf("Invalid active area!\n");
                goto err;
            }
            for (int x = raw_info.active_area.x1; x < raw_info.active_area.x2; x ++)
            {
                int b = bright[x + y*w];
                int d = dark[x + y*w];
                if (MAX(b,d) < white_darkened)
                {
                    delta[delta_num++] = b - d;
                }
            }

Of course you are right about the problem being in the "Horizontal Stripe Fix" code. Running "cr2hdr --no-stripe-fix *.dng" will convert all the files without errors.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3217
  • 60Da / 1100D / EOSM
Re: cr2hdr Dual ISO conversion issues
« Reply #10 on: November 18, 2015, 11:02:45 PM »
Ok, then it's something else. One thing you can do if you can't get valgrind or some other debugger working, is to just put in a bunch of different print statements (I would suggest one before each statement that has a square bracket, as its most likely one of those), then see which one was the last to execute the before the bug, that should narrow it down to the exact line.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #11 on: November 19, 2015, 07:30:53 AM »
Good idea. Here's what I did:

Code: [Select]
if (use_stripe_fix)
    {
        printf("start of problem. use_stripe_fix\n");
        printf("Horizontal stripe fix...\n");
        printf("int* delta = malloc(w * sizeof(delta[0]));\n");
        int* delta = malloc(w * sizeof(delta[0]));
        /* adjust dark lines to match the bright ones */
        printf("start of loop - for (int y = 0; y < h; y ++)\n");
        for (int y = 0; y < h; y ++)
        {
            /* apply a constant offset (estimated from unclipped areas) */
            printf("int delta_num = 0;\n");
            int delta_num = 0;
            printf("for (int x = raw_info.active_area.x1; x < raw_info.active_area.x2; x ++)\n");
            for (int x = raw_info.active_area.x1; x < raw_info.active_area.x2; x ++)
            {
            printf("int b = bright[x + y*w];\n");
                int b = bright[x + y*w];
            printf("int d = dark[x + y*w];\n");
                int d = dark[x + y*w];
                printf("if (MAX(b,d) < white_darkened)\n");
                if (MAX(b,d) < white_darkened)
                {
                printf("delta[delta_num++] = b - d;\n");
                    delta[delta_num++] = b - d;
                }
            }

The error came up after several iterations of the inner loop. I ran it a few times to make sure it always seg faults at the same spot.

Code: [Select]
if (MAX(b,d) < white_darkened)
int b = bright[x + y*w];
int d = dark[x + y*w];
if (MAX(b,d) < white_darkened)
Segmentation fault: 11

So I figured that the next step would be to look at the values in that one line so I did this:

Code: [Select]
            {
                int b = bright[x + y*w];
                int d = dark[x + y*w];
                printf("b = %d\n", b);
                printf("d = %d\n", d);
                printf("white_darkened = %d\n", white_darkened);
                printf("if (MAX(b,d) < white_darkened)\n");
                if (MAX(b,d) < white_darkened)
                {
                    delta[delta_num++] = b - d;
                }

and got these results:

Code: [Select]
b = 187037
d = 1047786
white_darkened = 181185
if (MAX(b,d) < white_darkened)
b = 187037
d = 1035178
white_darkened = 181185
if (MAX(b,d) < white_darkened)
b = 187037
d = 1047786
white_darkened = 181185
if (MAX(b,d) < white_darkened)
Segmentation fault: 11

It doesn't make any sense to me because it looks like the same values worked several times before the segmentation fault occurred.

[EDIT: When I went to close the terminal I got a message that processes were still running. It created a crash log, don't know if it helps but I also got a message that my system was running low on memory and my browser closed down. Never had that happen before.]

Code: [Select]
Process:               cr2hdr [7312]
Path:                  /Users/USER/Desktop/*/cr2hdr
Identifier:            cr2hdr
Version:               0
Code Type:             X86 (Native)
Parent Process:        bash [3294]
Responsible:           Terminal [3121]
User ID:               501

Date/Time:             2015-11-18 22:02:15.572 -0800
OS Version:            Mac OS X 10.11.1 (15B42)
Report Version:        11
Anonymous UUID:        DF30BA7C-435C-699C-6073-326B108E5CFE

Sleep/Wake UUID:       B25CA8CF-4DDF-46C4-B25E-EE09A0947FFE

Time Awake Since Boot: 16000 seconds
Time Since Wake:       8300 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x000000007cfffffc

VM Regions Near 0x7cfffffc:
    MALLOC_TINY            000000007c800000-000000007cb00000 [ 3072K] rw-/rwx SM=PRV 
-->
    MALLOC_SMALL           000000007d000000-000000007f800000 [ 40.0M] rw-/rwx SM=PRV 

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   cr2hdr                        0x000b377c main + 29964
1   libdyld.dylib                  0x945816ad start + 1

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x7cfffffc  ebx: 0x0002da9d  ecx: 0xa315c000  edx: 0xffffffff
  edi: 0x7cfffffc  esi: 0x000ffcea  ebp: 0xbff5ced8  esp: 0xbff5baf0
   ss: 0x00000023  efl: 0x00010297  eip: 0x000b377c   cs: 0x0000001b
   ds: 0x00000023   es: 0x00000023   fs: 0x00000000   gs: 0x0000000f
  cr2: 0x7cfffffc
 
Logical CPU:     0
Error Code:      0x00000004
Trap Number:     14


Binary Images:
   0xa1000 -    0xbcffb +cr2hdr (0) <8082ACA9-43F9-308E-810E-2865A73A6425> /Users/USER/Desktop/*/cr2hdr
 0x1ae6000 -  0x1afffeb +libgcc_s.1.dylib (0) <F8C24D77-9744-3228-A808-7896D333395C> /usr/local/lib/gcc/5/libgcc_s.1.dylib
0x8fe9f000 - 0x8fed35bf  dyld (360.17) <E08B9CA8-D557-32D5-8DDD-E0B563900367> /usr/lib/dyld
0x905b8000 - 0x905c3ff3  libcommonCrypto.dylib (60074) <448055DC-6EBC-3C23-ADBC-4C86ABC49A87> /usr/lib/system/libcommonCrypto.dylib
0x90fb8000 - 0x90fcdffb  libsystem_coretls.dylib (82) <39CA721D-1FC1-3482-85ED-0FA23938C8D2> /usr/lib/system/libsystem_coretls.dylib
0x90fce000 - 0x90fe7fff  libsystem_malloc.dylib (67) <01C0F706-2B84-3F46-935D-7C4A6FC557E3> /usr/lib/system/libsystem_malloc.dylib
0x91e24000 - 0x91e24fff  libkeymgr.dylib (28) <37F88FF9-BB2F-3703-9283-5CC7BF7D84EA> /usr/lib/system/libkeymgr.dylib
0x91e62000 - 0x91e62fff  libsystem_blocks.dylib (65) <044CF869-766A-3B8C-853B-D840F7E768D1> /usr/lib/system/libsystem_blocks.dylib
0x92fc7000 - 0x92fcdff3  libsystem_platform.dylib (73.1.1) <8CB8B06A-D7CC-392B-8B97-20C53F36D39C> /usr/lib/system/libsystem_platform.dylib
0x93389000 - 0x9338cff7  libsystem_sandbox.dylib (459.10.4) <F1EE9704-9448-3CC6-B124-D129EB0681BF> /usr/lib/system/libsystem_sandbox.dylib
0x93453000 - 0x93457fff  libcache.dylib (75) <B848C84B-4D15-3B5C-A716-960BE2849ACC> /usr/lib/system/libcache.dylib
0x934ef000 - 0x93556fff  libcorecrypto.dylib (334) <855C8E99-B0B9-35E4-B8AE-68972AC1CCA1> /usr/lib/system/libcorecrypto.dylib
0x9454a000 - 0x9457dfe3  libsystem_m.dylib (3105) <AB252034-37BA-3B50-93F3-480B459D7E49> /usr/lib/system/libsystem_m.dylib
0x9457e000 - 0x94581ff7  libdyld.dylib (360.17) <FCD2E1E6-9BA5-3ED8-BF07-48211F2F6E52> /usr/lib/system/libdyld.dylib
0x9461e000 - 0x94623ff7  libmacho.dylib (875.1) <20C4FF2D-035C-3CDA-8DA2-F37EFCFBD3AF> /usr/lib/system/libmacho.dylib
0x94a11000 - 0x94a19ffb  libsystem_dnssd.dylib (624.10.1) <193C4DB6-2D7F-3010-B291-145CD18A0401> /usr/lib/system/libsystem_dnssd.dylib
0x95eb9000 - 0x95ec3fff  libsystem_notify.dylib (149) <26CF3479-D54B-319E-BDB5-74C6669C666C> /usr/lib/system/libsystem_notify.dylib
0x968bc000 - 0x968c5fff  libsystem_networkextension.dylib (384.1.2) <6575E989-C59A-377C-B3FB-BDD9DD53CA28> /usr/lib/system/libsystem_networkextension.dylib
0x9857f000 - 0x98581fff  libsystem_coreservices.dylib (19) <DACCE5EF-A15F-3B94-8CE7-F1CC60593B6E> /usr/lib/system/libsystem_coreservices.dylib
0x9a153000 - 0x9a155ffb  libsystem_secinit.dylib (20) <30529766-9A22-34B8-A521-5AD766833172> /usr/lib/system/libsystem_secinit.dylib
0x9a1a8000 - 0x9a1b0ffb  libsystem_pthread.dylib (137.1.1) <C34AB3EC-25DB-325C-BA96-123772887436> /usr/lib/system/libsystem_pthread.dylib
0x9a2b2000 - 0x9a2b4fff  libquarantine.dylib (80) <9FA02799-B68E-315B-9102-9065C259B3DF> /usr/lib/system/libquarantine.dylib
0x9a46e000 - 0x9a4d4ff3  libsystem_network.dylib (582.1.4) <068135E4-F7F2-3356-931D-69DCB1379DDC> /usr/lib/system/libsystem_network.dylib
0x9a68b000 - 0x9a68bfff  libunc.dylib (29) <6C251ECD-0449-384F-A5E3-D277C8DD9E13> /usr/lib/system/libunc.dylib
0x9acd2000 - 0x9acd2fff  liblaunch.dylib (755.1.19) <25AA00BF-EF6C-3304-9A5D-BA37569F5FB4> /usr/lib/system/liblaunch.dylib
0x9af14000 - 0x9af3efff  libdispatch.dylib (500.10.1) <5E474014-7270-37D8-9B5A-8D992C08CD5E> /usr/lib/system/libdispatch.dylib
0x9affc000 - 0x9affefff  libsystem_configuration.dylib (801.10.2) <48A1AA26-F29C-34F5-B744-FE6C68145DA5> /usr/lib/system/libsystem_configuration.dylib
0x9afff000 - 0x9b018fff  libsystem_asl.dylib (322) <D791C719-A838-3A13-922D-EC0149D2F11B> /usr/lib/system/libsystem_asl.dylib
0x9c022000 - 0x9c028fff  libunwind.dylib (35.3) <7639935C-D618-343C-AC20-90A8C4665971> /usr/lib/system/libunwind.dylib
0x9c029000 - 0x9c02afff  libSystem.B.dylib (1225.1.1) <92CA3288-0AF8-3926-B60A-1A61894AFDF6> /usr/lib/libSystem.B.dylib
0x9c325000 - 0x9c34dffb  libxpc.dylib (755.1.19) <E6BABCAF-E6DC-3475-BC47-8A79D54E0746> /usr/lib/system/libxpc.dylib
0x9cb4f000 - 0x9cb78ff7  libsystem_info.dylib (476) <CFEE2535-E53C-39FA-8D7F-4DC036FA5B50> /usr/lib/system/libsystem_info.dylib
0x9d674000 - 0x9d679ffb  libcompiler_rt.dylib (62) <424790B7-8170-379C-97FD-9ABF614C861A> /usr/lib/system/libcompiler_rt.dylib
0x9d687000 - 0x9d6a7fff  libsystem_kernel.dylib (3247.10.11) <5AF48D1A-FD7D-382A-A147-AE4B938384FC> /usr/lib/system/libsystem_kernel.dylib
0x9d99e000 - 0x9d9a7fff  libcopyfile.dylib (127) <C1D5D894-E71C-3E75-9639-82C846482A84> /usr/lib/system/libcopyfile.dylib
0x9d9a8000 - 0x9d9b9fff  libsystem_trace.dylib (200) <FEC2FA0C-7B2B-3735-87AB-D97A5F4D56FA> /usr/lib/system/libsystem_trace.dylib
0x9d9ec000 - 0x9da81fff  libsystem_c.dylib (1081.1.3) <CB2DB808-E449-3696-AE97-CC7F442063F2> /usr/lib/system/libsystem_c.dylib
0x9e7fa000 - 0x9e7fbfff  libremovefile.dylib (41) <59D0F532-1D12-3CB0-841A-E79E85616D70> /usr/lib/system/libremovefile.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 8120
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=52.9M resident=0K(0%) swapped_out_or_unallocated=52.9M(100%)
Writable regions: Total=145.0M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=145.0M(100%)
 
                                VIRTUAL   REGION
REGION TYPE                        SIZE    COUNT (non-coalesced)
===========                     =======  =======
Activity Tracing                  2048K        2
Kernel Alloc Once                    4K        2
MALLOC                           105.2M       16
MALLOC guard page                   16K        4
MALLOC_LARGE (reserved)           3600K        2         reserved VM address space (unallocated)
Stack                             8192K        2
Stack Guard                       56.0M        2
VM_ALLOCATE                          4K        2
__DATA                            26.6M       39
__LINKEDIT                        49.4M        5
__OBJC                               4K        2
__TEXT                            3548K       39
mapped file                      262.3M       67
shared memory                        8K        3
===========                     =======  =======
TOTAL                            516.5M      173
TOTAL, minus reserved VM space   513.0M      173
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3217
  • 60Da / 1100D / EOSM
Re: cr2hdr Dual ISO conversion issues
« Reply #12 on: November 19, 2015, 12:55:43 PM »
from what you say, the seg fault must be here:
Code: [Select]
int med_delta = median_int_wirth(delta, delta_num);

median_int_wirth is a macro for computing the median (it's done like this because it is faster that calling a function), it's defined in a separate header file, there must be something wrong with it, it will require further investigation.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #13 on: November 21, 2015, 02:00:00 AM »
Just wondering--I'm not really sure what horizontal stripe fix does but I haven't noticed any difference turning it off. In addition, it isn't one of the options in MLVFS. Maybe change the cr2hdr default to --no-stripe-fix? That only affects the horizontal stripe fix. AFAIK vertical strip fix is always on. Not running horizontal stripe fix also speeds it up considerably.

Of course the seg fault issue should be resolved but for now the default options should be stable.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11472
  • 5D Mark Free
Re: cr2hdr Dual ISO conversion issues
« Reply #14 on: November 23, 2015, 11:31:00 AM »
Sounds like a buffer overflow - it probably tries to fill the "delta" array past its end.

However, this sounds unlikely - this would happen if raw_info.active_area.x2 - raw_info.active_area.x1 would be greater than raw_info.width.

Can you print these 3 values before the crash?

What this step does: in some images, the two exposures cannot be matched accurately, and this basically fine-tunes the matching on a line-by-line basis. It certainly can be improved, as there is a theoretical chance for this to actually introduce horizontal stripe artifacts.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #15 on: November 23, 2015, 08:42:51 PM »
However, this sounds unlikely - this would happen if raw_info.active_area.x2 - raw_info.active_area.x1 would be greater than raw_info.width.

Can you print these 3 values before the crash?

Here you go:

Code: [Select]
raw_info.width = 1280
raw_info.active_area.x1 = 0
raw_info.active_area.x2 = 1280
raw_info.width = 1280
raw_info.active_area.x1 = 0
raw_info.active_area.x2 = 1280
raw_info.width = 1280
raw_info.active_area.x1 = 0
raw_info.active_area.x2 = 1280
Segmentation fault: 11

Those 3 values never changed.

This is where I put the printf statements:

Code: [Select]
            for (int x = raw_info.active_area.x1; x < raw_info.active_area.x2; x ++)
            {
                int b = bright[x + y*w];
                int d = dark[x + y*w];
               
                printf("raw_info.width = %d\n", raw_info.width);               
                printf("raw_info.active_area.x1 = %d\n", raw_info.active_area.x1);
                printf("raw_info.active_area.x2 = %d\n", raw_info.active_area.x2);
               
                if (MAX(b,d) < white_darkened)
                {
                    delta[delta_num++] = b - d;
                }
            }
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11472
  • 5D Mark Free
Re: cr2hdr Dual ISO conversion issues
« Reply #16 on: November 23, 2015, 08:46:22 PM »
Can you also print delta_num right before incrementing?

BTW, this looks like the answer to your valgrind issue: http://stackoverflow.com/questions/6385688/valgrind-error-cannot-execute-binary-file

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #17 on: November 23, 2015, 09:47:59 PM »
Interesting, I'm using the Homebrew version of valgrind so maybe that is the problem.

In any case, I'm doing this now:

Code: [Select]
                if (MAX(b,d) < white_darkened)
                {

                    printf("delta_num = %d\n", delta_num);               
               
                    delta[delta_num++] = b - d;
                }

Note that I followed your instructions and am printing delta_num just before it increments. The seg fault seems to happen just before the previous line. Now I seem to be in a "watched pot never boils" situation. cr2hdr keeps processing all 400 frames without a single seg fault! delta_num varies from 0 to 1279. I'll keep it going but so far it has run through all the frames 3x and that has never happened before.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #18 on: November 23, 2015, 11:15:49 PM »
Finally got one:

Code: [Select]
Input file      : M23-1338_frame_000284.dng
Camera          : Canon EOS M
Full size       : 1280 x 720
Active area     : 1280 x 720
Black borders   : N/A
ISO pattern     : BddB RGGB
White levels    : 10000 14882
Noise levels    : 8.00 8.00 8.00 8.00 (14-bit)
ISO difference  : 4.08 EV (1687)
Black delta     : 19.00
Dynamic range   : 9.96 (+) 10.65 => 14.72 EV (in theory)
Looking for hot/cold pixels...
Hot pixels      : 856
AMaZE interpolation ...
Amaze took 0.08 s
Edge-directed interpolation...
Semi-overexposed: 62.75%
Deep shadows    : 21.48%
Horizontal stripe fix...
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 0
delta_num = 1
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
delta_num = 7
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
delta_num = 7
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
delta_num = 7
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
delta_num = 0
delta_num = 0
delta_num = 1
delta_num = 2
delta_num = 3
delta_num = 4
delta_num = 5
delta_num = 6
Segmentation fault: 11

It didn't go through too many loops before the seg fault so I included all the information on that frame. Remember it isn't any one frame in particular that causes the seg fault, it happens on random frames. I don't see anything unusual in this but I'm not really sure what to look for.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11472
  • 5D Mark Free
Re: cr2hdr Dual ISO conversion issues
« Reply #19 on: November 24, 2015, 08:06:07 AM »
If you repeat only this file in a loop, can you reproduce the issue?

If yes, can you upload it?

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
Re: cr2hdr Dual ISO conversion issues
« Reply #20 on: November 24, 2015, 09:56:00 AM »
If you repeat only this file in a loop, can you reproduce the issue?

Just the one dng? I haven't tried that. The original is an mlv file shot with an EOSM. I used mlv_dump to make the dng files and it almost always reproduces the issue. Today was an exception. I had to run it three times (400 frames) before it happened but then when I ran it again it seg faulted on the very first frame.

The steps I took were simply:

Code: [Select]
mlv_dump --dng M23-1338.MLV
cr2hdr *.dng

Quote
If yes, can you upload it?

Link to the mlv was in the first post:
https://www.dropbox.com/sh/mfouwwdch8uauas/AACMTxQwzTUvGwmcgWYGMNGla?dl=0

I also uploaded a few dng's (including frame 284 that triggered the seg fault today) in that dropbox folder. I'll try setting up just that one frame in a loop tomorrow and see what happens.

I'm using OS-X and I haven't checked it on Linux or Windows so I'm not really sure if this issue only affects Mac or all platforms. I can try it next weekend when I have access to my Linux/Windows dual-boot system.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11472
  • 5D Mark Free
Re: cr2hdr Dual ISO conversion issues
« Reply #21 on: November 24, 2015, 10:22:16 AM »
Running this DNG under valgrind triggered the error on my first test run. However, after compiling with -g, the error was no longer there. Go figure.

Anyway, the error was:

Code: [Select]
Horizontal stripe fix...
==6277== Invalid read of size 4
==6277==    at 0x804F9FD: hdr_interpolate (in /usr/local/bin/cr2hdr)
==6277==    by 0x8055335: main (in /usr/local/bin/cr2hdr)
==6277==  Address 0x713f8d4 is 4 bytes before a block of size 5,120 alloc'd
==6277==    at 0x4027795: malloc (vg_replace_malloc.c:291)
==6277==    by 0x8049CAD: malloc_or_die (in /usr/local/bin/cr2hdr)
==6277==    by 0x804F7CA: hdr_interpolate (in /usr/local/bin/cr2hdr)
==6277==    by 0x8055335: main (in /usr/local/bin/cr2hdr)
==6277==
Full-res reconstruction...

so, it looks more like a buffer underflow.

edit: I think I've diagnosed it; dmilligan was almost right. Thanks for narrowing down.

edit2: fix commited. There might be similar crashes in different places, so I'm going to run my test suite (about 2 GB of CR2's from bug reports).

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3027
[RESOLVED] cr2hdr Dual ISO conversion issues
« Reply #22 on: November 24, 2015, 03:29:36 PM »
I've run my test file several times with your latest commits and so far no seg fault issues. Thanks!
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

Danne

  • Hero Member
  • *****
  • Posts: 4251
Re: [RESOLVED] cr2hdr Dual ISO conversion issues
« Reply #23 on: November 24, 2015, 08:21:28 PM »
Great fix here. THanks a lot