Silent picture capture with raw video format

Started by dmilligan, July 20, 2014, 01:48:25 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

barepixels

Quote from: josepvm on July 25, 2014, 11:22:55 PM
Maybe, but only with ETTR on ?

I don't know  I have never shot Full-res silent MLV this many frames.  we were doing 5 or 10 frames to solve the mlv_dump problem.

I just got done with 102 shots.  the single MLV fail to extract.  gave me only one file name 101 and that cant be open with ARC.

Going for my fourth try now.... but this time I will use DNG and ETTR  not MLV    for around 100+ frames.  My theory is if a frame get corrupted, with DNG the rest can be save but a similar corrupted frame may kill the whole MLV and its chunks.
5D2 + nightly ML

barepixels

5D2 + nightly ML

josepvm

Got it again. With 101 pictures, I get the error. Intervalometer set to 1 sec.  ETTR "Always on", saving as MLV. Only one MLV file written, 2.2 GB size

Output of "mlv_dump -v --dng *.MLV" :


MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '00000000.MLV'
   - Verbose messages
   - Convert to DNG frames
   - Output into '00000000_frame_'
File 00000000.MLV opened
File 00000000.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 3507273469957387890
    FPS         : 1.000000
    File        : 0 / 0
    Frames Video: 101
    Frames Audio: 0
Block: RTCI
  Offset: 0x00000034
    Size: 44
    Time: 838839.378000 ms
     Date:        25.07.2014
     Time:        23:50:47 (GMT+0)
     Zone:        ''
     Day of week: 5
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000060
    Size: 40
    Time: 838839.942000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    0 µs (1/inf)
Block: LENS
  Offset: 0x00000088
    Size: 96
    Time: 838839.969000 ms
     Name:        '1-65535mm'
     Serial:      ''
     Focal Len:   300 mm
     Focus Dist:  0 mm
     Aperture:    f/4.00
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 838839.984000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
Block: WBAL
  Offset: 0x0000013c
    Size: 44
    Time: 838840.034000 ms
     Mode:   0
     Kelvin:   5000
     Gain R:   1024
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x00000168
    Size: 52
    Time: 838840.046000 ms
     picStyle:   133
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: VIDF
  Offset: 0x0000019c
    Size: 27067688
    Time: 841076.753000 ms
   Frame: #100
    Crop: 0x0
     Pan: 0x0
   Space: 0

Vertical stripes correction:
  1.000  1.000    1      1      1      1      1      1 
[ERROR] Invalid block size at position 0x019d06c4
Processed 1 video frames
Done




Exactly the same setup and lighting but only 20 frames (~500 MB MLV file) no errors.



barepixels

now try again, same amount, but this time using DNGs.  See if you will get any corrupted frame.  i sent you a PM
5D2 + nightly ML

g3gg0

there is no RAWI in this MLV.

can you upload the first few hundred MiB of a buggy file?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

josepvm

My last run for today ( after this one I go to sleep )  ::)

Same setup, same lighting, same number of pictures (101), but this time ETTR disabled.

And the error appears again:


MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '00000000.MLV'
   - Verbose messages
   - Convert to DNG frames
   - Output into '00000000_frame_'
File 00000000.MLV opened
File 00000000.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 1572878673380009968
    FPS         : 1.000000
    File        : 0 / 0
    Frames Video: 101
    Frames Audio: 0
Block: RTCI
  Offset: 0x00000034
    Size: 44
    Time: 714192.069000 ms
     Date:        26.07.2014
     Time:        00:19:37 (GMT+0)
     Zone:        ''
     Day of week: 6
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000060
    Size: 40
    Time: 714192.635000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    125000 µs (1/8.00)
Block: LENS
  Offset: 0x00000088
    Size: 96
    Time: 714192.694000 ms
     Name:        '1-65535mm'
     Serial:      ''
     Focal Len:   300 mm
     Focus Dist:  0 mm
     Aperture:    f/4.00
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 714192.710000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
Block: WBAL
  Offset: 0x0000013c
    Size: 44
    Time: 714192.759000 ms
     Mode:   0
     Kelvin:   5000
     Gain R:   1024
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x00000168
    Size: 52
    Time: 714192.771000 ms
     picStyle:   133
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: VIDF
  Offset: 0x0000019c
    Size: 27067688
    Time: 716170.234000 ms
   Frame: #100
    Crop: 0x0
     Pan: 0x0
   Space: 0

Vertical stripes correction:
  1.000  1.000    1      1      1      1      1      1 
[ERROR] Invalid block size at position 0x019d06c4
Processed 1 video frames
Done


So now it's clear the problem is not ETTR related, It seems related to MLV files with more than 100 pictures.

I don't know if the bug is in mlv_dump, or in "silent.mo" module. 



josepvm

@g3gg0

These are the first 200Mb of the MLV file in my last test.  I have used the linux command "split --bytes=200M 00000000.MLV mlv_part"

https://www.dropbox.com/s/ca5qbkpeiv7vai7/mlv_partaa

josepvm

I see that the "mlv_dump -h" output is much shorter for a MLV file with 101 pictures (with errors)


MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '00000000.MLV'
   - Verbose messages
   - Verify file structure
   - Dump all block information
File 00000000.MLV opened
File 00000000.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 3507273469957387890
    FPS         : 1.000000
    File        : 0 / 0
    Frames Video: 101
    Frames Audio: 0
Block: RTCI
  Offset: 0x00000034
    Size: 44
    Time: 838839.378000 ms
     Date:        25.07.2014
     Time:        23:50:47 (GMT+0)
     Zone:        ''
     Day of week: 5
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000060
    Size: 40
    Time: 838839.942000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    0 µs (1/inf)
Block: LENS
  Offset: 0x00000088
    Size: 96
    Time: 838839.969000 ms
     Name:        '1-65535mm'
     Serial:      ''
     Focal Len:   300 mm
     Focus Dist:  0 mm
     Aperture:    f/4.00
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 838839.984000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
Block: WBAL
  Offset: 0x0000013c
    Size: 44
    Time: 838840.034000 ms
     Mode:   0
     Kelvin:   5000
     Gain R:   1024
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x00000168
    Size: 52
    Time: 838840.046000 ms
     picStyle:   133
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: VIDF
  Offset: 0x0000019c
    Size: 27067688
    Time: 841076.753000 ms
   Frame: #100
    Crop: 0x0
     Pan: 0x0
   Space: 0
Processed 1 video frames
Done


than in the MLV file shot exactly in the same conditions but with only 20 pictures:  dump_h.txt

These two files have been shot using ETTR.

barepixels

well, my 142 pics shot as DNGs and not MLV came out OK.  Not a single corrupted file.  Just no EXIF info in any of them.

I dont think the problem is mlv_dump because previously other programs were able to extract from MLV  but they have black bars issue.  Now those  programs can't even open these large MLV.
5D2 + nightly ML

josepvm

Tested again, same setup, 101 pictures in a MLV file, this time using a new build of the latest "fullres-silent-pics" branch, including the latest "silent.c" file by dmilligan in this commit

Same problem, only one DNG extracted, completely black.



  MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '00000001.MLV'
   - Verbose messages
   - Convert to DNG frames
   - Output into '00000001_frame_'
File 00000001.MLV opened
File 00000001.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 7892295102139271782
    FPS         : 1.000000
    File        : 0 / 0
    Frames Video: 101
    Frames Audio: 0
Block: RTCI
  Offset: 0x00000034
    Size: 44
    Time: 724172.532000 ms
     Date:        26.07.2014
     Time:        09:56:06 (GMT+0)
     Zone:        ''
     Day of week: 6
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000060
    Size: 40
    Time: 724173.094000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    457000 µs (1/2.19)
Block: LENS
  Offset: 0x00000088
    Size: 96
    Time: 724173.153000 ms
     Name:        '1-65535mm'
     Serial:      ''
     Focal Len:   300 mm
     Focus Dist:  0 mm
     Aperture:    f/4.00
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 724173.169000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
Block: WBAL
  Offset: 0x0000013c
    Size: 44
    Time: 724173.214000 ms
     Mode:   0
     Kelvin:   5000
     Gain R:   1024
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x00000168
    Size: 52
    Time: 724173.226000 ms
     picStyle:   133
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: VIDF
  Offset: 0x0000019c
    Size: 27067688
    Time: 726217.946000 ms
   Frame: #100
    Crop: 0x0
     Pan: 0x0
   Space: 0

Vertical stripes correction:
  1.000  1.000    1      1      1      1      1      1 
[ERROR] Invalid block size at position 0x019d06c4
Processed 1 video frames
Done




g3gg0

okay it seems the frame #100 is getting saved into the first MLV file.
there it is overwriting RAWI and first part of the first frame.
-> this is an implementation failure from silent.mo, dmilligan has to fix that


as the overwritten RAWI has 0xB4 bytes, there are 0xB4 bytes crap right at the address mlv_dump prints.

how to fix those files:

if you see mlv_dump saying: [ERROR] Invalid block size at position 0x019d06c4 then
- open the file in a hex editor
- go to that offset
- mark 0xB4 bytes for deletion
- you should see a RTCI right after the deleted portion



- delete that
- save the file
- open a good file
- look for the RAWI block


- copy RAWI (0xB4 bytes) from the good file
- paste it before the first RTCI on the defect file (offset 0x34)
- voila


Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

josepvm

Thanks, g3gg0  ;)

I do not have any valuable images in the affected files, they are all only tests.

I was interested in testing the feature because I plan to capture a real landscape timelapse next week during my holidays ... now I know I cannot use MLV format until fixed. I will use fullres silent pics but saved as individual DNGs



glubber

EOS 550D // Sigma 18-200 // Sigma 18-70 // Canon 10-18 STM

barepixels

Hi g3gg0

I have a bad MLV file 2gigs  102 pics

I used your  method and were able to extract dng00 to dng55  (56 good ones)  and a dng101 (bad one)

still missing more files.  am unable to find the next error offset
5D2 + nightly ML

g3gg0

Quote from: barepixels on July 26, 2014, 02:16:10 PM
Hi g3gg0

I have a bad MLV file 2gigs  102 pics

I used your  method and were able to extract dng00 to dng55  (56 good ones)  and a dng101 (bad one)

still missing more files.  am unable to find the next error offset

can you send it to me?
e.g. install BitTorrentSync and use this share key: ATR2MPYDB6OGEKSPBKQZ2O2X73HPGQGQK
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

barepixels

5D2 + nightly ML

g3gg0

enter the key into the top field and select the directory to share with me.
then press ok.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

@barepixels:
you are sure there are > 100 images?
filesize 2159233448 / imagesize 38557376 = 56
thats all mlv_dump can get out of the file.

@dmilligan:
it looks like there is something odd with the file writing when reaching 2GiB.
can you limit the file size to approx 2GiB - 50M ?  (e.g. 0x7CD00000)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

barepixels

@g3gg0

your math sound right but I could swear when i shot it.. it was 102 pics.  also there was a file  name 101 when i extracted fail.  Possible it didn't write the next chunk ... am pretty sure I would copy to hard drive if there is more.  anyway thnx for testing... am going to test recover on a 9gigs shoot now.  5 segments

EDIT:  I just notice when doing mlv_dump -v  on the next set of files  the error is at the same offset

[ERROR] Invalid block size at position 0x024c585c
5D2 + nightly ML

g3gg0

thats why i say that there seems to be a file size bug >2GiB
maybe everything above 2GiB of a file is lost and misplaced

the seek call
> FIO_SeekFile(save_file, 0, SEEK_END);
seems not to seek to the end correctly. i know it wont handle positions > 2GiB if passed as parameter.

dmilligan could use the one i use for mlv_play to seek correctly:


/* as soon all cameras provide FIO_SeekSkipFile, we can remove that temporary code */
static uint64_t FIO_SeekSkipFile_emulate(FILE* stream, uint64_t position, int whence)
{
    const uint32_t maxOffset = 0x7FFFFFFF;
   
    /* the OS routine FIO_SeekFile only accepts signed integers as position, so work around to position absolutely */
    if(whence == SEEK_SET && position > maxOffset)
    {
        FIO_SeekFile(stream, 0, SEEK_SET);
       
        while(position > maxOffset)
        {
            FIO_SeekFile(stream, maxOffset, SEEK_CUR);
            position -= maxOffset;
        }
        return FIO_SeekFile(stream, (uint32_t)position, SEEK_CUR);
    }
    return FIO_SeekFile(stream, (uint32_t)position, whence);
}

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

barepixels

5D2 + nightly ML

barepixels

5D2 + nightly ML

g3gg0

Quote from: g3gg0 on July 26, 2014, 05:58:50 PM
thats why i say that there seems to be a file size bug >2GiB
maybe everything above 2GiB of a file is lost and misplaced

this ;)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

barepixels

yeah, am just documenting for the developer to analyze
5D2 + nightly ML

dmilligan

Quote from: g3gg0 on July 26, 2014, 05:58:50 PM
the seek call
> FIO_SeekFile(save_file, 0, SEEK_END);
seems not to seek to the end correctly. i know it wont handle positions > 2GiB if passed as parameter.
I guess the position parameter is signed 32 bit int which would make sense for a 2GiB limit. I think FIO_SeekSkipFile has been found on all cameras now, so should I just use that?