Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

Started by g3gg0, July 15, 2013, 10:58:23 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

g3gg0

with the latest commits, i added periodic updates of LENS, EXPO and RTCI blocks.
they get embedded into the unused space between two frames, so we have no additional slowdown due to those blocks.
this requires the ML core to be compiled again, as i need the msg_queue_count function i just added. (5D3)

by the way - did some of you run speed tests and comparisons between the old format and the new one?
dont have a reliable card to say anything about speeds, but i think the write rate will be a bit lower.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

gnarr

Great job g3gg0! Could you maybe post a tiny mlv file here, so we can start to develop converters? :)

AdamTheBoy

This is really exciting, full of great ideas and I'm really happy to see magic lantern raw looking more and more convenient and usable professionally.  The custom tagging and take number is a fantastic addition, I think something like this was really needed.  I'm wondering if it would be possible to have an option to name the .mlv files after their tag and take number.  It would be awesome if a file would come out looking like this: "Day 1 take 1.MLV" or "Night test take 5.MLV" or "Scene 3 setup 3 take 5.MLV"

On a set a script supervisor could be keeping track of good or bad takes and then you don't even have to process the takes you know were bad, those files could just be deleted and save precious time.  Really, this new format is great and I can't wait to see more, thanks so much everyone for your work.

AnotherDave

Just installed it, and I definitely notice the write speed is slower on the 5D3 with 64GB KomputerBay 1000x cards.  Card was getting 94.6/MBs previously and now only gets around 84.2/MBs which is on the edge of 4512 expected and continuous.

I'm on a mac, what can I use to convert and check the files?

Thanks.

gogauze

In case you are still wanting some of this g3gg0:


File Header (MLVI)
    Size        : 0x00000038
    Ver         : v2.0
    GUID        : E2B601B3F770EFC9
    FPS         : 23.976000
    Frames Video: 256
    Frames Audio: 0
Block: RAWI
    Size: 0x000000B8
    Time: 2.933000 ms
    Res:  1920x1080
    raw_info:
      api_version      0x0B152000
      height           4797520
      width            14
      pitch            2046
      frame_size       0x00003A98
      bits_per_pixel   0
      black_level      0
      white_level      1928
      active_area.y1   2074
      active_area.x1   0
      active_area.y2   0
      active_area.x2   33620224
      exposure_bias    1, 6722
      cfa_pattern      0x00002710
      calibration_ill  -635
Block: INFO
    Size: 0x00000010
    Time: 5.250000 ms
Block: RTCI
    Size: 0x00000030
    Time: 7.161000 ms
     Date:        29.06.2013
     Time:        07:44:32 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 209
     Daylight s.: 0
Block: EXPO
    Size: 0x00000028
    Time: 7.832000 ms
     ISO Mode:   0
     ISO:        1600
     ISO Analog: 104
     ISO DGain:  0 EV
     Shutter:    20845 µs (1/47.97)
Block: LENS
    Size: 0x00000060
    Time: 7.866000 ms
     Name:        'EF35mm f/1.4L USM'
     Serial:      ''
     Focal Len:   35 mm
     Focus Dist:  88 mm
     Aperture:    f/0.00
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000001
     Flags:       0x00000000
Block: IDNT
    Size: 0x00000058
    Time: 7.978000 ms
     Camera Name:   'Canon EOS 5D Mark III'
     Camera Serial: '131EF8CF14'
     Camera Model:  0x80000285


However, after tooling around with it and getting it to work, raw2dng nor RAWMagic recognize the file as a .raw   :'(

EDIT: Changing the file suffix from .raw to .RAW caused raw2dng to stop denying that it exists at all, and now just returns the standard "This ain't a lv_rec RAW file."

crazyrunner33

Quote from: AnotherDave on July 30, 2013, 04:50:01 AM
Just installed it, and I definitely notice the write speed is slower on the 5D3 with 64GB KomputerBay 1000x cards.  Card was getting 94.6/MBs previously and now only gets around 84.2/MBs which is on the edge of 4512 expected and continuous.

I'm on a mac, what can I use to convert and check the files?

Thanks.

My results are almost identical with the 64 GB KomputerBay card.
5D Mark III, 7D

tonybeccar

Quote from: AnotherDave on July 29, 2013, 09:51:54 PM
I'd settle for output only.  :-)

Hey anotherdave! Maybe there's a simpler and smarter solution! There are LOADS of android / iphone apps which control the camera, so: Why not connect it through usb and the phone would act as a TC generator, allowing via the headphone jack to give TC Signal to whatever devices you want? (by using miniplug splitters of course..) I mean if there are already apps for that it means that it has been done already, and comunicating with the camera also, the pieces are there!! We just need one big smart guy to put them together ;) haha

Cheers!

g3gg0

About write speeds:
when you chose the largest frame format, whats the difference in the frame count?
i.e. old 300 frames, mlv 250 frames
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Redrocks

5d2 Komputerbay 64GB:

a.d's 728e571a1276 @1872 x 1250 24fps = 260 / 295
g3gg0 25th build same set up = 230 / 250

g3gg0

Can you try two or three next smaller resolutions too?
Thanks!
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

AnotherDave

Quote from: tonybeccar on July 30, 2013, 08:50:32 AM
Hey anotherdave! Maybe there's a simpler and smarter solution! There are LOADS of android / iphone apps which control the camera, so: Why not connect it through usb and the phone would act as a TC generator, allowing via the headphone jack to give TC Signal to whatever devices you want? (by using miniplug splitters of course..) I mean if there are already apps for that it means that it has been done already, and comunicating with the camera also, the pieces are there!! We just need one big smart guy to put them together ;) haha

Cheers!

How is that easier than generating a TC sync from the camera itself?  I want this as a feature, specifically to allow this camera to be used on more professional shoots.  Making it complex isn't good for the client...

Redrocks

5d2, Komputerbay 64GB, a.d's 728e571a1276 VS g3gg0 28th build. 1872, 24fps, GD off:

1250 @3:2

294 / 229
262 / 234
273 / 246
260 / 218
299 / 236

1054 @16:9

1110 / 913
1184 / 975
1014 / 821
1341 / 660
1215 / 1025
1569 / 683

1012 @1.85:1
a.d. is near enough continuous at this ratio
/ 2754
/ 3005
/ 3042
/ 2334 card full; reformat.
/ 700
/ 3078

AnotherDave

Will this format store white balance info?  Picture profiles?

Still not sure how to convert .mlv to .dng too :-/

g3gg0

@Redrocks:
thanks a lot!

@AnotherDave:
why picture profiles and wb?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

AnotherDave

Quote from: g3gg0 on July 30, 2013, 08:44:35 PM
@AnotherDave:
why picture profiles and wb?

I recently shot a project for a client with the 5d3 ML Raw Hack.  It was a 5 day shoot for a short film.  We filled 42 cards (6 x KB 64gb) for a total sum of 2.68TB and 1,400 clips.

I had to spend a lot of time afterward color correcting the footage for the client, adjusting the WB since non of that data was stored, and applying a look that was similar to Similaar's Flat_10 PP, which is what I was displaying for them on the monitors.

If it could store that information, it'd mean that I could just hand them footage instead of leaving them with something they don't understand.

*I wouldn't want to hand a client footage that looked as if it was shot @5500k under tungsten lighting.  I know what raw is, and the colorist will know what raw is... but the client would just see it and think I didn't know what I was doing.

Also, I was shooting at -1 EV to aid in ETTR to improve my S/N ratio.  If the image fine tuning data could also be included, that'd be a big help too!

Dave

AnotherDave

I can continue to run more tests if someone can tell me how to convert the .mlv files :-/

Toffifee

Really excited for audio support. Go g3gg0!
Own a 5D MIII, not familiar with developing, but I'd love to help if you need testers.

g3gg0

Quote from: AnotherDave on July 30, 2013, 09:56:13 PM
I can continue to run more tests if someone can tell me how to convert the .mlv files :-/

this:

Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

Quote from: AnotherDave on July 30, 2013, 09:54:12 PM
I had to spend a lot of time afterward color correcting the footage for the client, adjusting the WB since non of that data was stored, and applying a look that was similar to Similaar's Flat_10 PP, which is what I was displaying for them on the monitors.

wb makes a bit sense, but color profile not much.
no clue how to tell the DNG format that a client application has to apply a color profile that is canon specific...
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

5D3 autoexec.bin + modules: http://upload.g3gg0.de/pub_files/50f8e2a45595335252a79b19aba2f422/raw_rec.zip

Redrocks:
can you try the latest version with the sizes you already posted?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Redrocks

Can you recheck the files for me or have I messed up? I put all your new .mo files in the modules folder, left the .bin in the root and deleted the .sym. Then the camera refused to fire up. Tried the build on three cards and ended up having to insert a clean card whilst the camera was switched on to roll back to the latest regular nightly.   

AnotherDave

Quote from: g3gg0 on July 31, 2013, 12:49:43 AM
wb makes a bit sense, but color profile not much.
no clue how to tell the DNG format that a client application has to apply a color profile that is canon specific...
If I had to give a reason... it would just be for consistency. 

If I needed a catch phrase... it would be "because the Red camera can do it" :-)

Dave

AnotherDave

Tonight's build is far slower for me...  I'm only getting 81.5MB/s 54ms idle, 4000-7000 frames expected,  takes a lot longer to works it way up to the speed too.

5D3, KB 64 1000x, 24fps, 1080p

dadinio13

i can't understood why it tell me "not a chdk dng" ?

MacBook-Pro-de-david:~ david$ /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/dng2raw /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW dng
reading /Volumes/FREECOM HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW (size=1382573000)
read 1382573000
not a chdk dng: Undefined error: 0
ifd 0: (28804)
ifd 1: (15936)
ifd 2: (16907)
Segmentation fault: 11


and why i get this when i compile mlv_dump ?

MacBook-Pro-de-david:raw_rec david$ gcc mlv_dump.c -o mlv_dump -I../../src
mlv_dump.c:109:65: error: invalid suffix "d" on floating constant
mlv_dump.c:163:83: error: invalid suffix "d" on floating constant
MacBook-Pro-de-david:raw_rec david$

g3gg0

Quote from: dadinio13 on July 31, 2013, 10:25:13 AM
i can't understood why it tell me "not a chdk dng" ?

why did you specify dng ? thats a raw file.

Quote from: dadinio13 on July 31, 2013, 10:25:13 AM
mlv_dump.c:109:65: error: invalid suffix "d" on floating constant

hm will check later, maybe a copy paste error and my compiler didnt warn?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!