Developers, would this be possible for RAW video recording?

[markodarko]

Hi Guys,

I literally just stumbled across Magic Lantern a few hours ago and have been reading and playing with all it's glorious goodies ever since! Great stuff, thank you so much for everyone's effort.

One thing I noticed though is that I'm using MLVFS to mount the MLV file in Mac OS X which works great, but as it mounts the MLV file pretty much instantly I'm guessing that the MLV file is just a container for the individual DNG files and the audio, is that right? If that's the case, instead of recording to a MLV file in camera, would it be possible to just save out the individual DNG frame files to a new folder each time we press record instead of streaming them to the MLV container?

If that were indeed possible then there'd be no 4Gb file limit on my 5DII(!) :-D, plus the extra step of converting to DNG would be avoided.

I'm guessing that this ISN'T possible else it would already have been done, but thought I'd ask just in case.

Thanks,

Mark.

[Walter Schulz]

No need to discuss merits: ExFAT not supported on 5D2 -> File size limit 4 GB.

[markodarko]

I think you misunderstood my point. The individual image frames of the video do not themselves exceed 4Gb so if they could be saved as individual DNG "frames" instead of all of the frames wrapped into one MLV file, then the 4Gb limit would never be reached - in the same way as it's not reached when you you take 1000 RAW images in photo mode.

So, if shooting @ 24 fps for 1 minute for example, a folder would be created and within that there would be 1440 individual DNG files along with the audio WAV instead of *one* MLV file encapsulating the whole movie.

Does that make sense?

[Danne]

I think dmilligan developed a dng module for this purpose. Not sure it,s working though.

https://bitbucket.org/dmilligan/magic-lantern/branch/ml_dng_lj92

[dmilligan]

So MLV is not a container for DNG data. MLVFS is actually doing work in the background to transform the MLV data to valid DNG files. This task is easy on a powerful modern desktop CPU, and MLVFS can do it in realtime; it's impossible (at realtime video framerates) on the measly little ARM CPU in the cameras.

The heart of the issue is endianness. The DNG spec requires that image data be big endian (the file headers can be either big or little, but the image data must always be big), the camera's sensor produces data that is little endian. So to create valid DNGs we must transform big endian data to little endian (this basically involves swapping all the bytes). In camera, for a single video frame this simple transformation takes on the order of several seconds (in a modern CPU with SIMD optimizations we can do a frame in several milliseconds).

So we can save DNGs in camera, and this is done for silent pics where we're not trying to save 24,30,50, or 60 frames per second. My PR about a DNG module is to simply replace some of the old DNG writing code that doesn't do a great job of handling DNG metadata properly (the endianness swap is still the same, written by g33g0 in assembly, so as to be as optimized as possible).

If the DNG spec allowed little endian image data, we might in fact be able save DNGs in camera for video (I don't know exactly what the overhead is for creating individual files so that could also prevent it from being possible).

[Licaon_Kter]

@markodarko: after 4gb are full MLV_REC module creates a M00 file and after another 4gb another M01 and so on. What's the issue exactly since RAW video in MLV format is NOT limited to 4GB anyway since MLVFS loads the files MLV and Mxx automatictly?

[Walter Schulz]

His idea is to record DNG frames instead of streaming to MLV file thus avoiding post-processing converting MLV to DNG.

[Licaon_Kter]

First he mentions the 4gb limit... where there IS NO such limit (for RAW anyway) in place actually.

[reddeercity]

He was referring to the 5D mark ii  limited to fat32 (4GB files)

@markodarko I'm a 5D2 power user , I think you are looking for the  highest resolution @ the fastest speed possible right ?
If so use the original "Raw" format version 1.0 not MLV version 2.0
Even Raw needs to be extracted to DNG or Cdng's . Note Raw Ver. 1.0 dose not support audio
But you will not be able to use MLVFS as it Dose not support "Raw ver 1.0"
Which in my opinion is very silly , it should be supported in there.
But there is other tools that support Raw files
"MlRawViewer 1.3.3 (CDNG/MLV/RAW Viewer & Encoder, Linux/Mac/Win)"
http://www.magiclantern.fm/forum/index.php?topic=9560.msg91165#msg91165

[dmilligan]

Which in my opinion is very silly , it should be supported in there.

*Sigh*

You just don't get it do you?

[Licaon_Kter]

He was referring to the 5D mark ii  limited to fat32 (4GB files)

I understand, the camera does not matter, repeat after me: "the FAT32 4GB FILE SIZE LIMIT DOES NOT MATTER for RAW when recording with MLV_REC as .MLV/.Mxx files"

[reddeercity]

It never ends Dose It !
and yes I do get , you don't thou .
have a nice life
bye.

[markodarko]

@dmilligan: Thanks for the explanation! I hadn't realised that the endian was different. Hmm - how about this then (assuming that the endian conversion time you mention takes several seconds per frame in camera during recording only - as the processor is already busy - and not when idle) - a module which runs when you're not recording and converts the MLV file into a folder full of DNG files? There's plenty of time between pressing record that this could happen for the last shot that was taken - well, in my case anyway. Perhaps something like this:

1. User loads the "Create DNG folders instead of MLV" module
When in this mode, ML does not record to a single MLV file but instead creates a folder of individual frame files. Let's call them GND files as the endian is backwards. 

2. When the user finishes recording the module shows the "busy" message on screen and converts the endianness of the frames that were just shot into correct DNG files.

That way we'd have a choice - either create a movie as an MLV (and have to convert to DNG in post) as we do now but with the benefit of immediately being able to press record once we've finished recording, OR... have a folder of separate DNG files + audio at the expense of some delay time after recording for the endian conversion process to happen.

Thoughts?

@Licaon_Kter: Interesting. Does MLV_REC continue the recording (audio and video) without any frames dropping?

@reddeercity: Yes, highest resolution with at least 24 fps (the 21fps choices are no good for my needs) are beneficial, but mainly I wanted to save time in post. I figured if the camera spat out a folder of DNG files instead of a single file then there'd be one less step to do in post, and with many many short clips, that one step could save a lot of time.

[Walter Schulz]

@Licaon_Kter: Interesting. Does MLV_REC continue the recording (audio and video) without any frames dropping?

http://magiclantern.fm/forum/index.php?topic=14644

[markodarko]

Thanks for the link, Walter. Good to know!

[dmilligan]

Hmm - how about this then (assuming that the endian conversion time you mention takes several seconds per frame in camera during recording only - as the processor is already busy - and not when idle) - a module which runs when you're not recording and converts the MLV file into a folder full of DNG files?

It won't make much difference. Converting to DNG in camera would probably take an hour or more for a 1 minute clip even when not recording or doing other things.

[markodarko]

Ah. Thanks @dmilligan.

Oh well, good to know. Worth asking all the same.