Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

Started by g3gg0, July 15, 2013, 10:58:23 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

jarandov

Hi guys

it could be great if someone create a program for mac like Rawmagic lite ( no problem to pay for it) if the program could handle mlv files correctly. I thinking about Mac version no windows. I hope that i can see some this program in close future.

Thanks in advance.

dmilligan

Quote from: jarandov on May 20, 2014, 01:11:16 PM
it could be great if someone create a program for mac like Rawmagic lite ( no problem to pay for it) if the program could handle mlv files correctly. I thinking about Mac version no windows. I hope that i can see some this program in close future.
Do you need CDNG output or just DNG? http://www.magiclantern.fm/forum/index.php?topic=9560

jarandov

Hi Dmilligan,

Maybe a stupid question. What is the difference between dng cdng? I have tried MlRawViewer 1.1.4 but the program could chose only between mov and dng. Not cdng. Does the program export also sound files from mlv?

dmilligan

Quote from: jarandov on May 20, 2014, 02:50:03 PM
What is the difference between dng cdng?
I'm not an expert, but AFAIK, the main difference is simply metadata, but some video editors (e.g. Adobe Premiere) only accept files with all the correct CDNG metadata (there may be some other issues as well including bit depth).

Quote from: jarandov on May 20, 2014, 02:50:03 PM
the program could chose only between mov and dng
Right. What I meant was, do you require CDNG output? If not, then MLRawViewer should work fine, it can output DNG but not CNDG. The fact that you don't know the difference between DNG/CDNG, tells me you likely don't require it.

Quote from: jarandov on May 20, 2014, 02:50:03 PM
Does the program export also sound files from mlv?
Read the thread ;)

arch

Quote from: jarandov on May 20, 2014, 01:11:16 PM
Hi guys

it could be great if someone create a program for mac like Rawmagic lite ( no problem to pay for it) if the program could handle mlv files correctly. I thinking about Mac version no windows. I hope that i can see some this program in close future.

Thanks in advance.

I've been using MLVMystic. Have you tried that program? It's working awesome for me, and it extracts the audio files for me too, but you still have to match audio and video in DaVinci Resolve.

ted ramasola

For questions, answers and info regarding raw post processing pls post here:  http://www.magiclantern.fm/forum/index.php?board=54
5DmkII  / 7D
www.ramasolaproductions.com
Texas

llirik

Quote from: arch on May 22, 2014, 01:55:12 AM
I've been using MLVMystic. Have you tried that program? It's working awesome for me, and it extracts the audio files for me too, but you still have to match audio and video in DaVinci Resolve.
This has been my workflow for the last 2 months or so. So far, it's working out well.

ted ramasola

Quote from: llirik on May 23, 2014, 01:20:23 AM
This has been my workflow for the last 2 months or so. So far, it's working out well.

do you understand the post I made above yours?
5DmkII  / 7D
www.ramasolaproductions.com
Texas

KMikhail

Has anyone managed to have fullhd raw recording at 30fps with 1.2.3 and sound enabled, including card spanning?

dpjpandone

I really like the way you can set mlv up to name your files sequentially, but I think it would be incredibly useful if there was a naming option that lets you set a prefix like "Cam1" or "CamB" and then it automatically adds the timestamp from RTC to the end of the file name - so if you start recording on camera B at 12:23:21 in the afternoon, the generated file name will be: CamB_122321

I know you can do stuff like this in asset management programs, but it would be nice to skip those and dump straight to your project folder if all you need is faux timecode.

What do you think?

fascina

Hello g3gg0, thank you for all your efforts regarding MLV.

Which graphical user interface do you personally recommend for mlv_dump on Windows for batch conversion? Do you recommend your MLV Browser?

dpjpandone

Quote from: dpjpandone on May 25, 2014, 04:44:40 PM
a naming option that lets you set a prefix like "Cam1" or "CamB" and then it automatically adds the timestamp from RTC to the end of the file name

I've been having a look through the mlvrec module tonight in my quest for a better way to sync in post. Can someone tell me if it's possible to get a higher resolution of time from "LoadCalendarFromRTC"  for example, does rtc give millisecond accuracy? if so, I think we should divide by framerate and add frames to the filename/metadata, is it possible to call "LoadCalendar" right before the first frame is recorded? So that performance does not influence accuracy? If we can do this, you can let your ext. audio recorder roll for the entire shoot, and you would only have to sync the first video clip manually. After that you can use the data from RTC to set timecode on the rest of the clips.

What do you think?

g3gg0

i dont have any clue about SMPTE and always wait for information from those who want to have support.

regarding your use case:
which information/field in MLV then should contain the time stamp so you can make use of it?

MLV itself contains a lot, but the RTC is only 1-second accurate.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

Quote from: fascina on May 25, 2014, 11:12:18 PM
Hello g3gg0, thank you for all your efforts regarding MLV.

Which graphical user interface do you personally recommend for mlv_dump on Windows for batch conversion? Do you recommend your MLV Browser?

you are welcome.
uhm i recommend the one you can work with the best.
MLVBroseSharp is just a reference application that uses mlv_dump.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

dpjpandone

Quote from: g3gg0 on May 26, 2014, 02:54:18 PM



which information/field in MLV then should contain the time stamp so you can make use of it?



I think it would be immediately useful to add the timestamp to the filename, (it's already in mlv field)

The reason is that when proxies are created (and I think most render to a proxy format like cineform or prores for editing) the mlv data is lost, but filenames are easily retained.

The timestamp is not very useful when the file is in MLV format, it's not needed until editing, so this is why I think there should be an option to add timestamp to filenames.

When editing software allows native editing on MLV format, and computer hardware that can do it efficiently exists, we can make use of it in MLV data as well


The use case is this, when I import all the converted video files into adobe premiere, I can right click on a file, and easily assign it a starting timecode. If the timestamp is in the filename, it's a matter of copying and pasting into timecode window, and then at least the clips will automatically synchronize within one second. This is a huge timesaver for cameras that do not have reference sound for sync. At least then they are in the ballpark and only have to be nudged a few frames in either direction to achieve sync.

Another advantage is that when you open a project on another computer, and it has to search for files, it will be highly unlikely that duplicate files are found, since each file will have a timestamp in the name. So your project should find all the files with ease.

Here is a video that may help you understand: http://library.creativecow.net/harrington_richard/DSLR-Timecode/1

They are showing an excellent way to do this with h.264, but this does not work with mlv -> acr -> proxy workflow, because MLV timestamp is lost in conversion.

Re: second accurate clock:

second accuracy is better than no timestamp at all, but I think we can find a workaround to get frame accurate timestamp:

Since each frame has it's own timestamp in the MLV, we must find the first frame that has an updated timestamp, and we should use that as our reference to which all other frames timestamps are derived from. if I was coding for arduino it would go like this:


 
previousTimestamp=LoadRTCCal;
currentTimestamp=LoadRTCCal;
if currentTimestamp != previousTimestamp;
reference = currentTimestamp;

MLVtimestamp = (reference - offset);

I know this isn't entirely correct, but I hope it is helpful.

We will need a counter that is reset when the first frame starts recording, and the counter will increment each time a new frame is captured, this will give us our offset

Once we have found our reference frame, and subtracted the offset, we should have a frame accurate timestamp.

Please let me know if you need more information, or if there is anything I can do to help.




g3gg0

i understand your workflow now. thanks.
this helps me in finding a good solution.

Quote from: dpjpandone on May 26, 2014, 04:45:31 PMeasily assign it a starting timecode.

in which format do you need the timestamp?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

dpjpandone

I would prefer:

UserDefinableText_DD_MM_YYYY_Hours_Minutes_Seconds_Frames

if that is too many characters, then eliminate DD_MM_YYYY and the user can enter a short form of date in UserDefinableText if needed.




g3gg0

i made some analysis of microsecond timer vs. RTC and found some interesting stuff.

the RTC (well, its crystal) has quite a high drift compared to the digic microsecond timer.
we dont know which one is the more accurate clock.
but comparing each other you will get this:



one dot per 2 seconds. Y axis shows 857 to 878 µsec "offset" to the digic µsec timer.
every dot shows when the RTC does a second-tick.
the higher jumps are due the RTC-internal offset correction that is happening every minute.


according to this longer trace (10 minutes) my 5D3 has approximately 1 ms drift per minute between those two clocks,
thats approximately one frame off after 30 minutes inaccuracy at 25 fps.

this means if i calibrate the two clocks on startup, which takes two seconds, and from that on
i only use the interal digic timer (which is simple to read and provides a 1µsec resolution) then after operating
the camera for 30 minutes you will have one frame offset in the timecode no matter if you recorded one or 20 clips.

is this a problem?

solutions:
- do nothing and live with that
- rerun calibration on every recording start or every 10 minutes (two seconds delay)
- add some drift correction (already tested, have a 2 msec inaccuracy and nearly no drift anymore)

the drift correction isnt that simple to do and would take 30 minutes, but that has to be done only once.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

oh and i forgot to mention:
we cannot name the file in camera that long. we can only do 8.3 there.
renaming to [prefix]_[DDMMYYHHMMSSff].MLV would only be possible on PC/mac.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Audionut

Quote from: g3gg0 on May 27, 2014, 12:59:49 AM
- add some drift correction (already tested, have a 2 msec inaccuracy and nearly no drift anymore)

the drift correction isnt that simple to do and would take 30 minutes, but that has to be done only once.

This would be my pick.  30 minutes once, for best accuracy.  If you're anal enough to care about accuracy, then 30 minutes is a non issue.

dpjpandone

Quote from: g3gg0 on May 27, 2014, 12:59:49 AM
the camera for 30 minutes you will have one frame offset in the timecode no matter if you recorded one or 20 clips.

is this a problem?


This is not a problem, I wouldn't mind correcting the drift in post.

are you saying it will take 30 minutes for you to code drift correction? What can I do for 30 minutes that will make your life easier?  That is of course the best solution and it is easy to choose when it is someone else's time...

For the second solution, adding 2 seconds delay before recording starts is undesirable, it already takes a while for recording to begin sometimes. I think I would rather have it drift. 


g3gg0


well for me to implement its one day of work and testing.
for running the calibration, you have to spend the 30 minutes mentioned.

after calibration the clocks are quite in sync at the given temperature and give you a few miliseconds accuracy.
i can also calibrate the drift correction so we will have <0.1msec accuracy to the RTC - AT THIS ENVIRONMENTAL CONDITIONS (air pressure, humidity, temperature).

so i see three parts of calibration that take some time

1. [2 sec] RTC/µsec offset detection (will be different after every poweron)
2. [10-30 min] RTC/µsec drift detection (depends lets say 50% on camera and 50% on environmental conditions) 
3. [2 sec] RTC correction rate detection (can be done along with step 2)

all together i guess thats 3 days effort.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

dpjpandone

This is so incredibly awesome. Whatever steps are necessary to correct the drift would be absolutely worth it. Having this frame accurate timestamp will save an incredible amount of time in post.

To get around 8.3 filenames, I guess that's exactly enough characters for HHMMSSFF  and any filebrowser will show they date they were created in the details anyways.

g3gg0

Quote from: dpjpandone on May 27, 2014, 07:56:48 PM
HHMMSSFF

okay but there is one thing i dont really get.

according to SMPTE standards, for NTSC 29.97 fps there are two frame drops every minute, except every 10 minutes.
so if its exactly 00:00:00 in the night and i start recording, then the first frame's time should be 00:00:00:02, right?
and if i start recording at 00:00:00 plus 100 msecs, which frame number should i pick?

or can i completely ignore frame dropping for the file naming stuff?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

dpjpandone

from this: http://www.larryjordan.biz/technique-drop-frame-vs-non-drop-frame-timecode/

"The rule of drop-frame timecode is: two frames of timecode are dropped every minute, except on the tenth minute, when nothing is dropped. This means that over the course of an hour, 108 timecode labels are skipped. The exact numbering goes: 59:28, 59:29, 1:00:02, 1:00:03... (Notice two timecode labels were skipped: 1:00:00, and 1:00:01."

so if recording at 23.97 or 29.97, yes. it starts at 00:02

I have to think about the second question for a while. My first thought is to just round up to next frame  for file naming. maybe I didn't think this all the way through. It's a unique system because we are using RTC as master clock.

There are some formulas for timecode math here:

http://www.andrewduncan.ws/Timecodes/Timecodes.html