Magic Lantern Forum

Using Magic Lantern => Raw Video => Topic started by: Midphase on March 06, 2014, 08:42:48 AM

Title: Is MLV all that it's cracked up to be?
Post by: Midphase on March 06, 2014, 08:42:48 AM
First of all, I want to make it abundantly clear that this isn't a criticism or a slam on ML and .mlv.

I'm just trying to get a sense of what exactly .mlv gives me that .raw doesn't. Looking at the EXIF data written to DNG files from .mlv files using MLVMystic and comparing them to the EXIF from CDNG derived from .raw files using RAWMagic, I'm not seeing a whole lot of additional information coming from the .mlv files. There is ISO, and some focal length info, but everything else is pretty much the same in both files.

Besides that, when recording in .mlv on my 5D3, I seem to get slightly worse CF card performance than when recording in .raw. I also noticed that DNG files derived from .mlv seemed to display more pronounced dead pixels than similar takes extracted from .raw as CDNG.

Last but not least, there is the severe lack of a good conversion app on the Mac, and pretty much nothing to get CDNG as opposed to DNG files. RAWMagic has worked flawlessly for me with Resolve, and to a lesser degree with Premiere Pro. .mlv on the other hand has presented challenges until very recently thanks to MLVMystic. The problem with MLVMystic is that it's still in fairly rough beta, it converts considerably slower than RAWMagic, and it doesn't spit out CDNG files which means, for better or for worse Premiere Pro wants nothing to do with them and import into Resolve is a wee bit more of a PITA.

Having said that, I am still wanting to give .mlv every chance in the book, but I have to ask -- what is the specific advantage of .mlv vs. .raw in real world usage?

Thanks for all your replies, I am really curious to know more about the advantages of .mlv over .raw.

PS.

Tomorrow I'm doing a camera shootout with the following cameras:

1. Arri Alexa
2. Red Dragon
3. Red Epic
4. Canon C500
5. Canon 1DC
6. Canon 5D Mark III (ML raw)
7. Sony F55
8. Panasonic GH4
9. Blackmagic Cinema 4k

Should I shoot in .mlv or .raw?
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 06, 2014, 08:56:40 AM
This looks like a pretty good summary to me.

Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
some technical facts:
- structured format
- extensible layout
- as a consequence, we can start with the minimal subset (file header, raw info and then video frames)
- multi-file support (4 GiB splitting is enforced)
- spanning suport (write to CF and SD in parallel to gain 20MiB/s)
- out-of-order data support (frames are written some random order, depending on which memory slot is free)
- audio support
- exact clock/frametime support (every frame has the hardware counter value)
- RTC information (time of day etc)
- align fields in every frame (can differ from frame to frame)

the benefit for post processing will be:
- files can be easily grouped by processing SW due to UIDs and file header information (autodetect file count and which files belong to each other)
- file contains a lot of shooting information like camera model, S/N and lens info
- lens/focus movement can be tracked (if lens reports)
- exact* frame timing can be determined from hw counter values (*=its accuracy is the limiting thing)
- also frame drops are easy to detect
- hopefully exact audio/video sync, even with frame drops
- unsupported frames can be easily skipped (no need to handle e.g. RTC or LENS frames if the tool doesnt need them)
- specified XREF index format to make seeking easier, even with out of order data and spanning writes

why a custom format and not reuse e.g. .mov?
- other formats are good, but none fits to our needs
- hard to make frames align to sector or EDMAC sizes
- they dont support 14 bit raw bayer patterns out of the box
- even when using a flexible container, nearly all sub blocks would need custom additions
- this means a lot of effort to make the standard libs for those formats compatible
- its hard to implement our stuff in a clean way without breaking the whole format

thats the reason why i decided to throw out another format.
it is minimalistic when desired (especially the first implementation will only use a subset of the frames)
and can be extended step by step - while even the most minimalistic parser/post processing tool
can process the latest video files where all stuff is implemented.

if you are a developer (ML or even 3rd party tools) - look over it and make yourself comfortable with that format.
in case there is a bug or something doesnt make sense, please report it.
i would love to get feedback.

here is the link of the spreadsheet that is some kind of reference when designing the format:
https://docs.google.com/spreadsheet/ccc?key=0AgQ2MOkAZTFHdHJraTVTOEpmNEIwTVlKd0dHVi1ULUE#gid=0

implementer's notes
green = fully implemented
blue= implemented, but not 100%
red = not implemented yet, just defined

[MLVI] (once)
- MLVI block is the first block in every .mlv file
- the MLVI block has no timestamp, it is assumed to have timestamp value 0 if necessary
- the MLVI block contains a GUID field which is a random value generated per video shoot
- using the GUID a tool can detect which partial or spanning files belong together, no matter how they are named
- it is the only block that has a fixed position, all other blocks may follow in random order
- fileCount field in the header may get set to the number of total chunks in this recording (the current implementation on camera isn't doing this right)

[RAWI] (once, event triggered)
- this block is known from the old raw_rec versions
- whenever the video format is set to RAW, this block has to appear
- this block exactly specifies how to parse the raw data
- bit depth may be any value from 1 to 16
- settings apply to all VIDF blocks that come after RAWI's timestamp (this implies that RAWI must come before VIDF - at least the timestamp must be lower)
- settings may change during recording, even resolution may change (this is not planned yet, but be aware of this fact)

[VIDF] (periodic)
- the VIDF block contains encoded video data in any format (H.264, raw, YUV422, ...)
- the format of the data in VIDF blocks have to be determined using MLVI.videoClass
- if the video format requires more information, additional format specific "content information" blocks have to be defined (e.g. RAWI)
- VIDF blocks have a variable sized frameSpace which is meant for optimizing in-memory copy operations for address alignment. it may be set to zero or any other value
- the data right after the header is of the size specified in frameSpace and considered random, unusable data. just ignore it.
- the data right after frameSpace is the video data which fills up the rest until blockSize is reached
- the blockSize of a VIDF is therefore sizeof(mlv_vidf_hdr_t) + frameSpace + video_data which means that a VIDF block is a composition of those three data fields
- if frames were skipped, either a VIDF block with zero sized payload may get written or it may be completele omitted
- the format of the data in VIDF frames may change during recording (e.g. resolution, bit depth etc)
- whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks

[WAVI] (once, event triggered)
- when the audio format is set to WAV, this block specifies the exact wave audio format

[AUDF] (periodic)
- see [VIDF] block. same applies to audio

[RTCI] (periodic, event triggered)
- contains the current time of day information that can be gathered from the camera
- may appear with any period, maybe every second or more often
- should get written before any VIDF block appears, else post processing tools cannot reliable extract frame time

[LENS] / [EXPO] / ... (periodic, event triggered)
- whenever a change in exposure settings or lens status (ISO, aperture, focal length, focus dist, ...) is detected a new block is inserted
- all video/audio blocks after these blocks should use those parameters

[IDNT] (once)
- contains camera identification data, like serial number and model identifier
- the camera serial number is written as HEX STRING, so you have to convert it to a 64 bit INTEGER before displaying it

[INFO] (once, event triggered)
- right after this header the info string with the length blockLen - sizeof(mlv_info_hdr_t) follows
- the info string may contain any string entered by the user in format "tag1: value1; tag2: value2"
- tag can for example be strings like take, shot, customer, day etc and value also any string

[NULL] (random)
- ignore this block - its just to fill some writing buffers and thus may contain valid or invalid data

[ELVL] (periodic)
- roll and pitch values read from acceleration sensor is provided with this block

[WBAL] (periodic, event triggered)
- all known information about the current white balance status is provided with this block

[XREF] (once)
- this is the only block written after recording by processing software, but not the camera
- it contains a list to all blocks that appear, sorted by time
- the XREF block is saved to an additional chunk
- files that only contain a XREF block should get named .idx to clarify their use
- .idx files must contain the same MLVI header like all chunks, but only have the XREF block in it

[MARK]
- on keypresses, like halfshutter or any other button, this block gets written for e.g. supplying video cutting positions
- the data embedded into this block is the keypress ID you can get from module.h


possible future blocks:

[OFFS]
- in-camera black and noise reference pictures can be attached here (dark frame, bias frame)
- to be checked if this is useful and doable

[VERS]
- a string follows that may get used to identify ML and raw_rec versions
- should follow the same format as INFO strings, "tag1: value1; tag2: value2"
- tag can be either "ml_core", "raw_rec" or any other string that is used to identify some module/component version
- possible content: "ml-core 20130912; raw_rec v2.1; mlv_fmt v2.0; smpte v0.0"



[MLV Format]
- the Magic Lantern Video format is a block-based file format
- every information, no matter if audio or video data or metadata is written as data block with the same basic structure
- this basic structure includes block type information, block size and timestamp (exception to this is the file header, which has no timestamp, but a version string instead)
- the timestamp field in every block is a) to determine the logical order of data blocks in the file and b) to calculate the wall time distance between any of the blocks in the files
- the file format allows multiple files (=chunks) which basically are in the same format with file header and blocks
- chunks are either sequentially written (due to e.g. 4 GiB file size limitation) or parallel (spanning over mutiple media)
- the first chunk has the extension .mlv, subsequent chunks are numbered .m00, m01, m02, ...
- there is no restriction what may be in which chunk and what not

[processing]
- to accurately process MLV files, first all blocks and their timestamps and offset in source files should get sorted in memory
- when sorting, the sorted data can be written into a XREF block and saved to an additional chunk
- do not rely on any order at all, no matter in which order they were written into a file
- the only reliable indicator is the timestamp in all headers


Quote from: Midphase on March 06, 2014, 08:42:48 AM
First of all, I want to make it abundantly clear that this isn't a criticism or a slam on ML and .mlv.

Then you should probably consider using a more appropriate thread title.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 06, 2014, 09:13:08 AM
No need to re-quote all that. I've read it and re-read it, and believe me, I didn't just fall off the turnip truck. I'm just merely raising the question on a "real world" level and not a programming level.

Even from a data stability point, I've been dismayed at getting more corrupted frames when using .mlv than I ever have in .raw.

I understand what the programmers are saying, but what I am asking is...does this all translate in useable real-world data that makes up for the poor (current) post-production support for .mlv?

I've always maintained that ML raw is a two step deal, one is what happens in the camera, but the second and equally important is what happens once the data leaves the camera. One without the other doesn't work, both have to be rock solid to be considered for professional work.

While there seems to have been a great deal of development time devoted to perfecting the in-camera software and adding more functionality to ML in general, the back end development has been severely lacking. Command-line solutions aren't solutions.

So once again...are the advantages of .mlv in real-world usage worth the extra PITA that comes with it on the back end vs. .raw; which, for better or worse, does have a useable OS X back end in RAWMagic.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 06, 2014, 09:28:20 AM
Quote from: Midphase on March 06, 2014, 09:13:08 AM
Command-line solutions aren't solutions.

That is your own personal opinion, based solely on your own abilities, and/or, expectations.

Lets consider this.

Adobe has some pretty nice front end software.  If you don't like Adobe, well then there is Resolve, or a bunch of other software that may contain a GUI that ticks all your boxes.

What do you think Adobe et all have, that ML doesn't?  Well, off the top of my head, massive amounts of resources for a start.  The sort of resources that are well outside the resources available to a bunch of guys coding stuff as a hobby.

QuoteWhile there seems to have been a great deal of development time devoted to perfecting the in-camera software and adding more functionality to ML in general

There is a pretty good reason for this, because, that's what the guys do.  The fact that they have developed the features they have, doesn't imply that they must then follow through with fancy GUI's that make the secondary process easier, for yourself.  These Linux guys live off command lines  ;)

In fact, being an open source software, implies collaboration.  So it shouldn't be an expectation of the ML development team to further develop applications, specific to the needs of a few, taking resources away from the main goal of the development process.

Quote
What is it?

Magic Lantern is a free software add-on that runs from the SD/CF card and adds a host of new features to Canon EOS cameras that weren't included from the factory by Canon.

I don't specifically see mention about designing front end software for video editing  ;)




Yes I am being argumentative, but your thread title implies an argumentative response.  You know that just as well as I do, or else the first line of your OP wouldn't have been a clarification!
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 06, 2014, 10:04:12 AM
You post IMHO exemplifies a key misunderstanding and gap between the guys who are coding ML and the guys who are trying to use it.

I have been a huge evangelist for ML raw, even wanting to organize a special presentation for the upcoming NAB and try to sort through much of the mis-information that's floating out there.

But make no mistake, sheer power without end-user friendliness is doomed from the start. So many programmers believe that just implementing amazing code in all that is needed. This is the critical difference that made Steve Jobs so successful, he understood that the end-user experience is as inherent to a product's success as the product itself.

Don't get me wrong, I don't believe in the need to appease the absolute lowest common denominator, but in the case of ML the situation is very much at the opposite end of that spectrum.

Perhaps I don't understand the end-goal here. Maybe ML is some sort of performance-art piece designed to prove that Canon are a bunch of dick-heads who have been keeping all of this power away from the masses for their own perverse reasons. If that's the case, good for ML because it has fully succeeded in achieving that.

But the reality of the situation is that ML exists on borrowed time and in about two years' time nobody will want to shoot video on DSLR's because there will be absolutely no reason to do so (and that's me being optimistic). There are other massive disruptive forces that are causing a rapid shift in how people shoot video. I live and breathe this industry on a daily basis in a town that fancies itself the World capital of film production. I am constantly interacting with cinematographers and editors, and believe me, when I show them a command-line solution they laugh in my face at the absurdity of it. So yes this is only my opinion, but an opinion shared by just about every single professional that I've interacted with for the better part of last year.

On OS X, RAWMagic has been the only solid application that seems to really get it. Unfortunately Thomas is not particularly motivated or available to bring it up to speed with .mlv, which kinda gets us back to the original question:

Is .mlv, in real-world scenarios, worth the extra hassle vs. .raw? Because all of the metadata in the world means nothing if the conversion apps or the ingesting apps don't use it or know what to do with it. And despite a more robust data-writing method, .raw has been rock solid since I first began using it back in June of last year. If anything, this speaks highly of how well the original developers nailed .raw, and why I am not seeing the benefits of .mlv right off the bat.

Lately, Andy has brought up the idea of ACES support in ML raw video. Now that makes much more sense to me, yet none of the developers really chimed into that conversation which leads me to believe that it probably fell on deaf ears.

But I digress....what I would appreciate above all else is if you would be so kind as to explain to me, in plain words, why .mlv makes more sense than .raw with the current state of tools available and its current implementation by third parties?

I appreciate your time and passion, I really do; and I sincerely hope that my posts aren't coming across as too snarky or negative...because in truth if we could hang and discuss this over a couple of drinks I'm sure we'd come to like one another!  8)
Title: Re: Is MLV all that it's cracked up to be?
Post by: Marsu42 on March 06, 2014, 10:16:42 AM
Quote from: Midphase on March 06, 2014, 10:04:12 AM
You post IMHO exemplifies a key misunderstanding and gap between the guys who are coding ML and the guys who are trying to use it.

I'd like to add that you are using nightly or alpha versions to even be able to try these features, there is absolutely nothing done concerning an "end-user" friendliness (yet). Probably due to the lack of people willing to do extremely boring things like writing docs it will remain this way, and there seems to be no schedule for any "stable" 3.0 release since it's more interesting to develop than to bug-hunt.

My point: These are all well-known facts, and while some people might come across a bit nerdy or gear-headed it's not nuclear science to see that yo good ol' Adobe-Apple creative user isn't in the mood to wrestle with command line parameters. It's just that unless more resources pop up out of nowhere, any discussion or lamenting is a bit pointless. Are you volunteering to step up for a gui & usability project?
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 06, 2014, 10:43:00 AM
To further the response from Marsu42, you seem to be expecting ML raw to be some professional solution, with all of the end user support and resources, that are expected from expensive professional software.

Quote from: Marsu42 on March 06, 2014, 10:16:42 AM
Are you volunteering to step up for a gui & usability project?

This!

The developers have enabled the features, it's up to others to take these features in the directions required.

Yes, it is that simple.  ML is a software addon to enable extra functionality in Canon EOS cameras.  No more, no less.
We should all be extremely grateful to g3gg0 for taking the time to make the raw format extensible (MLV), and easy for third party developers to take this to the next level.  Either professionally, with fancy, easy to use GUI's, and end user support, or simple GUI applications designed to provide some aspect of simpleness.

It should not be an expectation, that the developers move from 1 person coding in the garage, to providing GUI's rivalling that of software giants.

You're best bet, would not be discussing the situation here.  Start writing letters to the big players.  They already have the core functionality, and GUI's sorted.  Adding MLV support would be relatively easy in comparison.

Quote from: Midphase on March 06, 2014, 10:04:12 AM
But the reality of the situation is that ML exists on borrowed time and in about two years' time nobody will want to shoot video on DSLR's because there will be absolutely no reason to do so (and that's me being optimistic).

Really?  Borrowed time!  Photographers have been given a bunch of extremely useful features in the last 12 months.  And it only continues to get better.
Title: Re: Is MLV all that it's cracked up to be?
Post by: ItsMeLenny on March 06, 2014, 11:07:59 AM
As a friend of mine used to say "A computer is as only as smart as its user."
Title: Re: Is MLV all that it's cracked up to be?
Post by: Marsu42 on March 06, 2014, 11:13:52 AM
Quote from: Audionut on March 06, 2014, 10:43:00 AM
To further the response from Marsu42, you seem to be expecting ML raw to be some professional solution, with all of the end user support and resources, that are expected from expensive professional software.

The underlying problem is that with the damn expensive 5d3 ML has evolved into a market segment where people have $10000 worth of gear and expect everything to be pro-grade. ML has not and cannot adapt to that, good idea it doesn't support the 1d :->

Back when I started using ML it was a legacy 5d2 project and people were amazed that it even worked on their Rebels, unlocking unbelievable features like more than 3 shots bracketing only found in cameras well out of their financial reach. I was very thankful that after donating for a 60d I got some pre-alpha buggy version working that didn't brick my camera. I know this sounds like grandfather talk, but that's the origin of ML ~3 years ago, everything now is just a nice extra.
Title: Re: Is MLV all that it's cracked up to be?
Post by: ilguercio on March 06, 2014, 12:11:52 PM
Someone blames ML for giving us the most they can and not Canon for NOT giving users what they expect. Weird.
Title: Re: Is MLV all that it's cracked up to be?
Post by: tjaja on March 06, 2014, 12:17:00 PM
Canon is foolproofing their software and the camera has to work with features that not outclass their higher class camera's. Because busniss.
MLV is structured, so it should be easier for Adobe or some company to pick it up. And I think it is just a matter of time.
Title: Re: Is MLV all that it's cracked up to be?
Post by: N/A on March 06, 2014, 01:09:25 PM
Not to mention, .MLV has working audio.

I can see where OP is coming from, it can be frustrating to have all of these great new camera features handed to us only to constantly struggle with a convoluted post workflow. But considering raw video was just a dirty, hiv-ridden crack pipe dream just 11 months ago, we've come a tremendous way.

To be completely generic and cliche, Rome wasn't built in a day.

I moderate the "Share your Video/Photos" section, so I have a pretty good idea of the general type of shoots .RAW and .MLV are being used for. I see a lot of hobbyists and freelance videographers who shoot timelapses, commercials, music videos, etc. You know, the do-it-yourself type of users. We like to have complete control over our projects, and this is what ML gives us. We want the most bang for our buck, which is why most of us don't mind command line tools and different options on how to reach whatever goal we may have in mind for our labors of love.
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 06, 2014, 02:33:30 PM
Its real simple for me... raw if you don't want sound. MLV if you want sound.

Then you have MLrawViewer and you can make it into pro-res without many command line things. Otherwise you could break it out into dngs, I think MLV browser (or mlv-to-cdng) can do it but I only tested that on windows.
Title: Re: Is MLV all that it's cracked up to be?
Post by: dubzeebass on March 06, 2014, 02:43:00 PM
Or the recursive script I and some others wrote, for Mac.
Title: Re: Is MLV all that it's cracked up to be?
Post by: tonybeccar on March 06, 2014, 04:29:12 PM
Hehe.. hot conversation.. it shouldnt be like that. Marsu42, you are right.. and of course you too audionut. But there is a valid point.. we need a proper gui.. because command lines ARE laughable.. well they are very time consuming.. yes!
I know the main os is mac for this business.. but maybe you missed my thread Marsu.. i am developing a converter for windows.. which i am aiming to have everything and be thr ultimate solution.. its looking good.. and also chmee is working on cdngs.. which is thr way to go.. so my guess is in 2 months ML would be close to (or co.pletely) perfect. I am on holiday returning in a few days and will resume development on my app.. which i have good ideas for.
But as audionut said and its very true.. its up to collaboration to make the back end happen.. well it IS happening and soon it will be ready..
But yeah ML will be always for small budget productions who want to get close to really professional cameras.. so i dont mind myself a little dirt on the road :)
Anyway id love to compile my converter for mac.. i just dont know how to and currently have no mac to test it on.. will do soon when i resume work.though..
Anyway hope i brought some peace to both sides hehe..
Cheers!!
Title: Re: Is MLV all that it's cracked up to be?
Post by: reddeercity on March 06, 2014, 04:34:35 PM
@Midphase, I have same thoughts exactly! You have very good points that No one will
Address , without being defensive !
All you ever hear is the same BS over & over
How ML is open soure and if you don't like it do it your self etc....
And then the same people will jump all over you when make a suggestion
Or a comment toward the Delveopment of Raw/MLV. To me it seems
There is no direction and if there is , it's known to a FEW Select
People .
And there agian I know everyone will take may comment
Negatively , but it must be said.
I'm absolutely a very big supporter or the ML , if I wasn't
I wouldn't be posting my concerns and be active on the form.
I will have more to say on this topic later on. :)
Title: Re: Is MLV all that it's cracked up to be?
Post by: Marsu42 on March 06, 2014, 04:54:13 PM
Quote from: reddeercity on March 06, 2014, 04:34:35 PM
All you ever hear is the same BS over & over
How ML is open soure and if you don't like it do it your self etc....

If you really call this bs and not just for trolling purposes, I think you missed out big on the whole idea. That's why I came to code ML patches, because I wanted to change something w/o begging the devs for every single implementation. Now that the core is cleaner and modules are here, it's even much easier than before - and if you're stuck, you're getting near-instant help around here.

Quote from: reddeercity on March 06, 2014, 04:34:35 PM
And then the same people will jump all over you when make a suggestion
Or a comment toward the Delveopment of Raw/MLV.

I guess it depends on the tone. As written above I'd agree there are some less-than-practical photogs around who might not have optimal experience with actual real-life implementation of the ML features for production purposes. But it's up to your social skills to deliver the intended message so it's well received ... reading your post above, you could do with some practice :-p
Title: Re: Is MLV all that it's cracked up to be?
Post by: reddeercity on March 06, 2014, 05:01:21 PM
Sorry you  see it that way, I couldn't help but Tell the truth.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Marsu42 on March 06, 2014, 05:33:55 PM
Quote from: reddeercity on March 06, 2014, 05:01:21 PM
Sorry you  see it that way, I couldn't help but Tell the truth.

Ok, if your opinion from your perspective is the same as the truth imho that's plain trolling - thank you for notifying so no further energy is wasted.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 06, 2014, 05:53:29 PM
"Its real simple for me... raw if you don't want sound. MLV if you want sound."

Thank you 1% for being pretty much the only one to answer my question. With that basic bit of info, I think I'll stick with .raw.

To the rest of the people who feigned outrage...I think you're missing the point I was trying to make and which I have neither the time, nor the inclination to re-explain (given the defensive posture that just about everyone immediately took).

Thank you developers for the hard work.
Title: Re: Is MLV all that it's cracked up to be?
Post by: mixmastermike on March 06, 2014, 08:42:10 PM
Quote from: reddeercity on March 06, 2014, 04:34:35 PM
@Midphase, I have same thoughts exactly! You have very good points that No one will
Address , without being defensive !
All you ever hear is the same BS over & over
How ML is open soure and if you don't like it do it your self etc....
And then the same people will jump all over you when make a suggestion
Or a comment toward the Delveopment of Raw/MLV. To me it seems
There is no direction and if there is , it's known to a FEW Select
People .
And there agian I know everyone will take may comment
Negatively , but it must be said.
I'm absolutely a very big supporter or the ML , if I wasn't
I wouldn't be posting my concerns and be active on the form.
I will have more to say on this topic later on. :)

Well said, I don't think it is unreasonable to want MLV to function with sound or at 30p.

But at the same time I understand it's not me doing the coding either, so catch 22 on that one.

If this is open source does that mean there is no structural development plan? Or is it even unreasonable to ask that question?

The aggressive nature of posters taking this or that stance reminds me of one of my favorite Einstein quotes

"Great spirits have always encountered violent opposition from mediocre minds"

And here's one free from me;

"If you find yourself arguing with an idiot, you have to ask who the idiot is"



Title: Re: Is MLV all that it's cracked up to be?
Post by: Marsu42 on March 07, 2014, 01:32:33 AM
Quote from: mixmastermike on March 06, 2014, 08:42:10 PM
If this is open source does that mean there is no structural development plan? Or is it even unreasonable to ask that question?

Of course not, what gave you the idea? Personally, I keep mentioning that the current situation cannot be satisfactory to end users as they are forced to used the pre-alpha nightly bleeding edge releases as a kind of "new stable", there are no milestones or point releases except for specific camera models then and again. Imho needs some predictable release management to de-confuse the situation.

At the same time, people don't seem to realize that the bulk and tricky part of ML is essentially coded by one person, maybe with specific additions from another coder. How much of a structural development plan does a single person need? I'm not a dev and thus don't have any insight in what's going on betwen them, but it boils down to everyone is doing what he likes or wants, and this about what you can expect to get for free.
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 07, 2014, 03:13:33 AM
AFAIK a1ex + g3gg0 are mainly into photography vs video production. If you need some sort of milestones to mesh ml RAW with your workflow there needs to be some sort of discussion as to what is exactly lacking.. and not unreasonable stuff like 4K filming and in camera pro-res. So far my list includes inconsistent HDMI and inability to preview full crop mode while recording without major speed hits or footage destruction. Also the FPN noise + hot pixels at high ISO.

The post worflow is getting worked out since there are a few competing solutions, the ML raw viewer is really a godsend in terms of realtime preview and one click conversion.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 07, 2014, 06:08:03 AM
I agree with you 1%, MLRawViewer is a step in the right direction with someone who is willing to listen to feedback and not get all up in arms when someone asks for needed improvements.

MLVMystic is also promising, but it's still ways away from where it needs to be. It's currently very slow, lacks any sort of progress bar (which in turn makes it feel slower than it probably really is), lacks the ability to select directories, lacks drag and drop functionality, doesn't export to CDNG which is (in no small part due to Blackmagic) gaining more and more acceptance and support from apps like Premiere Pro, would be nice it could export to other formats like ProRes (yeah, I know MLRawViewer does that already), and some sort of preview would be cool too. In essence what I am asking is for an app that's a combination of MLRawViewer and MLVMystic (with a dash of RAWMagic thrown in).

My point about ML raw (and DSLR video) having a limited shelf life still stands. Today's test with about every camera in existence (including the GH4 and the Blackmagic 4K) supported the fact that the landscape is changing quite rapidly. As a fan of ML raw, I want it to still be a relevant and useful format for as long as possible. But in reality there really isn't too much time to screw around before the limited momentum that ML raw has achieved grinds to a halt due to a difficult post production data pipeline.

I realize that this is all free stuff, done as an open source project. I totally get it. But I think that the ML community should be open to having a serious conversation about what is really needed to solidify this software.

As much as hacking cameras is a fun puzzle box that I am sure piques the fancy of the developers, I wish they'd stop focusing so much on adding new features, and put more time into fixing the elements that need fixing (all the things that 1% just mentioned for instance) as well as strengthening the back end tools.



Title: Re: Is MLV all that it's cracked up to be?
Post by: Markus on March 07, 2014, 06:43:54 AM
Quote from: 1% on March 06, 2014, 02:33:30 PM
Its real simple for me... raw if you don't want sound. MLV if you want sound.

Then you have MLrawViewer and you can make it into pro-res without many command line things. Otherwise you could break it out into dngs, I think MLV browser (or mlv-to-cdng) can do it but I only tested that on windows.

Is a working sound module for standard raw rec impossible or just not something anyone is interested in putting time in developing?
Title: Re: Is MLV all that it's cracked up to be?
Post by: budafilms on March 07, 2014, 06:46:06 AM
Have someone two 5D Mark III to put one in RAW and the other MLV at same time, equal shoot, same lens, etc?

In my case, if .RAW have sound, it's the same.

I did a Full Movie - about 4 month - in RAW without sound reference and it's a pain...

but it's true .RAW it's more stable when you shoot 8 hours a day.

I'ts my experience!
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 07, 2014, 06:51:43 AM
Quote from: Midphase on March 07, 2014, 06:08:03 AM
I wish they'd

Ah, wishful thinking.  I wish I had $100,000,000.
Wishing, and wanting, and expecting, doesn't make things happen.

I can assure you, that adding MLV support to existing software, is magnitudes easier then developing an entire NLE.

Have you started writing letters to Adobe et all, yet?
Title: Re: Is MLV all that it's cracked up to be?
Post by: andy kh on March 07, 2014, 06:55:03 AM
@budafilms yeah its a pain in the ass without sound reference so i never shoot raw and was never interested. for making films i prefer mlv since i can sync up audio which i recorded with my external recorder and with the reference audio i can do ADR.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Markus on March 07, 2014, 06:57:46 AM
I also find the question in this thread relevant. The following argument reminds me of a reaction that is all too familiar for one who has been in a lot of feedback situations making/creating films and videos. It's not easy to be open when you worked your ass off on something in a particular direction. But with time you learn you have a lot to gain if you can be open and listen to the curses in the church. 

By the way, the mlv module seem to be getting really good lately but the robustness, speed, instant start, and simplicity of the standard raw format is a set of powerful strengths!
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 07, 2014, 08:06:43 AM
Quote from: Audionut on March 07, 2014, 06:51:43 AM
Ah, wishful thinking.  I wish I had $100,000,000.
Wishing, and wanting, and expecting, doesn't make things happen.

I can assure you, that adding MLV support to existing software, is magnitudes easier then developing an entire NLE.

Have you started writing letters to Adobe et all, yet?

Why the attitude? Is it so wrong to communicate what I think are shortcomings of ML raw or some of the apps? Why shouldn't you be interested, even happy to get useful feedback from guys who are trying to use these tools in real-world situations?

Perhaps something got lost in translation, but I never asked for ML to develop its own NLE. I merely asked for ML to develop a more user friendly way to get the data from the camera into an NLE. Without a solid back-end, all of the wonderful work that is being done on the in-camera raw video is kinda moot.

P.S.

Yes, I have repeatedly put in requests to both Adobe and Resolve for .raw and .mlv support.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 07, 2014, 08:11:52 AM
Quote from: Midphase on March 07, 2014, 08:06:43 AM
even happy to get useful feedback from guys who are trying to use these tools in real-world situations?

I must have missed the useful feedback.  Care to provide a link?


Quote from: Midphase on March 07, 2014, 08:06:43 AM
Without a solid back-end, all of the wonderful work that is being done on the in-camera raw video is kinda moot.

In your own opinion, or the opinion shared by few.  Just because Midphase has a problem with CMD lines, doesn't make the current implementation, moot!

Dispense with the over exaggerations please.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 07, 2014, 08:24:46 AM
I take back what I said about you and I sharing a drink and getting along. You sound like a really angry person with a very thin tolerance for anyone who does not agree with you.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 07, 2014, 08:31:51 AM
So, no link then?

No need to get personal.  Discuss the matter in a manner that doesn't rely on over exaggerations.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 07, 2014, 08:48:02 AM
If you have nothing useful to contribute to this thread, could you please stop wasting your time posting on it? I'm sure there are tons more things out there that annoy the crap out of you that you could be tending to.

Regarding feedback:

I mentioned that Thomas' RAWMagic nailed a good solution. Unfortunately no such app exists with support for .mlv. Command-line conversion tools are simply not a real-world way to manage this data. Particularly not on a set. If command-line was such a great way to get things done, then we'd all be still using MS-DOS.

I mentioned that MLVMystic is on the right track, but I also outlined its limitations and what should be addressed to make it a robust option. I also mentioned that none of the current apps for the Mac output CDNG from .mlv, I think this is an omission that should be corrected as Adobe is more likely to support CDNG than DNG in the future.

If none of this sounds like useful feedback, please refer to the first paragraph of this post and kindly oblige with my request.
Title: Re: Is MLV all that it's cracked up to be?
Post by: a1ex on March 07, 2014, 02:04:57 PM
Post-processing:

These post-processing tools are not developed by ML, other than some reference implementations (raw2dng/mlv_dump), which are command-line tools to keep them simple.

The programs you are talking about are developed by other authors (not by ML devs), so their authors will most likely welcome your feedback.

Recording modules

Judging from user feedback, my impression is that raw_rec is pretty much stable (except for some small quirks), but minimalist (no sound, no metadata), and mlv_rec is still work in progress, has the big advantage of audio recording, but write speeds are a bit inconsistent, and being quite a bit more complex, there are more things likely to break. However, it's designed to be extensible (friendly for future development), which can't be said about the RAW format.

An example regarding extensibility: suppose you may want to do some FPN correction, and for that you need to record optical black data (either complete or summarized somehow) or some sort of bias or dark frames. With the RAW format, you have no other choice but creating external sidecar files. With MLV, you can simply add this optical black data to MLV spec and implement it. The converters that don't know about this new feature will simply ignore it gracefully (this is allowed by the design of MLV format); the ones that do will use this correction data and output better images.

Once mlv_rec will be better in all aspects, raw_rec will most likely go away. For now, my impression is that it isn't there yet, so I still keep the old raw_rec in the nightly builds.

The cause of pink frames is not yet fully understood, and recent versions of mlv_rec were reported to have fewer pink frames at least on 550D. On other cameras, raw_rec may be better, I don't know.

If you report things like one module seems to be more solid during long usage, a useful feedback should include some statistics. For example, say you've recorded a bunch of footage at settings XYZ, and got 15 pink frames out of 12345. Then, say you've changed setting Y and got 30 pink frames out of 1234. This feedback is much better than vague reports about intermittent failures, but providing a way to reproduce a bug is even better.

I use the camera primarily for photos, so I didn't actually use the raw recorders in the last few months.

Quote from: Markus on March 07, 2014, 06:43:54 AM
Is a working sound module for standard raw rec impossible or just not something anyone is interested in putting time in developing?

Probably quite easy, but since the MLV file format is extensible, I don't expect further development on raw_rec (other than maybe some basic maintenance), but anybody is free to give it a try.

I don't exclude a lite version of mlv_rec at some point in the future, rewritten with performance and robustness in mind, rather than feature-completeness. But I'm confident that MLV as file format is here to stay.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Markus on March 07, 2014, 04:07:30 PM
Quote from: a1ex on March 07, 2014, 02:04:57 PM
Post-processing:

Probably quite easy, but since the MLV file format is extensible, I don't expect further development on raw_rec (other than maybe some basic maintenance), but anybody is free to give it a try.

I don't exclude a lite version of mlv_rec at some point in the future, rewritten with performance and robustness in mind, rather than feature-completeness. But I'm confident that MLV as file format is here to stay.

Thanks for a good and extensive answer. Interesting as you describe future capabilities of mlv and I also think it is here to stay. Actually I don't really think anyone here doubts that. However a snd module for raw-rec would be a great tool in the toolbox. Would be greatfull to anyone who would pick that up. It would give us a complete stable solution whilst mlv development continues.
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 07, 2014, 05:07:10 PM
raw_rec with sound would need a bunch of changes, I don't think its as simple as adding the hooks from mlv_snd... mlv marks buffers specifically for sound and records a different way. if you add the mlv writing/buffering methods to raw_rec then you'll have mlv_rec and might lose the speed advantage.

Title: Re: Is MLV all that it's cracked up to be?
Post by: mixmastermike on March 07, 2014, 05:29:59 PM
Quote from: a1ex on March 07, 2014, 02:04:57 PM
Post-processing:

These post-processing tools are not developed by ML, other than some reference implementations (raw2dng/mlv_dump), which are command-line tools to keep them simple.

The programs you are talking about are developed by other authors (not by ML devs), so their authors will most likely welcome your feedback.

Recording modules

Judging from user feedback, my impression is that raw_rec is pretty much stable (except for some small quirks), but minimalist (no sound, no metadata), and mlv_rec is still work in progress, has the big advantage of audio recording, but write speeds are a bit inconsistent, and being quite a bit more complex, there are more things likely to break. However, it's designed to be extensible (friendly for future development), which can't be said about the RAW format.

An example regarding extensibility: suppose you may want to do some FPN correction, and for that you need to record optical black data (either complete or summarized somehow) or some sort of bias or dark frames. With the RAW format, you have no other choice but creating external sidecar files. With MLV, you can simply add this optical black data to MLV spec and implement it. The converters that don't know about this new feature will simply ignore it gracefully (this is allowed by the design of MLV format); the ones that do will use this correction data and output better images.

Once mlv_rec will be better in all aspects, raw_rec will most likely go away. For now, my impression is that it isn't there yet, so I still keep the old raw_rec in the nightly builds.

The cause of pink frames is not yet fully understood, and recent versions of mlv_rec were reported to have fewer pink frames at least on 550D. On other cameras, raw_rec may be better, I don't know.

If you report things like one module seems to be more solid during long usage, a useful feedback should include some statistics. For example, say you've recorded a bunch of footage at settings XYZ, and got 15 pink frames out of 12345. Then, say you've changed setting Y and got 30 pink frames out of 1234. This feedback is much better than vague reports about intermittent failures, but providing a way to reproduce a bug is even better.

I use the camera primarily for photos, so I didn't actually use the raw recorders in the last few months.

Probably quite easy, but since the MLV file format is extensible, I don't expect further development on raw_rec (other than maybe some basic maintenance), but anybody is free to give it a try.

I don't exclude a lite version of mlv_rec at some point in the future, rewritten with performance and robustness in mind, rather than feature-completeness. But I'm confident that MLV as file format is here to stay.

First of all thanks for Devs getting involved in this thread , there are answers that I would like to pose.

RAW_REC to me seems pretty stable too, however for the application I need I need sound as I only have 2 people involved in the project I would love to record in RAW.

MLV_REC is conceptually great, however the recording speeds/buffer performance is very up and down. Im working on a cooking show for web distribution 24p is not the look I need at all. the motion blur is too filmic, 30P would be ideal. If card spanning can return to full HD 30p with MLV and MLV_SND I will be a very happy man. I will also feel that ML has achieved a a fully fictional raw video camera with sound, a real video camera so to speak, a complete functional unit to rival video cameras. In post this means I can use Pluraleyes to auto sync external professional sound, reducing post processing time.

Is it too early to say if 30P full HD will be possible?

Is there any way you could explain what is going on with the write speeds to a laymen, Im big on solving problems but just unfortunately not coding ones. I built a shoulder rig to film the show and be able to start and stop recording, and audio recording all while the camera is on my shoulder. I built the rig with the fact a 5D mkiii might not acutaly be the camera to do this project, so the baseplate can be changed and as such the camera. However the canon sensor is amazing and produces superb skin tones and colors in my opinion much better that say a Gh3 or Black Magic (Subjective I know, lets not start another fight here)

I would just love to see this project done on MLV_REC and MLV_SND at 30P does anyone think that is possible in the future?

(//)

Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 07, 2014, 07:08:39 PM
Thank you a1ex for your very informative reply. Information from posts like yours and 1%'s goes so much further in clearing up some confusion that (at least for me) has crept up about the usefulness of switching  to .mlv


Title: Re: Is MLV all that it's cracked up to be?
Post by: Pierluca72 on March 08, 2014, 12:13:24 AM
My first post here... First of all thank you guy for the amazing work. Actually I love .mlv on Mac. Workflow isn't difficult at all. I created an automator workflow using the dump and command.. one click and I have my dng and wav. No preview on the mac? yes there is one, is Davinci lite (free). It can play in real time (almost) 22fps the dng files. However there is one thing that I'm struggling with and I would really appreciate some hints here. The audio track is slightly longer than the video track. Am I doing something wrong? I use the same fps settings in davinci as the clip. Thank you in advance
Title: Re: Is MLV all that it's cracked up to be?
Post by: Pierluca72 on March 08, 2014, 12:20:13 AM
Quote from: Pierluca72 on March 08, 2014, 12:13:24 AM
No preview on the mac? yes there is one, is Davinci lite (free).

I just realized that previewing a dng file is not the same as previewing a .mlv. I still have to convert to dng first so.. I take that back.. But still please a hint for the sync
Title: Re: Is MLV all that it's cracked up to be?
Post by: Frank7D on March 08, 2014, 12:43:33 AM
Quote from: Pierluca72 on March 08, 2014, 12:13:24 AM
The audio track is slightly longer than the video track. Am I doing something wrong? I use the same fps settings in davinci as the clip. Thank you in advance

All mine are that way too, yet they are in sync nonetheless.
Title: Re: Is MLV all that it's cracked up to be?
Post by: daisermac on March 11, 2014, 11:48:44 AM
MLV works perfectly for me, using nightly build Febr. 15th on 5Dm3. No card spanning, good old K-Bay 64GB 1000x, recording 1080p25. Not a single corrupted frame since then (had a lot with the first builds from december). I am even using HDMI, which is said to have bad effects on write speed. One in ten recordings stops because of slow speed, but I can live with that. Camera hangs if I switch of HDMI before waiting for camera shutdown - I can live with that too. Sound is a bless. MetaData (e.g. for automatic distortion correction) is a bless. MLRawViewer is a great software. Conversion with the shell script works without problems.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Pierluca72 on March 11, 2014, 03:56:08 PM
Quote from: Frank7D on March 08, 2014, 12:43:33 AM
All mine are that way too, yet they are in sync nonetheless.
OK thanks. They are in sync actually. I just have to trim off the audio track so that it matches to the video. Has anyone found a quick way of doing this when you work with many clips?
Title: Re: Is MLV all that it's cracked up to be?
Post by: ted ramasola on March 11, 2014, 04:22:41 PM
In resolve, once you have dragged all the dngs and wav files into the media pool, right click the root and right click auto sync based on time code. in edit choose new timeline unselect create empty timeline. all dngs will be linked and trimmed. Though in cases of longer clips, the wav seemed to show no waveform halfway, I manually linked those problematic clips only.
Title: Re: Is MLV all that it's cracked up to be?
Post by: andy kh on March 11, 2014, 05:55:46 PM
Quote from: Pierluca72 on March 11, 2014, 03:56:08 PM
OK thanks. They are in sync actually. I just have to trim off the audio track so that it matches to the video. Has anyone found a quick way of doing this when you work with many clips?

i will tell you the best way to deal with so many clips if you are using windows. simply convert to cdng using raw2cdng and import all those clips in premiere pro and wala..all your audio and video clips are in sync. u neednt do any manual sync
Title: Re: Is MLV all that it's cracked up to be?
Post by: budafilms on March 12, 2014, 02:14:17 AM
Quote from: ted ramasola on March 11, 2014, 04:22:41 PM
In resolve, once you have dragged all the dngs and wav files into the media pool, right click the root and right click auto sync based on time code. in edit choose new timeline unselect create empty timeline. all dngs will be linked and trimmed. Though in cases of longer clips, the wav seemed to show no waveform halfway, I manually linked those problematic clips only.

Hi Ted, I was trying that, and no result!
Have I put an specific module for time cod? or activate something?
Title: Re: Is MLV all that it's cracked up to be?
Post by: ted ramasola on March 12, 2014, 03:55:20 AM
It must be how the dngs were extracted. I'm using tonybeccar's windows based extractor. http://www.magiclantern.fm/forum/index.php?topic=10198.msg98334;topicseen#msg98334

Current one is buggy, someone posted a link to an older 1.6 version  that is working for me. :http://www.samuel-maurin.com/MLV-Converter-1.6.rar

What dng converter/extraction are you using?
Title: Re: Is MLV all that it's cracked up to be?
Post by: budafilms on March 12, 2014, 04:33:58 AM
Quote from: ted ramasola on March 12, 2014, 03:55:20 AM
It must be how the dngs were extracted. I'm using tonybeccar's windows based extractor. http://www.magiclantern.fm/forum/index.php?topic=10198.msg98334;topicseen#msg98334

Current one is buggy, someone posted a link to an older 1.6 version  that is working for me. :http://www.samuel-maurin.com/MLV-Converter-1.6.rar

What dng converter/extraction are you using?

I use Apple!
I open RAW file with RawMagic or MLV with Mystic.
After that I import into DaVinci.
And the edition with Final X.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Sganzerla on March 12, 2014, 07:04:03 AM
I'm one guy who have done 8 videos using RAW video with my MKIII and I have the same concerns as Midphase.

Still don't know if I should stick with RAW for now, 75% of me say yes.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Pierluca72 on March 12, 2014, 03:48:13 PM
Quote from: andy kh on March 11, 2014, 05:55:46 PM
i will tell you the best way to deal with so many clips if you are using windows. simply convert to cdng using raw2cdng and import all those clips in premiere pro and wala..all your audio and video clips are in sync. u neednt do any manual sync
Wow thank you so much!
Title: Re: Is MLV all that it's cracked up to be?
Post by: Pierluca72 on March 12, 2014, 03:49:22 PM
Quote from: ted ramasola on March 11, 2014, 04:22:41 PM
In resolve, once you have dragged all the dngs and wav files into the media pool, right click the root and right click auto sync based on time code. in edit choose new timeline unselect create empty timeline. all dngs will be linked and trimmed. Though in cases of longer clips, the wav seemed to show no waveform halfway, I manually linked those problematic clips only.
Thank you so much
Title: Re: Is MLV all that it's cracked up to be?
Post by: budafilms on March 12, 2014, 04:19:58 PM
Quote from: Sganzerla on March 12, 2014, 07:04:03 AM
I'm one guy who have done 8 videos using RAW video with my MKIII and I have the same concerns as Midphase.

Still don't know if I should stick with RAW for now, 75% of me say yes.

The Beep for external recorder in .RAW it's not bad...
Title: Re: Is MLV all that it's cracked up to be?
Post by: g3gg0 on March 12, 2014, 10:28:25 PM
Quote from: Sganzerla on March 12, 2014, 07:04:03 AM
Still don't know if I should stick with RAW for now, 75% of me say yes.

well, it is free open source software.
just try it. no need to spend 1k$ and realize its crap.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Sganzerla on March 13, 2014, 11:46:03 PM
Yeah, I'm using sound beep which is cool, but is not going to be very welcome in the next wedding I'm going to record, for example. I may very well end up using MLV instead of RAW with a lower volume bip (and the risk of having problems with audio sync).

I prefer PluralEyes sync with MLV instead of getting 100 takes and doing by hand with RAW, and the metadata is a good addiction as my lenses are color calibrated with DNG Profile Editor and I'm using ACR for CC. But I prefer much more the software batch/workflow that I'm already using compared to what is possible with MLV for now. In huge projects this can be a serious problem.

I know there are questions without answers yet, as software implementation has nothing to do with the ML RAW Team, but as I said before, I can really understand the questions raised by Midphase with this topic.
Title: Re: Is MLV all that it's cracked up to be?
Post by: eatstoomuchjam on March 19, 2014, 02:23:53 AM
Quote from: Midphase on March 06, 2014, 10:04:12 AM
You post IMHO exemplifies a key misunderstanding and gap between the guys who are coding ML and the guys who are trying to use it.

Speaking as a guy who uses ML and doesn't code it, your posts seem to exemplify a key misunderstanding between someone who thinks he is buying a product from a commercial entity who needs his money to continue operating and someone who realizes that other people have tinkered with something in their spare time and who, out of kindness and a desire to collaborate with others, are now allowing him to download the results of their tinkering.  If you don't evangelize or download anything from ML, the project will still do just fine.

Quote from: Midphase on March 06, 2014, 10:04:12 AM
But make no mistake, sheer power without end-user friendliness is doomed from the start. So many programmers believe that just implementing amazing code in all that is needed. This is the critical difference that made Steve Jobs so successful, he understood that the end-user experience is as inherent to a product's success as the product itself.
Don't get me wrong, I don't believe in the need to appease the absolute lowest common denominator, but in the case of ML the situation is very much at the opposite end of that spectrum.

I'll help you with a more apt comparison.
Many years ago, a group of people developed an OS kernel known as Mach.  It was a microkernel which attempted to move big chunks of the kernel into user space (among other things, but that'll be a lot of reading - if you're interested, wikipedia has a nice write-up http://en.wikipedia.org/wiki/Mach_(kernel)#Mach_concepts).  It was designed as a drop-in replacement for the traditional Unix kernel.
When Steve Jobs left Apple for NeXT, they needed an operating system and after looking around for a bit, they settled on using a Mach kernel with a userland heavily based on BSD, a former academic OS derived from AT&T unix which was developed largely, as the name would imply, at Berkeley, with descendants still available today (FreeBSD, NetBSD, OpenBSD, etc).  At NeXT, they developed a number of graphical abstractions on top of Unix.
When Apple acquired NeXT to bring Steve Jobs back, he brought NeXTStep with him and it was retooled to become one of the early versions of OSX.

In this example, Magic Lantern would probably be most similar to the BSD userland or Mach microkernel.  They're powerful tools in the right hands and thanks to their inclusion in OSX, they are running on most desktop computers in the world.

BSD was not built as a commercial product.  It was built by hackers who wanted something to make their own lives better and who were kind enough to share it with others to make their lives better too.  Thanks to the work of those wonderful hippie nerds in the 70's, you may enjoy a Mac today which isn't running a derivative of OS9.

Quote from: Midphase on March 06, 2014, 10:04:12 AM
But the reality of the situation is that ML exists on borrowed time and in about two years' time nobody will want to shoot video on DSLR's because there will be absolutely no reason to do so (and that's me being optimistic). There are other massive disruptive forces that are causing a rapid shift in how people shoot video. I live and breathe this industry on a daily basis in a town that fancies itself the World capital of film production. I am constantly interacting with cinematographers and editors, and believe me, when I show them a command-line solution they laugh in my face at the absurdity of it. So yes this is only my opinion, but an opinion shared by just about every single professional that I've interacted with for the better part of last year.

Two years, huh?  Looking at the progress that has been made by Canon with their cameras in the last 2 years, I can TOTALLY see why you'd extrapolate that in 2 more years, they'd be obsoleting the current crop.  They just released... what, like the 5th consumer camera in a row using almost exactly the same body as the last 4 and pretty much the same sensor as the 7D had when it was released in 2009?  Their least expensive raw-capable video camera is in the 5 digits price range now and 2 years ago, it was in the... 5 digits range.  Every camera being released by a major vendor now is still using some form of h.264/avchd/mp4 video, the same as 2 years ago.  Canon's already mentioned how many new camera bodies they're releasing in 2014.  It's not a large number.  So...  whose cameras, exactly, do you think are going to unseat the major vendors?  Black Magic are small and feature-poor still.  Digital Bolex finally shipped something, but their longer-term prospects seem iffy at best.

Quote from: Midphase on March 06, 2014, 10:04:12 AM
I appreciate your time and passion, I really do; and I sincerely hope that my posts aren't coming across as too snarky or negative...because in truth if we could hang and discuss this over a couple of drinks I'm sure we'd come to like one another!  8)

Sure.  I'm sure the developers would love to sit down for drinks with some whiny Internet user who rants at length about everything he thinks they're doing wrong and how difficult he finds their software.  Who wouldn't want that?
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 19, 2014, 05:34:36 AM
QuoteBut the reality of the situation is that ML exists on borrowed time and in about two years' time nobody will want to shoot video on DSLR's

I'm more in line with the thinking that you'll be able to rent or even buy something that shoots 4K, albeit in 4:2:2 H264/etc. Rather than deal with the huge file size and complicated workflow people can just go for that option. So a production with the extra hassle of a DSLR raw might not be so enticing and the compressed won't compare. There are always the interchangeable lenses and the tool that you have is better than the one you don't.

Oh and well the whole photography thing.

Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 19, 2014, 07:21:12 AM
Quote from: 1% on March 19, 2014, 05:34:36 AM
I'm more in line with the thinking that you'll be able to rent or even buy something that shoots 4K, albeit in 4:2:2 H264/etc. Rather than deal with the huge file size and complicated workflow people can just go for that option. So a production with the extra hassle of a DSLR raw might not be so enticing and the compressed won't compare. There are always the interchangeable lenses and the tool that you have is better than the one you don't.

Oh and well the whole photography thing.

I agree, but just to clarify I was strictly speaking about production uses. My point of view is from someone who shoots and works on film productions for a living and what I'm seeing is a huge interest in raw video capture coming from both production and post production sides. Most of the color houses in town don't want to deal with compressed footage anymore. The hardware and storage are up to speed to where they need to be. Oh, and 4K is here to stay.

My point is that DSLR shooting for professional production arose as a bit of a fluke with the 5D2, it's had a useful lifespan; but ultimately it was a workaround for the lack of better (or at least affordable) capture hardware. Blackmagic has broken the price barrier for a sub $5k super-35 camera capable of producing 4K footage. It's not perfect, but it has a strong post production pipeline that is just going to keep getting better. NAB is a couple of weeks away and I'm sure some major announcements will be made from Blackmagic and other manufacturers. Interacting with several professionals it's clear that the writing is on the wall as far as DSLRs for film production is concerned. This is neither good nor bad, it's just the natural progression and evolution of technology.
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 19, 2014, 04:36:58 PM
There is also the matter of nobody knowing DSLRs shoot raw or thinking its 5DIII only. But then eyes glaze over from the C100 and its digitally pushed ISO 80k. The premiere CC neutering isn't helping things either, don't know how good support is in FCP. The price barrier for raw is actually pretty low now, a 50D is much cheaper than BMCC. As a primary camera though DSLRs are pretty sketchy in a larger production.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 19, 2014, 05:32:22 PM
Yeah. Well, for me the 5D3 has been the one camera that could be fit for production. Recent developments with MLRawViewer and another product that I'm not quite at liberty to talk about; coupled with the Blackmagic's 4K's lackluster low-light performance could breathe new life into if for pro uses. As a matter of fact tomorrow I'm the ML raw consultant/DIT on a music video that's being shot on the 5D3 with ML raw (nighttime shoot, low light setups).

Just yesterday I was discussing with someone in the industry how cool it would be if ML could be ported to the 1Dx or the C100 (the C100 in particular would be very appealing since many DP's prefer its ergonomics and stronger production features).
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 19, 2014, 07:09:01 PM
I've seen some kick ass 50D/7D stuff on here and they are sub $1000. They easily do anamorphic 2k. People that already have 5DIIs can get new use out of them too. ML-raw does lower the cost of entry quite a bit if people actually knew about it. 1Dx/C100/etc are all in a similar price range, 5DIII and BM are slightly cheaper only.

Renting red or arri is what? $700-1000 a day? If you have a budget and already paying for a union crew + just want to get done....

Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 19, 2014, 08:19:34 PM
Anamorphic isn't really a good solution since these cameras can't be adapted to PL mount lenses (it's probably ok anyway, anamorphic cinema lenses like the Angenieux line can easily cost or rent as much as an Alexa per day or more). I had high hopes for anamorphic shooting, but ultimately I had to give up due to far too many impractical issues (particularly when done with projection lenses).

I mentioned before that part of the problem with ML raw (particularly on the 5D3 which yields a full HD resolution with low moire, high sensor ISO, and robust media) is that it's a little too good. What I mean is that the quality really is remarkable and it puts it in the class of productions that originally made the 5D2 shooting so attractive (i.e. independent features with not quite enough budget to afford a $20k or more camera rental package). It also makes it a very attractive b-camera on higher end productions, and certainly a good camera to be mounted in tight spaces or in situations where the camera might be disposable (i.e. dropping it off a building).

There are undoubtedly many impressive videos out there done even with low end ML raw cameras. But there is a big difference between a leisure/week-end warrior shoot and a pro or semi-pro shoot. I think ML raw is so close, if the post pipeline was only stronger (similar to the way RED did it...I know I know, billion dollar company...I know). This lack of an efficient post pipeline is IMHO the reason why industry magazines and higher visibility cinematographers like Phil Bloom and Shane Hurlbut haven't embraced this remarkable tool.

To paraphrase Marlon Brando "It coulda been a contender."    ;D
Title: Re: Is MLV all that it's cracked up to be?
Post by: g3gg0 on March 19, 2014, 08:53:47 PM
Quote from: Midphase on March 19, 2014, 08:19:34 PMI think ML raw is so close, if the post pipeline was only stronger (similar to the way RED did it...I know I know, billion dollar company...I know). This lack of an efficient post pipeline is IMHO the reason why industry magazines and higher visibility cinematographers like Phil Bloom and Shane Hurlbut haven't embraced this remarkable tool.

then dont talk about what is missing, but take a development environment and put the missing pieces into the puzzle.
Title: Re: Is MLV all that it's cracked up to be?
Post by: 1% on March 19, 2014, 08:56:29 PM
QuoteBut there is a big difference between a leisure/week-end warrior shoot and a pro or semi-pro shoot.

Yep... but not everyone who shot with the other CF cams was a weekend warrior as you imply. There were and are people using DSLRs even in H264 to crank things out and getting paid. 5DIII isn't the be-all end all. Who would chose a $4k camera to destroy when they can just wreck a $400 50D... both do 1080P.

Quoteif the post pipeline was only stronger

This is the time for the community to shine too. I can see not everyone being into camera fw hacking but I'm sure there are quite a few coders who can do windows/mac and image manipulation. Its a fairly different skill set. Red had to fix the workflow or they wouldn't sell cameras. Here you already have people who bought the hardware and are just getting a very nice extra for free.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 19, 2014, 10:17:13 PM
Quote from: 1% on March 19, 2014, 08:56:29 PM
This is the time for the community to shine too. I can see not everyone being into camera fw hacking but I'm sure there are quite a few coders who can do windows/mac and image manipulation. Its a fairly different skill set.

Agreed, that's why I see promise with tools like MLRawViewer and the other aforementioned app that I shouldn't discuss.

Title: Re: Is MLV all that it's cracked up to be?
Post by: Kharak on March 21, 2014, 04:45:32 PM
MLRawViewer is the biggest step since RAW.

Being on shoots and not being able to watch the footage was embarrassing and a great cause of concern.

The post process is hard, but thats the price we pay for the quality of RAW. In the long run, if post process will not be improved, I don't see that too many people will be using it on semi-pro/pro shoots in the future.

I think Midphase has pointed out a couple of steps that are missing. And I see some "hate" against Midphase for pointing this out. But for real world work, there are steps missing for Magic Lantern to be accepted shoots other than weekend warriors as he points out so well.

At the moment its something that needs to be thoroughly convinced to people.

my 5 cents.

ps.

I love ML dont hate.
Title: Re: Is MLV all that it's cracked up to be?
Post by: albert-e on March 21, 2014, 04:50:12 PM
I agree totally  :)
Title: Re: Is MLV all that it's cracked up to be?
Post by: Audionut on March 21, 2014, 04:56:49 PM
There is no hate, simply a misunderstanding of the expectations of the developers of the Magic Lantern codebase.

As a user, it is significantly easier just to ask for stuff.  When the response is not one which is expected, continued persistence is not conductive.
Title: Re: Is MLV all that it's cracked up to be?
Post by: g3gg0 on March 21, 2014, 08:54:47 PM
Quote from: Kharak on March 21, 2014, 04:45:32 PM
I think Midphase has pointed out a couple of steps that are missing. And I see some "hate" against Midphase for pointing this out. But for real world work, there are steps missing for Magic Lantern to be accepted shoots other than weekend warriors as he points out so well.

thats not hate or anything related.

situation:
- things are not ideal for use case X

A:
- a person shows up and asks provocatively "is that all?"

B:
- a person shows up and explains:
   * his/her situation
   * the workflow he/she has
   * realistic things to improve on that workflow


pick your favorite.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 22, 2014, 05:04:35 AM
g3ggo...if you're interested in having an adult conversation about this, PM me and I can go into as much detail as you want.
Title: Re: Is MLV all that it's cracked up to be?
Post by: Kharak on March 22, 2014, 06:02:42 AM
@g3gg0

Perhaps the word "hate" was ill chosen.

And I'd choose (B) ofcourse, who wouldn't. The thread has gone off-topic in many directions from its original post (including this one), but the later posts have also raised some fundamental questions, in my opinion.

I do also think the name of the thread is a bit provocative even though he clearly states at the top of his post, that it's not meant to be.



Edited after post below: Look what I found. http://www.magiclantern.fm/forum/index.php?topic=7486.0
Title: Re: Is MLV all that it's cracked up to be?
Post by: Midphase on March 22, 2014, 06:37:53 AM
Thank you Kharak.

My original question was really made in an effort to figure out what I am missing out on by not moving over to .mlv from .raw considering the fact that:

1. Writing .mlv files seems to yield worse performance in camera than .raw

2. The conversion tools to CDNG were at the time non-existent (amazing what a few weeks difference made thanks to MLRawViewer and the "other" app).

3. The metadata saved with .mlv files doesn't seem to improve the post production workflow since Resolve, Premiere or FCP don't appear to make any use of it.


But yeah...it did branch out.
Title: Re: Is MLV all that it's cracked up to be?
Post by: g3gg0 on March 22, 2014, 12:23:20 PM
i would prefer it the other way:
-> have the adult discussion in public forum and the childish ones per PM.

Quote from: Midphase on March 22, 2014, 06:37:53 AM
2. The conversion tools to CDNG were at the time non-existent (amazing what a few weeks difference made thanks to MLRawViewer and the "other" app).

thats why everything is tagged as "EXPERIMENTAL" and "UNDER DEVELOPMENT"