History of ML cracking RAW

[Milk and Coffee]

For those of us who are new, can a veteran of the forum provide a TLDR explanation of how ML achieves RAW video? And in layman's terms as much as possible?

I think I recall someone eventually gained access to EDMAC (is that the right acronym?) How did we get there, and then what followed?

[names_are_hard]

I'm not really a veteran, but I understand the process at a "big pieces" level.  Are you actually asking for a history, as the title says, or a description of how it works?  I can't give you a history.

Big pieces description:

The sensor can obviously take raw images.  A video is just a sequence of images.  But each image is large in raw format, and using the CPU to copy to the card is too slow for video.

DMA is Direct Memory Access.  This is a technology that allows copying from place-to-place without the CPU being involved, removing that bottleneck.  EDMAC is Canon's term for their DMA controller - the chip that coordinates DMA transfers.  Learning how to control EDMAC allows fast enough transfer from the sensor to the card for video to work.

There's a lot more details, but a) I don't know them b) I don't know what you want to know

[Milk and Coffee]

Names_are_hard, this is a great start! Yes, this is what I'm asking for. Basically a quick "big pieces" roadmap that says how ML turned canon cameras into RAW video beasts.

I suppose when people join the forum to get started understanding ML, there isn't a great source that lays it out quickly and simply. I have found that a quick explanation helps me wrap my head around how we got where we are, and the advancements made.

I think this could help clear up a lot of confusion for newcomers, and eliminate a lot of random questions in threads (of which I am guilty of too.)

[Walter Schulz]

In the background (=discord server) there is some talk about reorganization. Audionut took some initial steps clearing some things up here in the forum. I'm convinced a complete redo of main website is in order. IMO ML makes a pretty good example for Vogonism.
ATM I'm working (more or less) on 2.5 wiki pages:
Glossary: A low-tech approach to explain some ML related terms. General rule: Don't be afraid of oversimplification. Technical incorrect entries are ok if reader gets the general idea.
House of Cards: Everything about card handling for ML users. Rule: Correct in technical terms, if possible. (ATM very rudimentary)
And 0.5 for long-term sleeper project: https://wiki.magiclantern.fm/camera_help Should replace missing in-cam help for menu entries.

Devs have created their own wiki pages to preserve some recent findings. Take a look at wiki site map

Personally I don't think a RAW/MLV history is that useful to help new users to find their way through current doc mess. But if you want to make your own wiki: Be our guest! And feel free to participate in other wikis. ;-)

[Milk and Coffee]

Walter, that is great news! I also think it might be time for some cleaning up!

Names_are_hard,
1. was access to the DMA (EDMAC) what enabled us to capture RAW video? Are there any caveats to the RAW files (its truly RAW???)
2. normal h.264 video had to go from the DMA to the CPU, then into the card correct?

[Kharak]

Vogonism, thanks I learned a new word today and I fully understand it from the ML perspective.

[names_are_hard]

In ML, it's very rarely the case that you understand one thing and then you can do a complicated feature.  It's a lot of little things, added together.  DMA is a necessary component, you need it for high enough internal data rates.  Many other pieces are required.  Yes, raw video files are sequences of frames of raw sensor data.

DMA circumvents the CPU.  It's an industry standard:
https://en.wikipedia.org/wiki/Direct_memory_access

I imagine DMA is also used for h264, the h264 encoder is presumably dedicated hardware, not CPU, so you'd want to offload the memory transfer when using it.  I haven't checked this but it's the way you'd build it.

[Milk and Coffee]

I know that there were two formats of Magic Lantern RAW video files. The current, .MLV. Are both formats just as RAW as the other? Any benefits or caveats to either format?

Also, is the data stored RAW Sensor Data (not debayered)? Or is it a "flavor" of RAW similar to Blackmagic's .BRAW (already debayered)?

[Walter Schulz]

How to get info for (most) features:
Top of page -> Downloads -> Downloads (Drop down menu) -> Nightly Builds -> Features
Now look for entries in Modules section named mlv_lite and mlv_rec. Names are linked to first posts of forum threads covering a feature.