Writing raw video to USB storage (as opposed to SD/CF slots)

[DigitalVeil]

I'm curious if it's at all possible to write raw video files out through a camera's USB port directly onto something like a USB flash drive, something like having the EDMAC copy the sensor readout to the USB controller instead of the SD controller.  Of course, all current ML-supported cameras have USB 2.0 or slower, which in real-world situations tops out around 20MB/s, so this feature would only be practical if it were possible to dual-write to both the SD/CF and USB port at the same time (same way the 5D3 can dual-write to both CF+SD at the same time).

I know this idea has been kicked around here from time to time (like here: http://www.magiclantern.fm/forum/index.php?topic=6128.msg57345#msg57345) but it's never really been seriously pursued.

If this could be done, it could provide a pretty significant boost to all cameras (at least 2.0 models) in terms of raw recording capabilities.  Cameras like the 650D, 700D, 70D, and EOS-M could theoretically write up to ~60MB/s.

[a1ex]

Benchmarks first, please.

[pursehouse]

I don't know what the speed boosts would be, but it would certainly be an advantage for storage size when shooting raw video. Sending video to a 1tb usb2/3 drive would make raw/dualIsoRaw video MUCH easier/realistic to use in real life scenarios without a crazy amount of money spent on cards

[chmee]

if it ends in such a split-file-combination you just raise the upper datarate-limit - but it cant avoid copy/merge-orgies. and it wont(!) give advantages for storage size, its just an option on top. the main speed is still on the cf-card.

[DigitalVeil]

if it ends in such a split-file-combination you just raise the upper datarate-limit - but it cant avoid copy/merge-orgies. and it wont(!) give advantages for storage size, its just an option on top. the main speed is still on the cf-card.

Every bit of speed counts, especially with these SD-slot cameras 

[chmee]

honestly, the design-/testing-/stable-tests cost more than to buy a 5DIII - if not money, so it costs time, a lot. such a thing makes no sense in terms of economics. if you re ready to show a stable build, canon has built two other generations of eos c-cameras, blackmagic pocketcams cost 400Eur and the aperture axiom is already in a stable state. sorry.

[DigitalVeil]

Trust me, if I had the financial resources, I'd own one of those cameras.  And if my programming knowledge was more advanced, I'd do it myself.  Just trying to see if there's anything left to squeeze out of these DSLRs.

[g3gg0]

yeah its honorable.

but we have thought of so many things we could use to write at higher rates.
its not about "connecting edmac a and thing b and we are done".
its about massive work to be done

[Walter Schulz]

For whatever it may be worth:
Just copied a big file (about 2 GByte) from 650D to Computer's SSD via USB 2.0.
About 22 MByte/s transfer speed, handstopped.

OS: Windows 7/64.
Virus Scanner active (AVG free)
Card was 8 GByte SanDisk Extreme Pro (95 MByte/s, UHS 1)

[pursehouse]

I use a 5d Mark3, and even though the RAW/DUALISO feature is amazing and I love that the coders here made it happen, it's not very practical in a normal environment. It's useful if you know exactly what you are shooting and it is all setup beforehand, but you can't use it out in the wild. If external drive storage is fast enough, it would really put things over the top on level like how the Red cameras have hard drives for recording their raw video. So, my two cents.. if it is possible to do that, and the speed is good ( usb2 should be 60mbps ), then it would be a pretty awesome feature. Especially since we have to assume that the new cameras will start having USB3 connections at some point right? Hopefully the 5dIV has usb3.

[Walter Schulz]

60 MByte/s for USB 2.0 is just theory. In practice you will have less. USB 2.0 is known for terrible protocol overhead and you will see only few devices reaching 40 MByte/s.

[g3gg0]

50 MiB/s is doable with bulk transfers.
(did that already using a CY7C68013A)

[pursehouse]

Is it even possible for a camera to act as a USB host to an external usb-drive? That part is kinda crucial haha

[dmilligan]

Is it even possible for a camera to act as a USB host to an external usb-drive?

AFAIK, No. But...

That part is kinda crucial haha

No it's not, you just have to offload to a computer/tablet/raspberrypi/whatever that acts as the host, thats running a program that will receive the data via PTP. One of the many complications of implementing this (alluded to by g3gg0), is the need to write this client.

[DigitalVeil]

60 MByte/s for USB 2.0 is just theory. In practice you will have less. USB 2.0 is known for terrible protocol overhead and you will see only few devices reaching 40 MByte/s.

Yeah I rarely see USB 2.0 transfers reaching even 30MB/s.  But that's pretty good if you're adding it to the existing 40MB/s via dual-writing.  And as g3gg0 just said, faster is possible under the ideal circumstances (like no compression).

As far as my own experience, usually Windows reports the exact transfer rate to me while a file transfer is in progress, but for some reason it wont tell me when copying from camera to PC so I had to hand-measure it.  I was able to copy a 1.13GB h264 video from my 700D to my SSD in ~45 seconds.  In base 2 definition, that's 25.89MB/s.  Of course there's going to be some human error in hand-measuring, but it's definitely at least 20MB/s.

yeah its honorable.

but we have thought of so many things we could use to write at higher rates.
its not about "connecting edmac a and thing b and we are done".
its about massive work to be done

Right, I didn't expect that it would be some easy task or anything like that.  I was first and foremost wondering if it was even a possibility.

[emphram]

I haven't had the time to read this thread correctly, but I wonder if it has been established whether or not you can write to the USB and the SD card at the same time?

If you could do this, then in theory, and the advantage would probably only be for the 600D and 550D, you could store half a frame on a usb device, and the other half on the SD card, allowing you to store at a higher resolution, which I estimate would be 1280*720 (assuming that the combined transfer rate is 40mbps which gives you 720p on the 650D and 700D) on the 600D, maybe a stop lower to make up for the processing work requiered.

Of course this presents several issues not only for deving in ML but also client side, and how to process the video, combining info from both files into a new file. If the ML devs made this type of recording possible, assuming it is possible, then I offer to write a Windows App to process the raw files.

Edit: Also, I wouldn't rule out hooking up a SD card to the micro usb port with a card reader, or if necessary a custom card reader.

Edit2:

No it's not, you just have to offload to a computer/tablet/raspberrypi/whatever that acts as the host, thats running a program that will receive the data via PTP. One of the many complications of implementing this (alluded to by g3gg0), is the need to write this client.

I might know someone who could do this, but I can't guarantee it.

[dmilligan]

The MLV format already supports spanning, and on the 5D3 you can take advantage of this to write to both CF and SD cards at the same time, and all of the existing MLV converters already support reconstructing these spanned files correctly.

The bottom line is: we know how to do this, and it would work, but it is a lot whole lot of work to implement, and might not be worth the effort (compared to what could be done spending that time on other things like FRSP for example)

[emphram]

I might be a bit misinformed regarding FRSP, and perhaps the lack of sleep for the last few weeks is getting to me, but how does FRSP improve video capabilities on the 550D and 600D? Wasn't it demonstrated a while back that you couldn't use the ARM of a DSLR to compress images fast enough to write them to an SD or CF card? Unless FRSP provides some way of capturing higher quality than h.264 frames at an acceptable frame rate (and I'm assuming that it would do this by using something other than the ARM), I don't see how this would improve video quality on the lower end models.

[emphram]

Don't mean to bump this, but I spoke to another programmer who has more experience with a much broader range of programming than me (I've only done desktop apps, web and some pic processors), and he thinks he could do a PTP client for windows, open source, allowing others to port it. He would like to first review documentation on MLV (assuming that MLV is going to be the format in which this is going to be recorded in), and whatever other documentation he may need to see from the ML team. If he could do the windows client, would this get this moving?

[dmilligan]

I don't see how this would improve video quality on the lower end models.

It wouldn't. But not everyone here cares solely about video. These are primarily still cameras after all. All I'm saying is that perhaps it's time better spent for a feature that would be more useful for more people (even if those people aren't those who only care about video). Note how much interest there is in FRSP thread. People beg for a build pretty much every day.

If he could do the windows client, would this get this moving?

There's a ton of in camera work that would have to be done, and these things would really need to be developed in parallel, as they would be so heavily dependent on one another. You couldn't  just develop the client first in a vacuum (it's also the "easy" part, since coding/running/debugging is much much easier on a computer than it is in camera).

[DigitalVeil]

The bottom line is: we know how to do this, and it would work, but it is a lot whole lot of work to implement, and might not be worth the effort (compared to what could be done spending that time on other things like FRSP for example)

I might be a bit misinformed regarding FRSP...but how does FRSP improve video capabilities on the 550D and 600D? ... I don't see how this would improve video quality on the lower end models.

Both of these things could actually work together and be extremely worthwhile for future Canon DSLRs with USB 3.0.  It's maybe a longshot, but something monumental could be accomplished on the 7D Mark II with both FRSP and USB writing combined: "5K" raw video.  Obviously, the 7D2 is the first Canon DSLR with USB 3.0.  It also has a CF and SD slot to go along with it, which together can reach a combined ~175MB/s.  USB 3.0 in the real-world can reach at least 400MB/s (if not more when there's no encoding).  If you could span files across all 3 interfaces (not sure more than 2 is even possible) we're looking at a theoretical 575MB/s (or more) possibility.  If it's limited to just 2, USB+CF is still 500MB/s which is crazy.  Now, I don't remember what the max speed of the EDMAC is (500MB/s?) nor have I calculated the speed required to readout a full 20+ megapixel sensor at 24fps continuously, but I think you get the point.

As it has already been established, it would require a huge amount of collaborative work, and maybe it's not worth it for older USB 2.0 models, but I think it could allow for some incredible things with the 7D2 and future Canon models.

[g3gg0]

i tend to postpone this discussion until there are measured speeds.
with computers 1+1 isnt always 2. sometimes its 1.1, sometimes 1.4 and rarely its 1.8. it even may happen that it is 0.9.