Shooting a short next weekend, scared to not handle RAW properly

Started by disfordrums, April 26, 2014, 06:43:57 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

disfordrums

Hey guys,

We've rented a 5D III before to run some tests with Komputerbay 64gb cards.
The ones we bought seemed to be doing well in 1920x818 30p.
However, since we're short on budget, we send the cards back to Amazon.

I've already bought 8 Komputerbay 64gb cards, but have no way to test them until
we're picking up both Mark III's. Which will be next Friday. The shoot is will be Sat-Sun.

1. Are there any reliable tools I can use to test the memory cards (on Mac)?
2. Magic Lantern automatically stops a recording when frames are dropped?
3. We're renting a 7" HDMI Monitor, is Magic Lantern compatible?
4. Is RAWtoDNG sufficient on set, or should we just dump the .RAW files directly?

Is there anything else I should be weary of when using Magic Lantern on set?

Thanks,
Michael

-sandro-

 I can answer question 3. You can choose in the settings whether to stop or continue,

WeekendWarrior

Quote from: disfordrums on April 26, 2014, 06:43:57 PM
Hey guys,

We've rented a 5D III before to run some tests with Komputerbay 64gb cards.
The ones we bought seemed to be doing well in 1920x818 30p.
However, since we're short on budget, we send the cards back to Amazon.

I've already bought 8 Komputerbay 64gb cards, but have no way to test them until
we're picking up both Mark III's. Which will be next Friday. The shoot is will be Sat-Sun.

1. Are there any reliable tools I can use to test the memory cards (on Mac)?
2. Magic Lantern automatically stops a recording when frames are dropped?
3. We're renting a 7" HDMI Monitor, is Magic Lantern compatible?
4. Is RAWtoDNG sufficient on set, or should we just dump the .RAW files directly?

Is there anything else I should be weary of when using Magic Lantern on set?

Thanks,
Michael

I do know that the Komputerbay cards are supposed to be the least reliable with magic lantern. But if you have 8 cards, you might be alright.
1. I'm not too sure
2. I usually don't experience dropped frames, but if I did, I wouldn't want to continue recording.
3. To my knowledge, the only chance you have at using a monitor is having a Mark 3 with 1.2.3 firmware, and magic lantern 1.2.3. Not positive.
4. I personally would just dump footage on set, and worry about raw2dng later while you're not on the field.

Midphase

Just run a bunch of tests. Preparation is key. You can answer all of your own questions by simply testing your workflow thoroughly before your shoot.

Midphase

P.S.

If you're on a Mac don't use Raw2DNG. Use RAWMagic or MLRawViewer.

disfordrums

Thanks for the input guys, really appreciate it!

Quote from: WeekendWarrior on April 27, 2014, 04:59:22 AM
3. To my knowledge, the only chance you have at using a monitor is having a Mark 3 with 1.2.3 firmware, and magic lantern 1.2.3. Not positive.

This is very concerning, thank you! So basically if we want to shoot RAW, we can't have a monitor?
Is there any thread about this, maybe this issue has been fixed in a later version since you've tried to use it?
Or there may be some other way around it? Thanks again for the answer,
this would have been one of the "bad surprises" next week!

Quote from: Midphase on April 27, 2014, 07:53:05 AM
If you're on a Mac don't use Raw2DNG. Use RAWMagic or MLRawViewer.

Is this because Raw2DNG converts it automatically and RAWMagic doesn't?

To the testing. Like mentioned, we did a basic test but we didn't have a Monitor available.
This is one of the productions where me and my friend have to do everything from producing
to all the creative work. Our work schedules unfortunately prevented any other testing except
the tests we can run the night before the shoot :(

llirik

Will you be needing sound (whether record direct or sync later?)

Then you'd need to record to MLV instead of RAW.... and this you'd need to test. I was working with the Mar 17th build last week and it was extremely buggy and kept dropping frames (and the black level issue). Whereas some quick tests on the April 24th build haven't shown the same issue.

So you really do need to test, or ask people on a certain build, as they vary.

SteveScout

Hi, Michael!

Just shot half of a feature a few weeks ago on ML - 95% of the time it worked fine.

1. Komputerbay cards are definitely not the best and accident-free choice, but if you are on a budget (like I was) the only choice. From time to time they would drop frames or suddenly don´t start properly recording .. so we just switched cards (had five of them) and dumped and formatted the strange behaving one, most of the time it was fine afterwards again.
2. Shoot MLV with sound, so you can sync your external sound much easier with PluralEyes for example.
3. If the write speed is not enough (it is not on SOME of my KB cards) I cropped the image vertically slightly (1:85 to 1) as I am going for cinemascope anyway. Full 16:9 sometimes did not work with the audio recording on.
4. Make backups on set and have someone quickly looking into the MLV files randomly. It´s an overhead person, but in case a file gets corrupt you can reshoot. We had only ONE single file getting corrupt in a week´s shoot, and this was a very long take of about 10 minutes. Don´t know what happened, but generally everything was pretty reliable with the (back then) latest version of ML and of course the 1.1.3 firmware.

Now to your questions:
1. Test the card in camera. ML has a card test feature, it will give you the write speeds to see if it delivers.
2. yes!
3. I detached my monitor as it was behaving strangely. HDMI out definitely works, even on 1.1.3 firmware, but the image will be SD (bad for focusing!), so we rather went on using a hood loupe for the MK3s screen.
4. Dump the raw files and look into them with MLVrawviewer quickly, don´t do conversion on set, not needed at this stage. You just need to know if the file plays back fine, everything else can be done later without small external harddrives and a bigger RAID etc.

cheers and good luck!
Steffen

Midphase

Quote from: disfordrums on April 27, 2014, 08:27:27 AM

This is very concerning, thank you! So basically if we want to shoot RAW, we can't have a monitor?

That is not true at all. I shoot with my external 7" monitor in raw all the time. I'm using 1.1.3 firmware which at this point I would highly recommend you stick to for the time being.

Personally I think .raw is more stable than .mlv, so unless you really really in-camera audio, I would try to stick with .raw recording.

The system is stable, and reliable. You can do everything you can do with regular video shooting, and more.

I will say it again, run tests and get used to it. Preparation is really key. If you're planning on showing up on set without ever having used your particular set up and expect everything to work flawlessly right away, you're looking for trouble.


Midphase

Quote from: disfordrums on April 27, 2014, 08:27:27 AM
Is this because Raw2DNG converts it automatically and RAWMagic doesn't?


No, it's because Raw2DNG is a crappy way to do it, especially on the Mac. Here is why:

1. On several occasions, Raw2DNG deleted my original .raw files, this is very bad.

2. It exports DNG and not CDNG, so you can kiss compatibility with Premiere Pro goodbye if that's a concern for you.

3. Doesn't work with .mlv files

4. If you love command line apps, than this is the one for you. On the other hand if you're a normal person, you will hate it.

5. There is no real-time preview like there is in MLRawViewer

6. No progress bar, awesome!


In short, RAWMagic and MLRawViewer on the Mac offer much more elegant, reliable, and fast solutions for an actual working production.

I don't know when your shoot is, but I can't overemphasize (for the 3rd time) how important preparation and know all of your tools really is when shooting in ML raw. For a film that I worked on a few months ago, we prepped for ML raw shooting for about a month. We tested, re-tested, used various settings, and ran various conversion apps. The shoot went smooth for the most part (there were a couple of hairy moments still).

I also think it's critical to not only have your camera operator knowing ML inside out, but also your DIT.




Midphase

Also:

Komputerbay cards can work quite flawlessly. The main reason why there are more reports of Komputerbay cards failures on the forums is quite simply because more people bought them than other brands, so it appears that they're more troubled than other brands; but if you look around you'll find reports of failed Lexar cards as well.

I own 4 64Gb cards, they all rock it and I've never had a failure or needed to send them back. With 8 64Gb cards you should be in good shape (are you using 1 or 2 cameras?).

1080p24 recording should not be an issue, even with sound. There are a few settings that you should adjust and which will give you the best performance. Once again, .raw is much more reliable and fast than .mlv in my tests.

Format the cards in ExFAT on your computer, and most importantly set up the >4Gb setting in ML.

disfordrums

This is amazing :)! Thanks for all the feedback!

@Ilirik

So far we've been planning on shooting in raw since in most scenes, sound is not recorded on set. We do have one monologue though, but we use a slate to get the syncing right!

@SteveScout

You raise several good points. When you say your files got corrupted, do you mean visible artifacts, or something less noticeable? How bad is the SD image for focusing? We did actually get a hood loupe, but either the quality must have not been very good, or it is seriously hard to pull focus using a hood loupe. With it, I could make out the pixels on the screen, which gave everything kind of hard edges!

The MLVrawviewer really does sound like the best solution on set!

@Midphase

thanks for the uplifting words! Good to hear at this point!

The thing is, we did try it out one day and settle on a setup. And I've been using Magic Lantern on my 600D for over 3 years while being on the latest nightly builds. So I'm really familiar with the software. But just not that familiar with raw. We did manage to shoot it though. And that's actually all we need to do. Get footage that is not corrupt.

So do I understand correctly, RAWMagic is for conversion and MLRawViewer is just for looking at the footage without converting?

Thats a good point you raise, a common human fallacy actually. I'm sure there is some difference between the quality of the Komputerbay and Lexar cards but the Komputerbay cards really are not that bad!

Why the formatting to ExFAT? I actually don't recall formatting the Komputerbay cards we had besides in camera and that worked well?!

Thanks again everyone for the replies!

Midphase

Quote from: disfordrums on April 28, 2014, 02:03:41 AM
So do I understand correctly, RAWMagic is for conversion and MLRawViewer is just for looking at the footage without converting?

Why the formatting to ExFAT? I actually don't recall formatting the Komputerbay cards we had besides in camera and that worked well?!

Both RAWMagic and MLRawViewer can convert to CDNG. RAWMagic is a tad faster and a bit more elegant with drag-and-drop ease for .raw, but if you shoot .mlv, then MLRawViewer is the way to go (for now). What I like about RAWMagic is the simplicity of the interface, and the ease. If you need to export not as CDNG but as ProRes, then you also will need to use MLRawViewer since RAWMagic can't do that (for now).

About ExFAT....if you format in ExFAT that means you can record files larger than 4Gb without worrying about segmenting into a series of files. I prefer it for several reasons. First of all it's easier to keep everything organized and less cluttered. Also, it used to be that MLRawViewer would crash when playing back segmented files, but I think that's been fixed. I also think (perhaps incorrectly) that 1 single file is less susceptible to corrupted frames than multiple ones.

If you're shooting in .raw, I don't think you need to do anything more than simply format the CF cards on a computer as ExFAT. If you're shooting in .mlv, then you also need to enable the >4Gb option so that it knows to continue to write data larger than 4Gb on the card. The main negative of ExFAT is that you can't re-format the card in camera. Personally I don't think that's an issue since I try to discourage people from formatting in camera due to the great risk of losing footage by accident.

disfordrums

Ok, so just to see if I got it right:

1. Test cards in camera to see if they are up to speed
2. Format them with disk utility to ExFat to not be restricted by 4gb limit
3. Record RAW on set
4. Just dump the footage as is and check randomly using RAWMagic
5. Use slate to sync sound with image

Also playing around with the idea to get an iPad slate app, instead of using a physical one to save some time...

Midphase

Quote from: disfordrums on April 28, 2014, 09:49:51 PM
Ok, so just to see if I got it right:

1. Test cards in camera to see if they are up to speed
2. Format them with disk utility to ExFat to not be restricted by 4gb limit
3. Record RAW on set
4. Just dump the footage as is and check randomly using RAWMagic
5. Use slate to sync sound with image

Also playing around with the idea to get an iPad slate app, instead of using a physical one to save some time...

1B. Test the entire workflow, settings, etc. Things on a set are crazy, you'll be juggling 20 things at once.

2. Yes, but double check for that >4Gb setting in ML, should be in the raw recording menu. I forget if RAW_REC doesn't use it, but I know MLV_REC does.

4B. Use MLRawViewer to view the footage (you can do this straight from the card if necessary. This will tell you if everything looks good or not. Try not to blink when you review the footage so as to spot any corrupted frames.

4C. Make sure you have tons of hard drive space. What I recommend is this workflow: Recorded CF cards get a red gaffer tape piece stuck to them. When the CF card is handed to the DIT, he/she copies the files to 2 separate drives. Keep in mind that transfer time for 64Gb on USB 2 is about 30 minutes. Using USB 3 is drops to about 10 minutes. Also, make sure you have a good CF card reader, the one that everyone recommends including Komputerbay is this one:

http://www.amazon.com/Kingston-Memory-Card-Reader-FCR-HS3/dp/B005ES0YYA/ref=sr_1_10?ie=UTF8&qid=1398716185&sr=8-10&keywords=cf+card+reader

Once the CF card contents have been transferred to both drives, and some of the footage reviewed for integrity, then you can re-format the card in Disk Util to ExFAT, remove the red tape and hand it off to the 2nd AC. Rinse/repeat. Expect to fill up about 500Gb of drive per shooting day, even more if you're using two cameras simultaneously.


Big NAY on the iPad slate. Seriously, it sucks and your assistant editor will hate you for it. The beep for sync is too faint and isn't clearly visible on the audio waveform, also if the iPad isn't fully in focus, information is hard to read. We used it on our production and not a day goes by that I don't kick myself for not using a regular slate (which I own anyway...duh!). Seriously, it will not save you time, it will waste you time in post.


BTW...feel free to PM me for more detailed info on my experience.

Steven

With firmware 1.2.3 monitoring works flawlessly, full 1080p, no delay/glitch after pressing record (only when going into menu and back). I recorded a backup to a recorder in case the raw files had glitches.
Drawback: With the build from march that I tested all overlays on the camera screen disappeared, when connecting a monitor. So I was without cropmarks and other little helpers.

ExFat is not necessary, because ml supports file spanning into multiple 4GB files. (or did I miss any other benefit from the exfat format?)




SteveScout

You can try the HDMI-image for focusing right away - it´s the same quality without magic lantern. The HDMI output of the 5DMK3 with 1.1.3 firmware was never Full-HD, so you can judge even without ML or MLraw running what the quality will be like and if it is enough for your focusing needs - for me the onboard monitor was the better choice.

But if the new firmware works fine for some - great. My personal choice was rather take some quirks into account (like broken files that are completely unreadable, but I would restart the take anyway once I get a strange error which I got several of in a shooting day) and have sound to get PluralEyes working, because at our speed of shooting a slate would not have worked.

Definitely don´t use one of that ipad apps .. I love smarthphone/ipad stuff on set .. but they are unusable. Either too small for the distance or not loud enough for sound ... not working in the real world.


Mei Lewis

Quote from: disfordrums on April 27, 2014, 08:27:27 AM
This is one of the productions where me and my friend have to do everything from producing
to all the creative work. Our work schedules unfortunately prevented any other testing except
the tests we can run the night before the shoot :(



Quote from: Midphase on April 27, 2014, 08:09:03 PM
I will say it again, run tests and get used to it. Preparation is really key. If you're planning on showing up on set without ever having used your particular set up and expect everything to work flawlessly right away, you're looking for trouble.

I'd emphasise what Midphase said.

If you're trying to do everything yourselves and you've not properly tested it all, that's a big risk.
There are a lot of things that can potentially go wrong.

Yes, ML RAW can work and be great but almost everyone seems to have teething troubles with it and most people on this forum are shooting small test videos that don't involve other people and scheduling. It sounds like you're making more a proper film.

On this production is the extra quality of shooting RAW really worth the risk of delays, faulty cards, losing footage and all the extra time and disk space needed? I'm not saying those things WILL happen, but given your lack of testing they MIGHT.
Aren't the plot, the acting and how easy it is for the actors on set, the composition of the images and so on more important than an improvement in image quality?

Mei Lewis

Quote from: disfordrums on April 26, 2014, 06:43:57 PM
I've already bought 8 Komputerbay 64gb cards, but have no way to test them until
we're picking up both Mark III's. Which will be next Friday. The shoot is will be Sat-Sun.

So you have 8 cards and 2 cameras, so you're doing a 2 camera shoot with about 4 cards per camera?

Each card records what, about 8 minutes total?
And how long does it take to copy off to a computer/hard drive? Probably a fair bit longer than 8 minutes.

Is that enough cards? Have you tried runnign through the whole workflow, seeing how long everything takes and if there are any delays introduced waiting for cards to copy?


disfordrums

Hey people,

so in the end we didn't shoot the film this weekend. There were other problems with the camera
rental place and the lens rental place that contributed to this decision, but part of it was also due to the
overwhelming advice of this thread to run more tests.

Which is what we're doing at the moment.

We got a 5D Mark III today and already ran into some problems. I installed Magic on one of the cards, no problem.
Installed it on a second card, forgot formatting the second card to ExFat, also no problem.
Installed in on a third card, all of a sudden the camera doesn't boot into magic lantern and instead
freezes and I have to take out the battery to restart it.

I've tried to install it on three more cards, and all show the same problem. Magic Lantern is not
loaded automatically and if I choose to "update Firmware", it installs it, but then freezes when rebooting.

1. Does anyone have an idea what I might be getting wrong?
2. How does Magic Lantern number each file, seems to be pretty random when using two different cards?
3. Can MlRawViewer change the aspect ratio?


@Mei Lewis

we have a fast usb 3.0 card reader which takes about 10 minutes to dump the footage.
Considering the time it takes to change between scenes, I think 4x64gb per camera
is enough. We will go out into the field tomorrow though and test the whole thing.

@Steven

Does that mean when not using ExFat, I'll just have several .RAW files for one longer take?


Thanks again everyone for the input!

Midphase

Quote from: Mei Lewis on April 30, 2014, 12:26:22 PM

Each card records what, about 8 minutes total?
And how long does it take to copy off to a computer/hard drive? Probably a fair bit longer than 8 minutes.

64Gb cards on the 5D3 record roughly 12 minutes of 1920x1080 at 24fps. Takes roughly 10 minutes to copy the data of a full card to a hard drive with a USB3.0 reader.

Midphase

Quote from: disfordrums on May 03, 2014, 12:05:09 AM
1. Does anyone have an idea what I might be getting wrong?
2. How does Magic Lantern number each file, seems to be pretty random when using two different cards?
3. Can MlRawViewer change the aspect ratio?

1. Yes. You forgot to run the Make DSLR Bootable script on the card.

2. Typically you set the file naming convention in the Canon menu and ML should abide by that. I think there are also some new settings to enable you to name takes and such in ML.

3. I don't think it can. You're stuck with whatever you recorded at.

disfordrums

Quote from: Midphase on May 03, 2014, 05:35:55 AM
1. Yes. You forgot to run the Make DSLR Bootable script on the card.

2. Typically you set the file naming convention in the Canon menu and ML should abide by that. I think there are also some new settings to enable you to name takes and such in ML.

3. I don't think it can. You're stuck with whatever you recorded at.

1. I didn't actually. Just figured out that I screwed up the order. Seems obvious in retrospect, but it was a long day yesterday. Thanks for the tip, works now!

2. I couldn't find anything in ML but will have a more thorough look once all the memory cards are running!

3. I took a look at the MLRawViewer options and the thing I was looking for was the hotkey "a". This changes aspect ratio from stretched to normal when viewing!


One other thing I noticed, is that my mac tells me that the 1080p cDNGs are actually 1916x1076. Any thoughts?
Also I'm not completely sure which buffer size write speed is the important one to be able to tell if RAW can record
at certain resolutions? Is it 16384k?

And something else odd I've noticed, is that when I try and shoot RAW, the Histogram disappears :?

Actually, another thing I just noticed, is that when filming 48fps 16:9 at 1728x606, there appear to be pink bars
on the right and bottom of the screen (just a few pixels in width). Will see if some different resolutions change that..

Cheers


g3gg0

Quote from: Mei Lewis on April 30, 2014, 12:20:21 PMbut almost everyone seems to have teething troubles with it and most people on this forum are shooting small test videos that don't involve other people and scheduling. It sounds like you're making more a proper film.

on forums you always see those who *have* problems.
of those who had *no* trouble after all, only 10% report positive feedback

thats the reason why the forums are full of negative posts.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

SteveScout

For the histrogram: Ist Global Draw turned ON for recording?