Messages

1

Modules Development / Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)

« on: September 11, 2018, 01:16:21 AM »

I made the mistake of processing a directory full of MLV files (using the excellent Switch.app on macOS) and then deleting the MLV files before checking the resulting DNG sequences. 4 of the DNG sequences were my first attempt at using dual ISO, and I did not have Switch set to enable dual ISO processing. So I have several DNG sequences that have the interlaced dark and light line look of unprocessed dual-ISO. I've checked my most recent backups, and unfortunately they were made before I copied these new MLV files to my RAID. It's a perfect storm of absent-mindedness on my part.

Do any of the available dual-ISO tools have the ability to process DNG sequences rather than MLV files? I figure the same information is right there in the DNG waiting to be utilized.

Thanks for any help or information.

2

Raw Video Postprocessing / Re: Best Apple Pro Res for 10, 12 bits RAW.

« on: April 05, 2018, 10:39:45 PM »

Just figured I'd pass this new Apple white paper on the new Prores RAW codecs along:

https://www.apple.com/final-cut-pro/docs/Apple_ProRes_RAW_White_Paper.pdf

3

Modules Development / Re: MLV Lite

« on: November 23, 2017, 11:08:28 PM »

Hmm, I tried the same file with mlv_dump.osx (a very old mlv_dump, dating back to 2014, since I can't seem to find a binary for the latest mlv_dump (searches just lead me to old forum threads). The result was a bunch of nice looking dng files and a 44-byte wav file that seemed to contain no data. I looked at both its contents and those of the wav file that Switch had generated and they both appeared to be pretty much content free, but the Switch version included some Blackmagic metadata that must be there for compatibility with Davinci Resolve.

So I guess I'm recording files with blank data.

4

Modules Development / Re: MLV Lite

« on: November 23, 2017, 10:53:54 PM »

Hmm, I'm trying this with 5D113 firmware and "magiclantern-croprec4kmlvsnd.2017Oct23.5D3113", 3072x1308 12-bit lossless, and have made a few little recordings. I'm processing the resulting MLV files with the 11/7 version of 'switch'. Each time, I end up with a director full of nice-looking 3072x1308 dng files and a single wav file of 2k in size that's silent.

It's possible that the audio is just not getting properly recorded, but I don't know for sure that the audio isn't present just fine in the MLV file and that Switch isn't just having trouble extracting it. I'm not sure how to tell since I don't know the best, approved manner by which to extract audio ffrom MLV Lite recordings.

5

Raw Video Postprocessing / Re: Switch for macOS Sierra/Linux (formerly cr2hdr.app)

« on: September 19, 2017, 06:21:30 PM »

You know you could submit a pull request in Bitbucket so anyone who wants to test out your suggestions can easily do so and give feedback. Though knowing Danne he is probably already trying out the '-s' flag.

Ah, interesting. I figured there was some sort of standard procedure. Have never used bitbucket.

On a different note -- I think the "-s" flag for sort order is a feature of BSD find but not of GNU find. So "-s" will work for OS X and BSD varietals, but likely not for Cygwin or Linux.

According to the man page '-s' tells 'find' to traverse the directories in lexicographical order. It appears that this just happens by default without the flag present with HFS+ filesystems, but OpenZFS filesystems are an exception. I suppose there might be others.

6

Raw Video Postprocessing / Re: Switch for macOS Sierra/Linux (formerly cr2hdr.app)

« on: September 19, 2017, 04:37:18 AM »

For those of us on an openzfs filesystem on OS X (am I the only one?), Switch (as did cr2hdr before it) produces prores files with all the frames running in random order. This is pretty cool to look at, but is definitely not as useful for normal purposes as standard sequential output. This can be fixed by simply adding a '-s' flag to the 'find' commands in the FFmpeg_produce_0*.command files. All this flag does is make explicit that find returns files in 'lexicographical order', which appears to also mean numerical order.

Here's an example of one of the invocations of 'find' from those files, with the '-s' flag inserted:

#export ProRes4444
    find -s "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw +M $h2 $o $S \
-c -6 -W -q 3 $gam $wb $pix $br | ffmpeg -loglevel warning $wav1 -f image2pipe -vcodec ppm -r $fps -i pi\
pe:0 $sd -vcodec prores_ks -pix_fmt yuv444p10 -n -r $(echo $fps / 2 | bc -l) $tbl$cin$cin_01$cin_02$cin_\
03$cin_01b$cin_02b$cin_03b$scale$denoise$sharpen "$out""$out2""$date"_ProRes4444/"$name".mov

I've done the search and replace on all instances of 'find "' to change them to 'find -s "' in my local copies of FFmpeg_produce_0*.command, and now my prores output is watchable. I'd suggest this as a fix for the main branch of Switch but I don't know if the '-s' argument works in non-OSX versions of 'find'.

7

Other experimental builds / Re: 12-bit (and 10-bit) RAW video development discussion

« on: February 05, 2017, 07:01:48 AM »

This is likely a question that was answered earlier in the thread - but I searched around and didn't find it. With this new feature can one shoot 10-bit on the 5D3 at higher than 1080p resolution in full-frame mode (i.e. not crop-mode)?

I've been doing fine recording 1080p23.98 to my Komputerbay 128Gb cards (1 1050x, 1 1066x) at 14-bit, which requires 82.97Mb/s. But if 10-bit is possible, then 2272x1278 @ 23.98 would be nearly the same bitrate at 82.99Mb/s - and that might be a perceptible increase in detail, even if downscaled to 1080p in post, or at least it'd leave a little extra headroom for stabilization. And if 10-bit drops too much usable color information, 12-bit at 2064x1161 23.98 is 82.19Mb/s.

Or maybe the card's capable of more and I'm aiming low with that bitrate, I haven't benchmarked. And maybe card spanning would be a help. It'd be fun to see how high a resolution can be recorded at 23.98.

8

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: December 06, 2016, 12:51:58 AM »

I ran a test on the same MLV file on an HFS+ filesystem and the resulting quicktime was all in order. So I was pretty sure the results I was getting must've been due to the 'find' command somehow returning a list of dng files in jumbled order when run on a directory of dng files that are stored in a ZFS volume.

So I opened all the FFmpeg_produce files and added " -s " to every invocation of "find". I ran it on the MLV stored on HFS+ and there was no adverse effect. Then I ran it on the MLV stored on ZFS and the result was a quicktime file with frames in the proper order. So the find invocations looked like:

find -s "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw +M $h2 $o $S -c -6 -W -q 3 $gam $wb $pix | ffmpeg $wav -f image2pipe -vcodec ppm -r "$fps" -i pipe:0 $sd -vcodec prores_ks -pix_fmt yuv444p10 -n -r "$fps" $cin$cin_01$cin_02$cin_03$cin_01b$cin_02b$cin_03b$scale$denoise$sharpen "$out""$out2"$(date +%F)_ProRes4444/"$name".mov 2> out_01

Just that little "-s" flag did the trick. So that seems to be a solution.

Pretty weird little quirk of find and openzfs. Interesting.
https://openzfsonosx.org/

9

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: December 01, 2016, 08:50:32 PM »

I'll have a look when I get a chance -- am about to have a few days away from the computer. My bet is that for whatever reason the find command is coming back with frames out of sequence for zfs. Maybe I need to change it to something like:

find "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | sort | xargs -0 dcraw ...

or maybe 'sort' wouldn't produce the right order but adding the '-s' flag to 'find' would do the right thing.

10

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: December 01, 2016, 05:24:58 PM »

I've added the folder containing the dng sequence to the dropbox folder. It should show up in there in a couple of minutes.

I'm running OS X 10.11.6.

The MLV file and the directory to which dng and quicktime output are directed are both stored on a thunderbolt RAID that's a ZFS filesystem. I suppose that's maybe a little bit of weirdness I've injected into the process.

What's the command that sends the dng files through to ffmpeg? Maybe I can just have it echo the filenames in order rather than pipe it into a process, and see if it's getting them jumbled and track that down.

11

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: December 01, 2016, 04:53:16 AM »

Here's a folder on dropbox containing a small-ish MLV file, the quicktime file that cr2hdr made from it, and the output text file:

https://www.dropbox.com/sh/76steg8004016hf/AACiMj5cdU_6YcWxtlwj37Uma?dl=0

12

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: November 30, 2016, 09:18:04 PM »

I'm running "Magic Lantern Nightly.2016Sep10.5D3113" firmware on a Canon 5D mk III, capturing to the MLV 2.0 format with audio at 1080p 23.976.  I think the resulting files are 14-bit raw.

I ran cr2hdr on several additional MLV files, and in each case the resulting quicktime file featured random frame order. On the plus side, the dng frames are in sequential order.

I'm in the process of running the version of cr2hdr.app that will print text files next to the dng. I don't see any text files yet and several dng sequences have appeared. I'll leave it running a while.

I'll see if I can find a smallish MLV file that exhibits the problem to post to dropbox.

13

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: November 30, 2016, 01:54:03 AM »

That makes sense.

I ran cr2hdr - it generated a Prores Proxy and Prores 4444 quicktime with the same random-seeming frame order, just like MLP.

On a different note -- is there a way to get cr2hdr to keep its terminal window around so I could watch the output of the commands and thus get visual feedback during processing (might also help for diagnosing problems like the one I'm experiencing)?

14

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: November 29, 2016, 11:27:56 PM »

Thanks for the reply. I'll check with cr2hdr.

After a bit more googling, I came across this page: http://www.cinelogdcp.com/news-updates/2015/1/27/important-information-for-magic-lantern-users
It looks as though the white balance metadata requires the use of a separate matrix for each camera. They claim MLVFS is doing it right and provide a link to the commit that includes the matrices --maybe mlv_dump doesn't use them?

15

Raw Video Postprocessing / Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)

« on: November 29, 2016, 10:08:16 PM »

Apologies if this is covered somewhere in the previous 32 pages of this thread (I looked a bit but haven't read every post), but I've just installed MLP and tested it by using it on a folder containing a single MLV file. I didn't change any of the 3 txt files that appeared in the A_lut_hold folder. I noticed the following, when I compared the results to the output of MLVFS and to a simple invocation of mlv_dump:

• The generated prores4444 quicktime file seemed to be constructed of all the frames running in some random, jumbled, unwatchable order. I've got 8 cores, maybe they ran in parallel and concatenated frames to the file out of sequence? I'm just puzzled by this result.

• When I open the generated dng sequence in Davinci Resolve, its embedded white balance settings are very different than the white balance settings of a dng sequence created from the same MLV file simply using "./mlv_dump.osx --dng" from the command line --and those have slightly different white balance settings than the dng files generated from the same MLV using MLVFS. Any idea why all 3 of these methods would embed different white balance values in the resulting dng sequence? For what it's worth, in Davinci Resolve's camera raw settings for each clip I've got all 3 set to "Decode Using: Clip" "White Balance: As Shot" "Color Space: Blackmagic Design" "Gamma: Blackmagic Design Film" with "Highlight Recovery" checked. Here's an image of what I'm seeing for frame 1 of each sequence:

https://dl.dropboxusercontent.com/s/3yagz5n2lur1ptg/2016-11-29%20at%2012.57%20PM.jpg

Apart from that white balance metadata, the content of the DNG frames appears to be the same - if I change the color settings of the MLP to match those of the MLVFS they look identical. I'm just curious why these 3 methods are all coming up with different metadata for white balance and why MLP's are so different from the other two.

Thanks for any help or information.

16

Raw Video / Re: CF CARD FORMAT WITH EL CAPITAN OSX

« on: September 11, 2016, 07:45:08 PM »

This is only a data point - but I had no luck changing a card's format to exFAT if it was first formatted in camera. But if I just selected the card (not its partition) in Disk Utility and chose Erase, exFAT, and MBR, the result seems to work just fine in my 5D3 running last night's nightly ML.

I'm on OS X 10.11.5, FWIW.

17

Hardware and Accessories / Re: Rolling Shutter Fix: Tessive Time Filter

« on: March 08, 2016, 08:44:58 PM »

Has anyone done any more research into ways of syncing external devices with the camera's frame rate? Just for curiosity's sake, I'm tempted to try and build my own LCD shutter using a lens from a pair of 3D glasses (would only be large enough to cover a very small lens, or to work on the rear element of a lens).

18

Feature Requests / Re: ACES for Magic Lantern Raw Video

« on: February 09, 2015, 07:53:39 PM »

I don't suppose anyone has one of these devices lying around:
http://www.image-engineering.de/index.php?option=com_content&view=article&id=452&Itemid=198

I'd love to have IDTs for the Canon mkIII and to try the ACES workflow in Resolve with DNG sequences captured using MLV, but I have neither reflective targets nor neato illuminating boxes at which to aim the camera. But it'd be neat to have profiles specific to the sensor.

19

Post-processing Workflow / Re: Merging Spanned .MLV Files over 4gb?

« on: November 28, 2014, 10:47:09 PM »

Hah, I'm an idiot. The file's 145.3 Mb, not Gb. So I'm not missing tons of footage. It's also from a shoot earlier this month, not last night's long take.

I'm still not sure the proper way to deal with all these duplicately-named files though. I'll try renaming and see what happens. Are there supposed to be all these files with the same name on my two cards?

20

Post-processing Workflow / Re: Merging Spanned .MLV Files over 4gb?

« on: November 28, 2014, 09:59:34 PM »

Yup, I was able to convert the 4.29Gb M01-1242.MLV and 1.37Gb M01-1242.M00 files from the SD card into a 5.27Gb DNG sequence and wav file. But I'd love to put the larger .M00 file in the same folder as the .MLV in order to extract its frames and audio as part of the sequence (it was a very long take, so 145.3Gb is not a mistake). Unfortunately even though every resource I can find on dealing with spanned files says to put all the parts of the capture in the same folder, those two identically-named .M00 files can't coexist in the same folder unless I rename them -- and if I need to rename them I don't know if that'd make them no longer match the names at which the MLV file addresses them?

Do I need to rename the .M00 file from the CF card to .M01, and will that work? If so, do I have to stick to a naming convention of .M## or can I rename that file to something like .M00CF so that I remember its origin?

Just one batch of files from last nights shoot include 44 identically-named files on SD and CF that are each portions of that spanned shoot, so I'd be having to do a lot of renaming. I wish there was a way to tell MLV_Dump to look in two directories for components of a given .MLV file.

Surely others have run into this issue and I'm just not doing a good job searching the forum for the solution.

Here's a link to the output of "ls -lh" on the contents of my cards from last night in case anyone wants to see the details of all the duplicate files. The first directory is the contents of EOS_SD, the second EOS_CF.
http://pastebin.com/sLWAsTSy

21

Post-processing Workflow / Re: Merging Spanned .MLV Files over 4gb?

« on: November 28, 2014, 07:08:31 PM »

Sorry to pop into this thread so late, but I'm looking for info that seems related. I shot a bunch of spanned MLV raw last night, and I don't understand the process of getting mlv_dump to work on these files because it recorded identically-named files to SD and CF so I see no way to copy them to a single directory unless I start manually renaming files -- and at that point would mlv_dump know which files go with which other files?

For example, on my SD card are:
M01-1242.M00 (1.37Gb)
M01-1242.MLV (4.29Gb)

and on the CF:
M01-1242.M00 (145.3Gb)

I can't copy both M01-1242.M00 files into a single directory next to the .MLV file because they'll just clobber each other. I've got tons of files  This must be a simple basic thing, but I've been scouring the forums trying to figure out how to process these and am failing at finding the info.

22

Modules Development / Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

« on: December 31, 2013, 04:48:58 PM »

Set picture quality to jpeg? You mean in the normal Canon menu? Does that affect video recording for some reason with mlv_rec when it doesn't seem to have an impact on raw_rec?

23

Modules Development / Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

« on: December 30, 2013, 09:05:17 PM »

And just like that, after I pulled the files from the CF card to hard drive and deleted them, it's skipping frames again, still formatted to FAT32.

I guess I'll shoot with the normal raw video and second system sound at a shoot this afternoon, then I'll get back to trying to make mlv2.0 work for me later. It's very exciting that sound recording has been made possible! I'm thinking even a scratch track at 8-bit 22kHz could be used to automatically sync up the second system audio later (pluraleyes/fcpx/etc).

24

Modules Development / Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

« on: December 30, 2013, 08:29:22 PM »

FAT32 FTW!

I reformatted my CF card using OS X Disk Utility to FAT32, and now I can record long stretches with no skipped frames (where onscreen was previously displayed "X Frames Expected" now it just says "continuous frames OK"), and no skips at 4Gb split points.

So some issue must be getting in the way of recording MLV 2.0 to exFat formatted CF. But Fat32 seems to work a treat.

25

Modules Development / Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

« on: December 30, 2013, 07:25:31 PM »

BTW, here's another way to script the processing of MLV files, using tcsh:

% foreach foo ( *.MLV )
foreach? mlv_dump --dng -o $foo:r_ "$foo"
foreach? mlv_dump -m -o $foo:r_meta.mlv "$foo"
foreach? mlv_dump -v $foo:r_meta.mlv > $foo:r.txt
foreach? rm $foo:r_meta.mlv
foreach? rm $foo:r_meta.mlv.wav
foreach? end

I dropped my .MLV and .M00 file for one clip into a folder, then cd'd into that folder and typed the lines above (the "%" and later "foreach?" are prompts, no need to type those).