Messages

We've implemented a new version of dark frame subtraction and flat-field correction in Fast CinemaDNG Processor software. As usual, everything is done on GPU, so it's quite fast.

We can apply flat-field correction (FFC) and dark frame subtraction by providing two corresponding linear RAW images in PGM or DNG format. To prepare these images from DNG files we can use dng_validate.exe utility from DNG SDK.
This is a command line to get FFC image: dng_validate -16 -2 FFC <path\to\dng>

This will extract 16-bit linear RAW data into FFC.tiff. The FFC and dark frame width and height must be exactly the same as width and height of the processed RAW file. Otherwise the correction will not work.
To supply FFC and dark frame files, click "..." buttons next to "Dark frame" or "FFC frame" field. To temporary disable correction, uncheck "Enable dark frame and flat field correction" check box.
To disable whole FFC and dark frame corrections, uncheck "Dark frame and flat field correction" check box in the application processing options.

The latest version of Fast CinemaDNG Processor is here:
https://www.fastcinemadng.com/download/download.html

The latest version of Fast CinemaDNG Processor is here:
https://www.fastcinemadng.com/download/download.html
Now it's based on CUDA-10, so please update your NVIDIA driver before running the software.

@bouncyball
It's not straightforward to port complicated solution from Windows to Linux. We need to port not only GUI, but also full GPU image processing SDK. We are working on that, sorry for the delay.
We do need NVIDIA GPU to offer high performance solution.
Soon we will release new SDK with less requirements for GPU memory, so any NVIDIA GPU will be fine.

@tigs

I am aware of mlrawreview and fast cinemaDNG process, which is a demo.

Fast CinemaDNG is fully functional player for DNG, CinemaDNG, Blackmagic RAW DNG 3:1 and 4:1, MLV. Trial period is 6 months and you can always download and reinstall the latest version of the player. There are no restrictions for the player.

Could you please send that frame for me as well?
Thanks.

We will do that as soon as Apple start to work with NVIDIA. For our software we need PC/laptop with NVIDIA GPU.
https://www.forbes.com/sites/marcochiappetta/2018/12/11/apple-turns-its-back-on-customers-and-nvidia-with-macos-mojave/#12df8b7337e9

We've added dark frame subtraction and flat-field correction to Fast CinemaDNG Processor software. Currently we can use prepared images in PGM format for dark frame and PPM/TIFF for flat-field, GUI for calibration procedures will be added soon.
The latest version is here:
https://www.fastcinemadng.com/download/download.html
to @bouncyball: we do remember about Linux version, sorry for the delay.

@bouncyball
I need to do some clarification here. In Fast CinemaDNG we are using our own CUDA library for image processing, that we've implemented by themselves. Yes, this is quite complicated, but we think it's worth doing.
P.S. Sorry for intrusion. MLVApp is very good application, we have no doubts about that.

Development of the first stable release of Fast CinemaDNG Processor software is over – version 1.0.0 is ready and it could be downloaded here:
https://www.fastcinemadng.com/download/download.html

In the latest release of Fast CinemaDNG Processor we've added new names for denoising algorithms (Raw denoise and RGB denoise), menu for new MLV project and some bug fixes. You can download the latest version from the following link: https://www.fastcinemadng.com/download/download.html

@Kharak
Thanks for your info. Yes, please send me your links to MLVs and screenshots here or to PM, as you like.

First of I cant import anything, when I click to find a file path, be it add files to project or change the 3D Lut Directory the program stops responding and I have to exit. I can basically only choose which features I want, set up the project, go to options and change anything except Path's.

You are right – this is the way to open DNG series from selected folder, but it doesn't work for MLV files. Usually we drag-n-drop MLV file or folder with DNGs from Windows Explorer to player window of our software. Or you can do right button click on MLV file or on folder with DNGs to load data via context menu. Folder for 3D LUTs is correct and this is the path for 3D LUTs that we supply with our software. You can choose another path to a folder with your 3D LUTs in .cube format.

To get direct comparison between the output of MLV_dump and Fast CinemaDNG, please send us your files. Or you can download MLVs via drag-n-drop, it's working.

I also think the WB is skewed after exporting DNG, as the shot I am playing with is daylight WB so around 5500, but to get accurate WB I have to push it to 4800k in Resolve. (kinda like MLV App does when viewing MLV's).

When we do DNG export, we don't change any raw data, WB or color matrixes. Actually we can do that, but we don't. Please show me your example for checking.

As far as concerns denoising, we apply wavelet-based algorithm. If we say "Raw denoise", it means that it's applied to raw data. Usual denoiser is also based on wavelets and we apply it to RGB. You are right, RAW Denoiser should be on the left. We will also check the content in the Help for denoising. Thank you.

I'm sorry, noise suppression for a separate raw channel is not working at the moment. Interface for that solution is quite complicated and we haven't done it yet. The software is ready, but the interface is not. There is also one more opportunity to control raw data by applying raw curves before raw denoising. It's working.

Raw Spatial Denoising" or "Pre-Debayering Noise reduction".

These names are ok, but they are too long for tabs. We will check what could be done here.

Just a general question. Are there any parameters that will not affect the picture if exporting to DNG? will colour space affect the DNG output? Like MLV App has a bunch of parameters that will not affect the image if exporting to DNG as the output will be RAW, how is it with Fast CinemaDNG? Which parameters if so?

At the moment we export DNGs as is, without any raw transforms, we can just change compression method and we can cut raw image. Though some users ask us to include raw curves and raw denoiser to DNG export pipeline. I think we will do that in the future, it makes sense in some situations.

@Danne

I have an idea. Why not help the community out with some of your cpu code so we could speed up previewing in Mlv App?

Sorry, this decoder is not open source. Soon we are going to release open source application PGM2DNG to convert raw PGM images to DNG. I will publish the link to github as soon as it's ready.

@bouncyball

BTW as I understand lossless decoding in fastcinemadgn is realized on CPU not GPU? You mentioned somewhere that you reworked the lossless decoding and it is way faster now. Am I correct?

Yes, this is correct, we do decoding on CPU. We've implemented our own algorithm to accelerate Lossless JPEG decoder on CPU. You can see more info and decoding benchmarks at https://www.fastcinemadng.com/info/jpeg/lossless-jpeg-decoder.html

forget fast cdng as it is not for everyone. we need nvidia 6*** series to use fast cdng. it is totally useless for others who dont have nvidia 6 series

Not everyone needs smooth video output. Realtime cdng processing and denoising are not right things either. That's true.

Fast CinemaDNG takes it longer because we also need to allocate GPU memory and this is not fast procedure.

By the way, why wouldn't you add that index to MLV format? Everything is in your hands - just add to MLV specification such a feature at the end of file or you could allocate some space at the header to store offsets for each frame.

@12georgiadis
For your suggested workflow we need the following for testing: several MLVs, corresponding proxies, and fcpxml that you got from these proxies. Could you please send us several examples?

@bouncyball
Yes, August has gone, but Linux solution is not yet ready. I'm sorry about that. I will let you know as soon as it's ready.

@12georgiadis
Thanks, I've read this thread. I will write here when we could do that.

@12georgiadis
Thanks a lot for your info. You can use my email from https://www.fastcinemadng.com/contacts/contacts.html
or you can send PM to me.

@12georgiadis

Could you explain more the advantage of a server version?

That solution is not yet ready, so it's not worth speaking about advantages. We are working on the software which will work as a server, without GUI, to process big quantities of frames on one or several GPUs. As soon as we could have many different workflows, we need to develop scripting to be able to cope with different tasks. This is concerned not only with mlv/dng, but also with much more solutions for massive image processing.
We already can process mlv/dng over network, so there is no need to copy raw data to your local PC if your network is fast enough.

If you need some case examples of workflow, I can send it to you. I worked for feature and shorts with some very original & exotic workflows but in all cases, for time and money, offline/online was used.

Thanks for your suggestion. We will need your examples, and it would be better to start from standard, non-exotic workflow.

For cameras without h264 proxy's, the ideal workflow would be:
1) shoot mlv
2) import in fast cinema DNG and export all footage to ProRes proxy (36mbit) (it's one step more than with 5dmkiii)
3) edit in NLE
4) export the editing in xml/aaf/fcpxml
5) import xml/aaf/fcpxml in fast cinema dng, pre-correct, export ProRes c-log (444 or above)
6) conform in resolve with ProRes c-log, grade and export master.

There is also an opportunity to create MP4 with our software and FFmpeg at step #2,  then you won't have any problems with storage.

It is a good summary of my previous hypothesis. I perfectly understood that resolve is the problem and that your software is the fastest to process direct dng/cdng.

If this is the case, why are you going to process dng/cdng in Resolve? You can process mlv at FastCinemaDNG and get output with any intermediate codec to work with further. This is the way to bypass mlv reading and dng debayering at Resolve. Yes, you will need more HDD space (though not too much due to compression), but you will not work with mlv/dng at Resolve anymore. Why don't you consider that approach?

There is stills another way to bypass resolve's problem. If you add a feature that import xml/aaf/fcpxml and reconnect to the mlv, it can be amazing.

I agree that this is one more possible way to solve the problem, but it's not simple. We will need to get into xml/aaf/fcpxml formats and at the moment we don't have such experience.

1) if I shoot 100hours of shooting and at the end I have a 1hour film, do you think it's a good idea to process all Mlv files in dng before the edit ? No, especially if you have proxy h264 (for now only with 5dmkiii but it could be generalized if lossless 8...11 is done on digic4/5/6 caméras). This way you can edit with it and save storage + time!

I think that still it's a viable idea, as soon as there is a reason to preview all your footages with maximum image quality and at full resolution, not just with proxy h264.

2) so the reliable solution is to process direct mlv in your software with only the clips that have been edited in the timeline after the editing in proxy h264. To reference automatically the clip used in the timeline, we use Xml / aaf / fcpxml. It avoids the need to reference by hand and one by one the clips and then import it in your software. If you add an import feature with xml/aaf/fcpxml, we can have all the used clip in the batch list automatically. A timeline of one hour, it can go up to 400 clips. So, it's better to do it automatically, like it's done in cinema since 1990 with the Analog roll / proxy video clip and EDL/avid.
Ideally, we could pre-grade these clips and export in ProRes cine-log to have good performance in resolve.

I agree that this is an ideal case, but I can't tell you when it could be ready. Currently we are also working on server version of Fast CinemaDNG and it will work with multiple clips, multiple GPUs and with scripting. Your suggested workflow in not included yet, but I think that it could be possible.

Ok very interesting answer. I have another proposition:
If you add a feature to import xml/fcpxml/aaf, we can import the timeline's information at the end of the edit and detect by the clip name only the clips that have been used in the edit. This way, we process the edited files and still save a lot of storage (for a doc, we often use 1/30 of rushes or less). Plus it's automatic and a sort of conformation / consolidate combo. Do you think it's possible?

We understand that with MLVFS you have just one source, which is finally processed with Resolve, so you spend less HDD storage, but processing is slow due to both MLVFS and Resolve.

As we see, you would like to get fast MLVFS to solve your problem. With such MLVFS you will still process DNG not very fast due to Resolve. Unfortunately, that feature is outside the scope of our software. We process both DNG and MLV directly. For your task you can temporarily save DNG series to process them fast with our software.

The problem is that MLVFS is slow because it's CPU. The idea is to have a GPU solution for on the fly CDNG, because Resolve won't read MLV. And your software has good settings, good corrections and good performance CDNG, this can be a very good solution. Also because MLVFS doesn't have all the options that you have.

I don't think that MLVFS is the slowest processing module. If you disable focus dots removal with chroma smooth, the only task for mlvfs is to find frame data block on disk, to generate dng header and to send it to application. These stages are not CPU hungry, their speed is rather limited by HDD performance.

DNG processing is slow because of demosaic and denoise stages. High quality demosaic and denoise algorithms are CPU hungry. We've managed to develop extremely fast and high quality demosaic and denoise GPU filters. That's why dng processing in Fast CinemaDNG is faster than in Resolve. And this is the reason why it is useless for us to boost MLVFS with GPU. The most heavy tasks are perfomed by rendering application, not by MLVFS.

Fast CinemaDNG can produce synchronized proxy, CinemaDNG and audio from MLV. But still you have to process DNGs in Resolve, that wouldn't be as fast as mp4, h.264 or ProRes processing.

@12georgiadis

What could be improved with fast cinema DNG ?
1) generate ProRes Proxy via FFmpeg with a script that timestamp the same TC on proxy than on the MLV. Most cameras cannot generate proxy mlv in the same time (all except 5DmkIII) + script for 5DmkIII that correct black off frame from proxy & timestamp same tc on proxy than MLV
2) a system to generate on the fly CDNG with GPU rendering to speed workflow with resolve ? Or if not possible, generate CDNG that maintains the same TC and synch them to the proxy file (H264 or prores proxy)

Here you can download such a draft version:
https://yadi.sk/d/TTvMJIEKzCTVT

Please note that for FFmpeg you need to add the following to the command line:
-timecode 00:00:00.00 if fps = 29.97
-timecode 00:00:00:00 if fps = 59.94

As we understand, you will do the following:
Export CDNG series with FastCinemaDNG
Export to mp4 or anything else via FFmpeg (from Fast CinemaDNG)
Do the rest in Davinci or Final Cut.

Please let us know whether this is correct or not.

@theBilalFakhouri
We've fixed bugs with mlv audio and added .wav for MJPEG export option:
https://www.fastcinemadng.com/download/download.html
If you see any bugs, please let me know.

Sorry, we've fixed that bug for CDNG series, not for MLV. If you do fast export from CDNG to MJPEG, then wav file is also exported to the same folder. Will try to do that for MLV tomorrow.