Messages

Regarding any problems with split MLV files or color-ext color spaces having washed out colors, please update to v2.0.3 .

Hello,

I received a project that was edited in fcp7 and needs to be
Conformed to resolve. The proxy editing uses .jpeg exported from regular dng (original are ml raw) and dng converted from .arw (Sony). When I import the xml, the dng are not relinked from the proxy jpg. It's not the way I'm normally doing things but that's what I received. Any help is welcome. Thanks!


Envoyé de mon iPhone en utilisant Tapatalk

What do you mean by relink? It seems as though you developed a JPG sequence, which are different files, from the raw (un-demosaic'ed) DNG sequence. You'll probably need to find the original DNG sequence and put those through Resolve.

Sent from my SM-G930T using Tapatalk

@michael08 I ended up fixing a pretty niche issue that was affecting you. Basically, globbing was turned off in your shell! So running something like:

Code Select Expand


for f in *.dng; do echo $f; done


Simply doesn't work when globbing is off - it never expands the "*.dng" into all the DNG files. Which is a big issue, not just for convmlv.sh, but probably for any kind of shell script you try to run.

The solution is simple. In your shell, before running convmlv.sh, run:

Code Select Expand


set +f


That turns globbing back on - as it should be. I pushed a new commit to the repo, where I run 'set +f' at the beginning of convmlv.sh.

If one were to want to turn it off (as was probably done in your case), that'd be 'set -f'. But don't do that, please!

how do you think the problem is? on which particular Linux system do you work? I have an ubuntu installed

You messaged me about this, but I'll respond here.

You're trying to install convmlv's dependencies with:

sudo apt-get install $(convmlv -K 0)

That's only valid for the Debian dependencies, however, which won't be found on your Ubuntu system. You're on Ubuntu, and should try running:

sudo apt-get install $(convmlv -K 1)

I admit, this nuance could be much more clear in the docs. 0 gives you Debian deps, 1 gives you ubuntu deps, 2 gives you Fedora deps, 3 gives you Mac deps. I'll see about updating the PDF.

Sent from my SM-G930T using Tapatalk

[Caveats:]

Hey @arboldeconfianza ,

You're very welcome Glad it can be of use! Adding a LUT is quite simple - just specify -l and the path to the LUT. It can also be done in a config file (see the documentation here: https://github.com/so-rose/convmlv/releases).

To develop an MLV into an MOV while applying a LUT:

Code Select Expand

convmlv -m -l <path to lut> <path to MLV>

The only way to preview it is to develop a single frame (the 1st for example) of the file. The command below will give you a PNG with the LUT applied:

Code Select Expand

convmlv -i -t 2 -l <path to lut> -r 1 <path to MLV>


Caveats:

• Only 3D LUTs under (NOT INCLUDING) 65x65x65 size work, currently - this is ffmpeg's fault.
• I haven't found any other tool to work with LUTs, but have instead been developing my own - openlut (https://github.com/so-rose/openlut) - to work with 1D LUTs, 3D LUTs, pure gamma formulas, color matrices, etc. . It's not mature enough yet to integrate, but hopefully soon (It recently got 50x faster, so progress ).
• Before openlut is integrated, this limitation won't disappear.

[entire]

The DNGs that come out of MLRawViewer have never worked for me - last I tried, it had a pink cast, suggesting an incorrect black level in the metadata. These things are unlikely to get fixed, as the project is deprecated.

MLRawViewer is still a great previewing tool. But the good news is, there are a lot of quite mature projects here that solve your entire workflow problem:

Any platform, MLVFS gives you clean working DNGs. Take a look: http://www.magiclantern.fm/forum/index.php?topic=13152.0

Windows: MLVProducer is great. http://www.magiclantern.fm/forum/index.php?topic=15271.0
Mac/Linux: MLP is wonderful. http://www.magiclantern.fm/forum/index.php?topic=13512.0
Mac/Linux: Shameless advertising convmlv does a lot of stuff. http://magiclantern.fm/forum/index.php?topic=16799.0 (my personal workflow is, shoot MLV, convmlv, edit)

Setup of these takes some effort and research, but it's certainly worth it for the final output. No need to shoot in .RAW!

Link to MLVFS seems broken? Also for the Python indicator?

[Make a feature request as an issue on GitHub]

Hello!

As I was developing color management for convmlv, I ran into a lack a lack of free/OS tools to handle and apply LUTs (especially 1D LUTs), Color Matrices (for gamut transforms), and just simple gamma functions, to/from production image formats (especially 16-bit EXR files). I will be updating convmlv to use this soon.

Essentially, this solves that problem as a Python 3.5+ library for practical color management ("apply this LUT to this image and save it!"). Take a look at the feature list!

A big thanks to @Danne and @bouncyball for testing & feedback !

Download

Download the zip from GitHub: https://github.com/so-rose/openlut

Features

Code Select Expand


* Functional/OOP design scales from simple one-liners to complex systems.
* Load & save images in countless formats, including EXR and DPX, at up to 16 bits per pixel. The guts (numpy array) are easily accessible.
* Create 1D LUTs easily, from arbitrary contrast functions, arbitrary mappings, and more.
* Load and save 1D LUTs in the .cube format.
* Create, compose, invert, combine, and apply color matrices easily for gamut transformations.
* Apply any Transform, including LUTs, Matrices, and perfect Gamma functions, to any loaded image with ease (the .apply() method).
* Batteries included - comes with an image viewer, functions for sRGB, sLog2, Reinhard, Rec709, and more, as well as matrices for XYZ, sRGB, and Adobe RGB.
* Fast - transforms themselves are done in parallel C++ bound to Python.


More to come! Make a feature request as an issue on GitHub.

Info

See the GitHub page for the latest info on dependencies and usage. I'd rather just maintain that than also maintain this post .

I suggest looking at the test code at the bottom of the GitHub page.

If you need 3D LUT support today, take a look at pylut. It should fit your needs!

Plans

• 3D LUT support. A hard thing, but it would be a good thing!
• CLI Script. For those who don't know Python!

See Issues in GitHub for a more comprehensive list.

Feedback
I'd appreciate any feedback and testing (especially bugs)! I'm very happy to try feature requests too (within time & reason, naturally).

What about baking in an SLog curve manually into the DNG - wouldn't this "trick" RAW developers to output an SLog2 image? Would that preserve sensor detail?

Sent from my Nexus 6 using Tapatalk

All right, @nicholas_carroll, I fixed your bug! In fact, I fixed some other related bugs too, as well as some gotchas with DNG sequences. Let me know if you have any problems!

welcome back @so-rose!

Thanks! I bring features . Probably just made more bugs, though... This is really becoming a beast of a program at 1500 lines! 

Ah, been away for a while; apologies!

Hey so-rose, I am also using Blender and wondering whether you find limitations when using image sequences? For example, I found that the bundled Import Images as Planes takes movie formats but not image sequences.

First of all, vanilla Blender seems (in my experience) designed to work extremely well with image sequences over movie files. Especially exr sequences are very fast; the decoder for mov files isn't so great, on the other hand.

Second, it seems you've found a bug with DNG input (you're doing it 100% right)! I'm on it 

I looked back and noticed that the "Embedded Color Profile" was "None" - perhaps Quicktime, and Photoshop for that matter is too quick to assume that it's sRGB?

If values are being crushed, it leads me to think that the data is all Linear, nice and intact. Perhaps somehow modifying the image header to report a linear profile would help such programs along?

Sent from my Nexus 6 using Tapatalk

You could always put your images through Blender's compositor to process the input linearly (serving out correctly marked exr's, or videos, for DaVinci, etc.) - lots of tutorials covering the basics, here's the relevant bit (scroll down to Image Files): https://www.blender.org/manual/render/post_process/cm_and_exposure.html

Just a suggestion

Sent from my Nexus 6 using Tapatalk

Just updated; I ran into an interesting issue: How to transform a relative path into an absolute one.

On Linux, this is easy; readlink -f "relpath" does the job. Does anyone know if something similar exists on Mac?

Wasn,t aware you could change black level with dcraw.

DCraw isn't seeing the BlackLevel tag in the DNG at all, at least on my build of it. The level is correct in the DNG, usually around 2048, but DCraw doesn't automatically integrate it (a bug I expect). However, I'm able to specify the -k option, to manually override the black level that DCraw tries to develop with with the black level I read from the DNG.

Tl;dr I avoid a dcraw bug using exiftool.

The non universal solution(but faster) would be adding black level through hex. Dfort already cracked those numbers for white level.
Did you compile dcraw yourself?

*that was fo mlv files not dng files. My bad.

I'm just using the latest dcraw available in the Debian repository (testing), inserting the extracted BlackLevel tag after the -k option.

I only extract that value from one DNG, however - does it change? You mention that adding the black level through hex would be faster, but wouldn't I still have to specify -k, thus slowing it all down regardless?

Sent from my Nexus 6 using Tapatalk

@Danne I did, with no luck... I ended up exiftool'ing the Black Level from the developed DNG (works on Dual ISO footage as well), so that the user doesn't need to deal with it - this seems to be a universally usable solution.

It's interesting that the bug should be present in ufraw too. The dcraw on Debian at least has the bug.

Status update: I'm trying to modify mlv_dump! No idea when it's going to work, but work it shall!

In any case, I finally made my script aware of Black Levels. It may interest people to know that dcraw apparently doesn't read Black Level info from the DNG file it's processing, leaving a magenta/green color cast. It's all fixed now, though; I compared the output to Darktable and MLRawViewer to be certain.


Thanks for all the support everyone! 

[before]

@a1ex Absolutely! In fact, I did some thorough testing for you. You can scroll down to the conclusion if you wish!

Conditions:

• Global Draw was in LIVEVIEW mode (this is what is meant by ON), but "the works" (zebras, etc.) was turned off.
• "the works" refers to features of Global Draw: Zebras (LumaFast, over 99%), Spotmeter (Percent, AFbox), Histogram (RAW RGB, Log), Waveform (Small)
• I recorded at 2240x954 with an FPS override at 23fps.
• Active Modules: mlv_rec, mlv_snd, mlv_play

I recorded before the auto-shutoff:

• Global Draw ON, ALLOW during recording: Works
• Global Draw ON, DISALLOW during recording: Works

I recorded after the auto-shutoff, where I turned it off then on (sometimes this has issues, I think at least):

• Global Draw ON, DISALLOW during recording: Works

I let it auto-shutoff again, pressing a button to bring it back to life (not touching the power button):

• Global Draw ON, DISALLOW during recording: Works

At this point I tried turning on "the works" (simulating the conditions where it broke):


I let it auto-shutoff again, pressing a button to bring it back to life (not touching the power button). I activated "the works":

• Global Draw ON w/Features, DISALLOW during recording: Breaks

Clearly, it's the combination of LiveView Global Draw and either zebras, spotmeter, histogram, or waveform, inconjunction with auto-shutoff. All of which combine quite commonly in production situations!


For each broken test, I removed the battery, reinserted it after 5 seconds, then turned it on, off, on (modules loaded), off, on - as my instinct says resets any issues! Afterwards, I let it auto-shutoff for each test, then repeat for the next one:

• Global Draw ON w/Features, DISALLOW during recording, Zebras OFF: Breaks
• Global Draw ON w/Features, DISALLOW during recording, Zebras OFF, Spotmeter OFF: Works
• Global Draw ON w/Features, DISALLOW during recording, Zebras ON, Spotmeter OFF: Works
• Global Draw ON w/Features, DISALLOW during recording, Zebras ON, Spotmeter ON: Breaks

So, my conclusion is this: The Spotmeter feature of Global Draw is responsible, in conjunction with 3x Crop recording after the 7D has auto-shutoff, for the issue. Personally, I'll never use Spotmeter again .


But I hope I've been able to illuminate where the bug is in a reproducible fashion!

Still unable to reproduce your crashes.

@kamranjon I've been getting exactly this issue on my 7D, so I feel I should revive the thread. It happens on the newest and oldest builds alike:

CRASH00.LOG

Code Select Expand

ASSERT: !IS_ERROR( TryPostEvent( this->hTaskClass, this, EV_VD_INTR_LV, NULL, 0 ) )
at LVState.c:485, task ?
lv:0 mode:3


Magic Lantern version : Nightly.2015Aug19.7D203
Mercurial changeset   : 8d48d164e83a (unified) tip
Built on 2015-08-18 22:31:00 UTC by [email protected].
Free Memory  : 213K + 2686K

@Walter Schulz I have about 12 more similar log files from crashes, on a newer firmware as well (from March 29). I'll try to describe steps to reproduce as best as I can! 

• I shoot MLV's in 1080p crop, FPS Override at 24 (effectively 23.976). I have 64GB UDMA7 (160MB/s) SanDisk CF cards, both of which exhibit this issue.
• I installed ML on both CF cards freshly the same day, using the August 19 build, zeroing them as I formatted to FAT.
• It seems to happen when the camera turns off by itself; I awaken it, begin, and see ERR. It also seems to happen if the camera turns off, I turn it on, then off, then on. Three times fixes any issues.
• It remains paused on 0:00 recording time, the indicator red as if it's filling the CF card, until it either freezes, showing the above error, or until the camera seemingly shuts off, displaying "ERR" on the top right.
• I can fix the by taking out the battery, taking out the card, assembling it again, then turning it on, off, on, off, then on again (taking care to leave 5 seconds between each step). If any step is skipped, the issue persists indefinitely.
• Global Draw was on. Did not get to test if turning it off helped.
• Most of the time, recording functions fine. But seemingly if the camera is allowed to turn itself off, everything goes to hell.

I have no temperature issues. Tl;dr it seems to my untrained mind to be linked to the "auto-off" feature on the camera.



I hope a fix is possible! It's certainly a very debilitating issue (the director got pretty annoyed  luckily we got the footage we needed)!!

May the force be with you, @so-rose!

Many thanks!

the focus pixel fixing probably needs to be added directly into cr2hdr

Seems like a challenge. I shall dive. No promises...



...I opened up cr2hdr.c, and cs.c from mlvfs, and I now fear for my life ! Though I'm incredibly unsure how to do it, I have an idea of what I need to do (and quite certain I can do it due to the magic of GPL):

From cs.c, I need to take the function fix_focus_pixels(struct frame_headers * frame_headers, uint16_t * image_data, int dual_iso) and all that it needs to function.

This I need to plop into cr2hdr.c next to (or in?) find_and_fix_bad_pixels(int dark_noise, int bright_noise, int* raw2ev, int* ev2raw), in such a way that fix_focus_pixels executed as a step.




The issues I have even at this point are numerous:

• What is the uint16 array image_data, and how might I set the output's pixel data in cr2hdr.c?
• Are the raw2ev and ev2raw integer arrays the same in cs.c as in the argument of find_and_fix_bad_pixels()? Oh and what is it  ?
• Several attributes of the pointer frame_headers are referenced, including what seems to be crop values (frame_headers->vidf_hdr.panPos<X or Y>). raw_info in cr2hdr seems to be the corresponding struct, but I can't find any "pan" values. Something that looks promising is found on Line 774 in cr2hdr.c, where the script sets an attribute called raw_info.active_area.<x/y 1/2>.
  
  I assume focus pixel handling is different based on whether the footage is taken with a cropped frame or not (that's how it works in mlv2badpixels.sh). If so, where can I find or get/how can I use what I have in raw_info to get these "panPos" values that I don't have, but seemingly need, to power the focus pixels script?
• What is the parameter dual_iso?

@dmilligan and @anyone Any ideas?

If anyone's interested, I wrote some documentation for convmlv! It's a PDF found in docs->docs.pdf in the source code (https://bitbucket.org/so-rose/convmlv/src).

the focus pixel fixing probably needs to be added directly into cr2hdr

I'd be happy to give this a go at some point in time, but as I've never coded in C (only simple code in C++, and never image processing outside of Python), I'm not sure I'd have much success - would anyone be up for trying this?

in conjunction with DarkFraming Avg Process (Dual-ISO, Focus Pixel Fix and DF) all together ... My fault for the typo so-rose!

No problems  . Combining MLP and MLVFS does, at the moment, give the best results as far as I can tell; it seems like a bit of a roundabout way to do it, however. Implementing the focus pixel fix in cr2hdr would allow MLP (and convmlv for that matter) to process those three features - Dual-ISO, Focus Pixel Fix and DF - all by itself!

Nice work guys. Been following this thread and as it seems to get best results with Dual-ISO and Focus Pixel to work together is to combine MLP and MLVFS atm, correct?

Thanks .

It would seem that MLVFS has the golden ticket to badpixel removal in Dual ISO footage, so in a sense yes. However, I'm not quite sure what you mean by combining the two programs - if you're suggesting working on integrating the pixel blending code, then I agree fully! convmlv would benefit from such work as well.

[any difference]

@Danne I figured out darkframe subtraction! It's in 1.7.1; thanks for the pointer!

dmilligan worked out a pixel blending system specifically for Dual ISO. It should be possible to do something similar with dcraw but that involves some C coding and creating a custom dcraw build. Better yet, it is also be possible to add the focus pixel fixing code into mlv_dump. That would be a better solution because it would fix the focus pixels on the DNG files like MLVFS does.

A custom dcraw build specifically for Dual ISO is starting to sound a little scary, though it makes a lot of sense if focus pixel removal could be integrated into mlv_dump - or perhaps cr2hdr?


A question for @dmilligan: Looking through the MLVFS source code, I notice a function in 'cs.c' named 'fix_focus_pixels' and another named 'fix_bad_pixels'; I wonder how possible/painful it would be to isolate these little stumps of magic, using the resulting binary to simply take in a bad DNG (from the output of cr2hdr) and output a fixed one?



Also, open question, is there any difference between cr2hdr and cr2hdr20bit? If so where can I find/compile cr2hdr20bit?