Messages

[Technical doc]

Technical doc
- dual_iso.pdf (in-depth description of how it works)
- See also: ADTG and CMOS discussion

The technical doc link is down. However it can still be accessed here if anyone needs it: https://web.archive.org/web/20130821183804/http://acoutts.com/a1ex/dual_iso.pdf




@Milk The developers would have done it if there was an easy way to make the camera do it! 10 bit log would be almost indistinguishable from 14 bit.

Thank you so much for the in-depth reply and explanation.

@g3gg0 Good to know. And how would you interpret the LGPL in this case? Would you say it requires a 'relinking'/replacement mechanism for MLV structures? Or simply the inclusion of a license text+source?

Also do you know about:

How did propreitary apps like Martin Hering's colourcast, MLVProducer, fastcinemaDNG deal with this? Did they provide any way to relink with/replace mlv.h?

well, using MLV structures, like the C structures (and also the function prototypes if you wish) was always meant to be LGPL.

Does this statement include the latest version of mlv.h (with GPLv2 on it as names_are_hard says), or only the LGPL file released a while ago?


If you prefer discord I can switch over, just need to restore my login.

The use of an earlier version wouldn't cause incompatibility, but some newer MLV blocks would not be supported. These new blocks are not critical for reading the image data correctly, but communicate things like stretch factor for anamorphic modes (don't remember all this stuff in much detail, perhaps there is now critical stuff).

MIT/BSD vs GPL is an interesting argument vs adoption.  Companies are more wary of taking GPLv2 code, but, if they take MIT/BSD code, they don't tend to cooperate with upstream - there's no need.  They're also allowed to modify your code (making incompatible MLV files) and not share those changes, which would suck.  Linux is GPL and Intel contributes driver code to them directly, for example, because GPLv2 means that's easier for them and it means Linux servers buy more Intel kit.

True, I guess we'd just have to hope no one makes custom propreitary MLV files.

I think the dual licensing is a great temporary solution.

There's a version of mlv.h released under LGPL somewhere a few years ago by the main devs (of course it's not up to date).

Licensing with a permissive license would give the best chances of getting wider MLV support possibly even in propreitary software. Although MLV having popularity is looking less likely as the years go on, for many reasons including creation of propreitary raw formats with strong compression (BRAW prores raw etc).

My personal, not-a-lawyer guesses about the API / structures part: structs are not APIs and are much more likely to be copyrightable.  The layout of a struct is not "obvious" in the sense that there is only one way to write it given a specification.  Structs definitely are code (as are APIs once they are written in a given language).  An instantiated struct, e.g. a global, does become bytes in object code.  Linking against such an object file would probably be okay if the code was LGPL licensed, but as far as I can see, mlv.h is not LGPL.

If MLV were standardised and the structs documented in a PDF document, then the structs could be re-written GPL free, right? Even though it's gonna produce the same code as the GPL-d structs.

Propreitary software using LGPL libraries (dynamically linked) will contain object code shaped by function definitions, and possibly even structures from the headers - this seems tolerated with LGPL, even though it limits how much change you can make in re-linking.

(so many ways to see this 😭)

@Ilia: As long as MLVApp doesn't use the GPU, it can't be slowed down by CPU->GPU upload.

It has to upload to the GPU to display the image though, I think that is quite slow (it was my main difficulty in the Cocoa GUI), I assume it must be happening in Qt behind the scenes?

@masc Amazing to see! Has anyone ever achieved such high MLV App performance on computers other than the M1?

I suspect the M1 might have a huge advantage for MLV App over anything else out there, due to GPU and CPU sharing the same, extremely fast, memory (MLV App is usually slowed down by CPU->GPU uploads I believe).

Ah yes, I forget that I do actually have a discord account and I am in the magic lantern server.

Should I ask on there or message directly to g3gg0?

I want to continue work on LibMLV, I plan to copy the Lua API style.

EXIF could fit into the scheme.
...
what should it specify, what you do not have in the main blocks?

EXIF could do a lot: stuff like GPS, more than one colour matrix etc... Tonnes of stuff that's easier to outsource to EXIF than to implement in the MLV format ourselves.

but as you said you have to make sure that stuff doesnt cause redundancy.

It could cause some redundancy, if shutter speed and aperture gets in there, but it's not a concern as long as we keep this rule: An MLV decoder should be able to operate fully (extract frames) without using the EXIF block.



Again, about the licensing:

I would like to release LibMLV under MIT (permissive license), but how do we deal with the LGPL mlv.h? I think mlv.h is not code, it is structures.

The LGPL states that the user should be allowed to relink/replace LGPL components in a propreitary product. Problem: it would be much harder to relink header files containing structures. It would require dynamic loading of the header file.

As the American court ruled that APIs cannot be copyrighted (like the implementation can), and in my opinion, a header file with structures is much closer to API than implementation - it tells the compiler memory offsets and types, but does not become instructions, I think mlv.h may be used without LGPL restrictions. Does anyone agree?



How did propreitary apps like Martin Hering's colourcast, MLVProducer, fastcinemaDNG deal with this? Did they provide any way to relink with/replace mlv.h?

Yea I've written a separate program(i have to recompile to change any parameter)

It takes 1 minute to process a 20mp photo, and still has some issues with colour

Haven't actively been working on this but I always appreciate more samples.

Edit:

Just ran the images from Levas through the latest version of my little program:



"camera matrices" and everything

This looks great, from the very first shot. Very little moire problems nice colours, good shots, everything.

Are you using MLV App? If you are, you should enable highlight reconstruction to fix pink highlights (which you can see on the right side of the very first shot).

Nice! Is this only for new cams?

What about DIGIC 5 and (like 5D mark 3 or 7D)

Seriously this is potentially amazing.

I see that the cameras have 4GB of address space (32 bit), and that different parts seem to be mapped to different things (?)

The cameras obviously don't have 4GB of ram, only 128mb-1gb.

And I see the graphs here: https://www.magiclantern.fm/forum/index.php?topic=5071.50, is this just for normal RAM/main memory? I see that all of the graphs start at 0x40000000, which is 1GB offset from 0. What about firmware/ROM code, where in memory is that placed and is it in ram, or some special memory? I know it's not a direct mapping to the ROM, as you can't just write to that, but it does get copied to somewhere while the camera runs that isn't main memory??


I am curious to know what address ranges are mapped to what devices, and what can be done with them in terms of read/write/execute.

I could not find anything about this topic as a whole, only little fragments that I can't piece together.

Also what is ROM0 and ROM1 about? Why are they separate and mapped to different addresses... and why is there a 'mirror' double mapping or something

Pretty!

I think there is nice motion here, goes well with the music.

Remind me again what 3x3 means, full sensor or cropped in? Anyway looks beautiful, how did you process it?

Notice the sun going a little yellow at 0:18 though :0

Could you give me a sample MLV where that happens?

@Levas thanks again for the samples. They are extremely helpful for designing highlight reconstruction algorithms, and improving difficult colour reproduction.

Haven't had much time yet, but I finally got a new highlight reconstruction algorithm working... I would describe my highlight reconstruction algorithm as "ratio inpainting". Still glitchy and nowhere near finished though.

For highlight reconstruction, you should need some sort of minimum value in other channels to perform a nice detail reconstruction. If the value is below that threshold, it should not even try to reconstruct detail from other channels.

I thought about this, but it seems if one channel is bright enough to be clipped, the other two always have enough information, even with very saturated light, so no minimum is even needed, at least I haven't found an example.

Rawtherapee with "colour propagation" highlight reconstruction (left) vs my algorithm (right):


This image shows the worst artifacts of I've seen, very valuable.

Another image for comparison:



The red skews a little too yellow imo, and there's a weird cyan artifact, but an improvement over old mlv app

Considering he shoots 5-10 megapixel JPEGs on most cameras, this is a great idea. Ken has found a way to make high resolution sensors useful in his world. So congratulations to him

More great samples! thanks again @Levas

Very very nice looking footage. How did you add the grain? Also did you use the MLV app film effect thing? Or a lut?

@Levas thank you so much, thosee examples are great for testing highlight handling

Do you have any especially disastrous images ruined by blue/violet light?

We have this new RGB LED spotlights in a park, I'm gonna take a walk and shoot some clips today, will try to catch this negative-blue effect.

Sounds great!

I don't think you can directly translate the hue adjustments in Adobe profiles to MLV App's hue adjustments, as Adobe does hue adjustment in some way which we don't replicate exactly. What you do manually is probably no worse than what Adobe has in their profile. This stuff is just a bodge by Adobe to try and improve on 3x3 matrices imo.

@2blackbar the 5D2 has weaker red saturation compared to most cameras when using a 3x3 matrix, this is an issue in all raw converters that use these same matrices by Adobe. However Adobe's own profiles include some hue-based adjustments on top of the matrix, which we don't use, but that's how they correct the reds I believe.

@a1ex Thanks for the samples, exactly the kind I'm looking for. Did your highlight patch get merged to UFRaw?

Possibly related: https://twitter.com/CT_Bergstrom/status/1333614982647353347

This one is amazing Must be related. He provides raw files yay.

@Walter Yes, I would like to see pictures from any camera brand if it has interesting artifacts.

@Skinny I had forgotten that the black sun effect was a thing. Not sure of the best way to fix it other than lowering white level. It might also become less noticable with some MLV App changes I plan to make

The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

This post by the Oklab guy shows how much CIELAB skews purple between blue and white, definitely an outdated colour space that shouldn't be used anymore.

I have been experimenting with highlight desaturation/roll-off recently in Jzazbz, Oklab and xyY, all work quite well, but xyY is a little bit worse for blue. I want this to replace MLV App's "tonemapping" curve stuff.

Does anyone here ever have raw images with ugly clipping? Be that simple highlight clipping, blue light clipping in weird ways, or red lights going yellow, anything...

I'm looking for examples of these kind of images, they can be from any camera (from any brand) and in any raw format. I want them all. Let's make a collection of raw files with difficult colours. It would be really helpful for developing and testing raw conversion software (MLV App in my case)

Here's a particularly extreme example of what I want:



Would love more like this!


Fun fact: Using 3x3 matrices for raw conversion (which almost all software currently does) produces negative Luminance (Y) values for pure blue light. So why doesn't blue show up as black? ... per-channel clipping! (most often in ProPhoto RGB)




This sums up what I am looking for very well:

In this thread, we are looking specifically for sample images with extreme out-of-gamut clipping issues (extremely saturated colors that are rendered obviously wrong). Such issues can be found when using highly saturated light sources such as colored LEDs, or with concert photos/videos (with unusual lighting), or highly saturated scenes/subjects, such as flowers.