Reverse Engineering Picture Styles

[DeafEyeJedi]

So if you're saying that if you delete a custom picture style it fills up that block with zeros then that's a great clue to help figure out where the picture styles are stored by comparing a ROM1.BIN with and without custom picture styles installed.

This smells legit to me... Thanks for the wake up call @ItsMeLenny!

[ItsMeLenny]

So, this is in between the camera and host sending little bits of data to each other continuously in EOS utility, probably to make sure it's still connected.

The computer sends to the camera data with a length of 576 which contains the picture style name.
The computer then sends a mass amount of data with the length of 16284, regardless if it's writing a picture style or blanking it. (Although whether this is a result of blanking the same picture style, but I doubt it.)
The camera then sends back a different data with a length of 576 which also contains the picture style name.
It then sends data of a length of 19520, then a length of 13376, then data with a length that changes (probably between 400 to 600), then sends data with a length of 440 (which is almost the same every time, bits of hex change at the beginning and end).

In between all that data of less than a length of 120 is constantly being sent back and forth.

(Lengths are in bytes and I'm not sure if they're 100% correct)
(In fact, minusing 64 seems to correct them 576-64=512, i dunno)

[ItsMeLenny]

This smells legit to me... Thanks for the wake up call @ItsMeLenny!

I've been watching this post for a while and have been tempted to say something for a while.

There'd be a post somewhere in the forums that I made at least a couple of years ago when I first wanted to "reverse engineer picture styles".

I am not a fan of canons ugly "film curve", I am most interested in getting a perfect gamma curve of somewhere between 1.8 and 2.2 (or 0.65 to 0.45 (or whatever) if you work it out backwards).
I think gamma curves are a good place to start, or even better creating a completely linear picture style, it would be absolutely useless to use, but good for the theory of it.

[dfort]

...it would be absolutely useless to use, but good for the theory of it.

Great quote.

The pf2 format custom picture styles on the official Canon website and CineStyle (when unlocked for editing) will open up in Picture Style Editor with all the settings in their default positions. This seems to indicate that these picture styles were not created with the Canon Picture Style Editor but with something that has perhaps more precise control.

Our logNeutral, logStandard, etc. picture styles were an attempt at matching CineStyle using the Canon Picture Style editor. These styles had to be in pf3 format in order to save the RGB curve. These "reverse engineered" picture styles allow users to modify it if they want or to just look at what the curve looks like in the editor.

Creating picture styles that can be loaded on the camera without using Canon's EOS Utility is something else that would be interesting. This should allow switching out the 3 picture styles loaded into memory with others that are saved on the card. Why more than 3 custom picture styles? It might be absolutely useless but good for the theory of it!

[DeafEyeJedi]

I think gamma curves are a good place to start, or even better creating a completely linear picture style, it would be absolutely useless to use, but good for the theory of it.

Agreed -- very well said!

Creating picture styles that can be loaded on the camera without using Canon's EOS Utility is something else that would be interesting. This should allow switching out the 3 picture styles loaded into memory with others that are saved on the card. Why more than 3 custom picture styles? It might be absolutely useless but good for the theory of it!

Excellent theory actually.

[reddeercity]

Now this is interesting

watch at 41min 41 seconds i hope the link save that time
I didn't catch the software they are using but the Cam is a 1DX , came from a post on the  Research exemption for consumer devices (DMCA)
thread that nikfreak posted . As you see the cam setting are being access and presumably hackable .
So there some merit to look at the connection ptp/ip at least with the newer cams 5D3 and up

[ItsMeLenny]

I didn't catch the software they are using but the Cam is a 1DX

They're using their own python program on linux, however the underlying library is gphoto or libgphoto2 (whatever it is referred to nowadays). It's the go to for all cameras on linux (and pulling pictures from the camera as well as some other functions, controlling the camera, triggering the shutter, etc).
At the same time I don't think it's necessary in this instance, so long as you know how the usb device communicates with the computer for a certain function, one can just rewrite that function in a program, basically a single function ignoring the rest.

Great quote.

I try... I am camera confucious.

Creating picture styles that can be loaded on the camera without using Canon's EOS Utility is something else that would be interesting. This should allow switching out the 3 picture styles loaded into memory with others that are saved on the card. Why more than 3 custom picture styles? It might be absolutely useless but good for the theory of it!

In this case it would be absolutely useful! But bad in the theory of it
(There's been plenty of these "more than 3 picture styles" posts in the past, but I think the wrong question was always being asked "can we add more slots for picture styles").
Yeah, basically one wouldn't store a pf2 or pf3 on the camera and try to translate it in the camera, although a module could probably be made for that (absolutely useless to use, but good for the theory).
One would store the decoded data in a simple file and have it write to the area of the camera from the card. This would require the picture styles undergoing a translation (I'm sure a program can be made for that).
HOWEVER! Here's the nub. Back when I was exploring this in 2013 (none of the images of the comparison of curves of picture styles works anymore http://www.magiclantern.fm/forum/index.php?topic=4895.0 ) In discussion with someone (alex or geggo (maybe I'm just naming names here)) I was told that the registers for the picture styles are unsafe to write to, they are ok to read from though.
Ultimately, going back to writing a picture style from the camera, and the whole linear curve, if that's possible, a module could be made in the camera where one could customise a curve, how much gamma and what not, and write it straight in.

Also in that video, "the camera drops my connection if I don't ping it all the time" I assume that's all the small data between camera and computer.

[ItsMeLenny]

Did I kill this conversation? Has anyone had any further progress?

[dfort]

Did I kill this conversation?

No, not at all. I've been spread out a little thin lately trying to figure out focus pixel map files for a new crop_rec video mode for the EOSM, working on some lens information puzzles and helping out users that are having trouble compiling. Add to that it is the start of awards season so there are lots of screenings and I love watching movies. Oh yeah, and there's that work thing going on too.

It looks like we were onto the same idea about copying pictures style from memory and writing them back into memory. It seems possible, every time you boot up a fresh version of Magic Lantern it writes log files onto the card that we're told to save in case anything goes wrong. These log files, named ROM0.BIN and ROM1.BIN, are the camera's firmware saved at the state things were when Magic Lantern first loaded up. Apparently there's a way to restore this data to the camera if something goes horribly wrong. If that's so then there might be a way to restore just a segment of that data. It should be possible to build up a library of precompiled picture styles--but:

In discussion with someone (alex or geggo (maybe I'm just naming names here)) I was told that the registers for the picture styles are unsafe to write to, they are ok to read from though.

That's a bummer, wonder what the consequences are.

Another way to load up precompiled picture styles is through the USB port and it looks like that should also be possible. I've got to take another look at that video on Paparazzi Over IP.

In the meantime more and more "log" picture styles are popping up, especially for cameras that have yet to get the ML treatment like the 5D4, 7D2 and this one for the 80D:

I own an 80D and have done some tests with low-iso dynamic range. I made a picture profile that expands the dynamic range in video and I've been very impressed by the clean results in the shadows. It looks very similar to the 13-stops that come off of the BMPCC. I think this camera would be amazing in RAW video, as the stills captured with this camera can be pushed a lot and still look clean. Plenty of youtube videos showing this.

I'm open for helping with the testing of a ML firmware. Also, if anyone would like the Custom Picture profile I created, I'd be glad to share it for use in the meantime. Just PM me.

Thanks to everyone working on this.

A few of us have been checking out these home cooked picture styles and comparing them with CineStyle. It seems to me that Technicolor's CineStyle is still the gold standard of log picture styles when recording H.264 video on both the old and the newest Canon DSLR's.

What's the point of all this if we already have something that hasn't been improved upon? Because we're curious and there's a lot more to picture styles than I first realized. For example when shooting a feature project is is common for the director or photography and the digital image technician to come up with "looks" that are applied to the raw or log footage when reviewing takes on the set for the director. These "looks" are burned into the files that are sent to editorial and when the raw or log files are used in color grading the colorist can pull up these "looks" as a reference. There are tools like ASC CDL (American Society of Cinematographers Color Decision List) and Academy Color Encoding Specification (ACES) that sets standards over how these "looks" can be assigned to the footage. I'm not very familiar exactly how these systems work but it seems to me that custom picture styles can serve a similar role in a Magic Lantern raw video workflow--review footage on set with the look (picture style) applied and the picture style can travel through the post process as metadata all the way to color grading.

[ItsMeLenny]

So I spent a long time trying to find the quote of it being bad to write to, going back through all my past posts, here it is: http://www.magiclantern.fm/forum/index.php?topic=3896.msg21124#msg21124
From a user named "Francis":  Not possible. Picture styles are stored in a part of memory that is very dangerous to write to.


One of the first things I noticed when I got my camera was the awful canon curve, which between that and the rolling shutter actually made me regret buying it to an extent, I started shooting neutral straight away, but it still doesn't get rid of their "film style" curve.

There is a huge amount of user made picture styles, however I believe the majority of them are made in the picture style editor, as a result they don't interest me so much.

Noteable picture styles are probably, Video-x from canon, technicolors cinestyle, marvels cinestyle, and flaat (which has multiple versions of dynamic range).

[ItsMeLenny]

Another post of mine http://www.magiclantern.fm/forum/index.php?topic=10008 "Where is that long list of registers?" All of them are linked, probably most importantly though this one: http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force/C0F0Fxxx -- "C0F0Fxxx - picture style parameters maybe"

Rest of the links thanks to audionut at the time:
http://magiclantern.wikia.com/wiki/ADTG
http://magiclantern.wikia.com/wiki/Register_Map
http://magiclantern.wikia.com/wiki/Memory_map
http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force
http://magiclantern.wikia.com/wiki/Register_Map/550D


Also: http://www.magiclantern.fm/forum/index.php?topic=4830 "Contrast and picture styles, the real scoop"

[PaulHarwood856]

Hey ItsMeLenny,

    Just curious, is there a reason Visiontech isn't mentioned as a noteable picture style? I tried C-LOG in low light and it was really noisy and had banding. But Visiontech seems to work well in any situation I throw at it. I feel more people should try this picture style since it retains dynamic range without trying to make a log image in an 8 bit compressed color space. I realize Visionlog is a no go compared to Cinelog-C, but Visiontech seems to hold up well when color grading.

- Paul Harwood

[Lars Steenhoff]

yes I currently use cinestyle, visiontech, and flaat

[ItsMeLenny]

Hey ItsMeLenny,
    Just curious, is there a reason Visiontech isn't mentioned as a noteable picture style?

Because I've never heard of it and never used it. I haven't looked at picture styles for a long time, I shoot photos on neutral and video on videoX.
I'm not any kind of superior in terns of picture styles so anybody is welcome to mention their own notable picture styles. However probably not in this topic unless its mentioned along with what the topic is about.
I'll have a brief look at it when I get the time. I was just mentioning a few that attempt to get the most out of an image without being some kind of gimmick, as in one that changes colour or desaturates things or what not.

[PaulHarwood856]

Hey ItsMeLenny,

     Ok I understand. Thanks for your response.

- Paul Harwood

[dfort]

Part of what we're trying to find out in this discussion is which log picture style works the best with these cameras. That's rather subjective but if we just look at the math it appears that CineStyle is still the best, at least technically. It appears to have a 10-bit log curve. We can argue over which picture style gives the better "look" or is easier to grade and never come to an agreement.

@ItsMeLenny -- thanks so much for those links. We're building up quite a reference library on this topic.

[PaulHarwood856]

Hey dfort,

     Ok I understand. I just think people don't realize Visiontech's advantages, and whenever I film log in low light there's noise and it's difficult to grade. Is it better to just use a log picture style in good lighting? I apologize if I came off strong I just was surprised Visiontech hasn't been discussed, like if there was something people know that I don't about it. I didn't mean to deviate the discussion, sorry.

- Paul Harwood

[dfort]

I didn't mean to deviate the discussion, sorry.

Not at all and no need to apologize.

If you have a picture style that you like it would be great if you could open it up and see how it works and what makes it different from another picture style that doesn't quite do it for you. Unfortunately the people making these custom picture styles seem to be hiding something. We're not hiding anything here.

[ItsMeLenny]

I apologize if I came off strong I just was surprised Visiontech hasn't been discussed, like if there was something people know that I don't about it. I didn't mean to deviate the discussion, sorry.

I don't read emotion on peoples posts because it's impossible to tell, I just read what is said, that is also how my posts should be read, emotionless because there is noway to convey emotion unless you want to put it in brackets after every statement (thought invoking ). Also I was typing on my phone so it may have read more abrupt than usual.

If you have a picture style that you like it would be great if you could open it up and see how it works and what makes it different from another picture style that doesn't quite do it for you. Unfortunately the people making these custom picture styles seem to be hiding something. We're not hiding anything here.

This x 3 for each statement.

@ItsMeLenny -- thanks so much for those links. We're building up quite a reference library on this topic.

Yeah there's other posts from years ago that have already tried to explore this (I think I started at least 3). But I think those are all the relevant links.

I ultimately think a module could be made that can adjust the various curves inside the picture styles and write it to the camera from the camera, so long as those registers are safe to write to.
Back to the canon film curve that I'm always complaining about, whether that is somewhere in the registers as well and being mistaken as the register that is unsafe to write to, as the registers are unknown and partly undocumented for picture styles.
It'd be great to somehow pass half of the processing rather than equalizing it out, say if that film curve is somewhere in the pipeline if that step can just be jumped.


Now, this is another theory which I have not tested, but! I am unsure if the camera utility that uploads the picture styles detects what camera it is and writes it accordingly, if it applies extra modifications between cameras for the different sensors between cameras. So the picture styles are universal in the picture style format, but if you were to compare them on the cameras if, say for instance, one of the colour channels has a slight different curve applied to it to accommodate for that sensor. If that all makes sense.

[PaulHarwood856]

Hey dfort,

     Ok great. Thanks for letting me know. I might need to dive into this Visiontech picture style at some point. Could you let me know which tool to start with? And what to post first?

 Hey ItsMeLenny,

     Ok good point. Thanks for that.

- Paul Harwood
 

[Danne]

Could you shoot a comparison between cinestyle and visiontech?

[PaulHarwood856]

Hey Danne,

      Sure! A well lit shot and a shot in low light? Is there a chart I should have in the shot?

- Paul Harwood

[Lars Steenhoff]

you can also test this one if you want

https://marvelsfilm.wordpress.com/marvels-cine-canon/

[Danne]

Something to check shadows. Just the same settings, same motif, just for fun.

[PaulHarwood856]

Hey Danne,

     Alright awesome! I have a 7D and T3i which are both the same sensor, so I'll try up to six picture profiles. Visiontech, Cinestyle, Marvels, EOS HD CLOG, and Flaat 10 and 11. If there are any others please let me know. I'll get this done when I can.

- Paul Harwood