Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - aceflibble

Pages: [1]
Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: November 09, 2020, 09:01:50 PM »
Just a vague update, I've been in pretty poor health this last week or so—on-going back problems—so I haven't been able to sit down at the computer and work on that neutral-standard profile or, well, anything really. I realise this is inane to most people but since I did say I'd work on it and be back in a week, I just wanted to drop in quickly and explain why I'm a bit slow.

While I'm here I'd like to again request for help, from anyone capable and that has the time, in investigating whether we can get NakedPSE to open and save .pf3 files properly (thus keeping the useful 'six axis' changes from regular PSE) or if we can get regular PSE to honour the matrix and curve changes that NakedPSE can make. There are some changes that are simply quicker and easier to make with the .pf3 six axis option than within the matrix or the specific colour adjustments, plus it's a bit suspicious how the .pf3s are 100x larger than .pf2 files even when no changes have been made and it'd be great to know what's happening in all that extra data.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: October 28, 2020, 12:05:18 AM »
Still, I don´t really get the 10bit reference, or source, since all that was done creating the pic style is done with the 8 point(or 10, can´t remember), curve tool in the pic style editor. All in eight bit. Unfortunately no "blackbox magic" in the pic style.
But that's the thing, we all assumed that for years but it turns out it wasn't created that way. They could not have possibly put in these values in PSE. It's literally not possible for that software to do it. So either Canon clued them in to NakedPSE all those years back (unlikely given it has apparently only ever existed in Japanese and if it was being offered around you'd think we'd have heard of it sooner) or Technicolor had some other method not using any version of PSE in order to set those values. So our previous assumption that they just knocked something together in PSE is wrong, they had something else going on back then and I'd love to track down one of those developers and find out what they used.
But however they set the values that way still leaves me wondering why they used the 10-bit values, when they could have created effectively the same curve shape using the 8-bit values every other profile uses, and of course it doesn't explain at all how it is the Canon systems can recognise and action those 10-bit values when they're supposed to be working with 8-bit values and only applying them to 8-bit files. The only thing I can think of is the slim possibility that the cameras actually do work in at least 10-bit (if not higher) before the gamma curve is applied. This tallies with ML being able to get 10-bit video out of older cameras (albeit usually with a crop or strict time limit) but the assumption there has always been that that was some kind of extra-tough hacking wizardry that ML was creating and Canon had no clue about... but these profiles and the cameras supporting 10-bit values suggests Canon was working towards this themselves, but for some reason never completed the functionality. I mean, I just can't think of any other reason why Canon would make sure their system could work with 10-bit values on a post-capture processing ruleset unless they actually thought they might include some kind of 10-bit (or higher) recording. (Outside of raw photos, which of course aren't affected by these curves.)

As I said earlier, the whole thing is mind-bogglingly strange.

I really like the Cinetech picture style for my shooting style from Visioncolor for shooting 8 bit .h264. More dynamic range compared to neutral (prolost settings). Maybe a little bit more noise but you can clean this up in post.
I'm afraid as we've pulled apart the files, there isn't anything which can capture more dynamic range than the stock Neutral profile already does with its true bare-bones matrix. What you're seeing in those boosted styles is not more dynamic range but simply a reallocation of the range, typically brightening shadows up (hence why you're seeing more noise) at the expense of cramming the mid tones further together. There's actually a good example of this type of 'expansion' much earlier in the thread. If you look on page 5 you'll find Danne trying to work out a more linear DCP profile and thinking they may have seen more dynamic range (but to their credit, acknowledging they may have been imaging it) and dfort following up a few replies further down pointing out that after careful inspection there wasn't more range, it's just that the peak brightness was a little lower so it made the same detail look slightly better-defined. What you're getting with your noisier Cinetech profile is the same thing at the opposite end of the scale, as I said before; the same range, just gaining definition in one area at the cost of definition in another.

Canon's own plain ol' Neutral is the matrix that renders the widest range and the most detail. Not necessarily the most clearly-defined detail and range, but the most in a total sense. You simply can not get purer than that 1/1/1/0/0 matrix.

But, as I said as part of my much longer comment above, if what you're using is working for you and getting you the results you like then don't worry about whether or not some other method is or isn't technically better, or easier, or more popular, or whatever. If the final image you get is how you wanted it to be and you like the workflow then just keep doing what you're doing. 8)

Before anyone suggests it, I've already tried using 0.9 and 0.5 matrices and then boosting saturation back up later, and they don't improve anything. Dropping the initial strength of each channel does improve clarity in mid tones but takes it away from shadows and highlights, so again it's just a trade-off. There's no actual improvement on the simplest matrix, unsurprisingly.
If you want to shoot in black & white then I have found a totally even 0.3/0.59/0.11/0/0 matrix does seem to give more mid tone nuance at no cost to shadows or highlights compared to the stock monochrome profile, but personally I like my black & whites to be a bit more punchy than true-to-life.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: October 27, 2020, 12:13:30 AM »
Cool. My assumption is there's nothing harmful in a 1-4kb colour profile, but I always like to be as safe as possible. You never know when technology is going to disappoint you.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: October 26, 2020, 10:01:18 PM »
Like I said, it won't be possible to copy the Standard curve over 100% onto the Neutral colours as you requested above—it's not a copy&paste job, unfortunately—but I see no reason why something very close to the Standard tone curve could not be created on top of the Neutral colours. It'll just be a matter of trial-and-error, lot of tweaking, checking, tweaking a bit more, checking again, etc. Of course anybody else is welcome to have a stab at it as well, but in any case I'll give it a shot this week and let you know how I get on. At the very least it'll be interesting to try to replicate the curve without having the actual numbers on-hand to copy.

In the meantime, here is Fine Detail converted to .pf2. By default it will have sharpening set to 1 and everything else at 0, but of course that can all be changed however you like. I really like Fine Detail for all-round use, it's got a bit more saturation and life to it than Neutral but it's more realistic than Standard, I find.
I've only tested that profile with an EOS R (to verify it was producing the same colour as .pf3 Fine Detail), a 7D mark II and a 5D mark II, though. I can't think of any reason why a custom profile would damage any other camera, but just to be on the safe side, I suggest you try it out by loading it into a user style slot that isn't currently set as your default; take a raw photo with the camera set to one of the stock profiles, then in playback have the camera re-process the raw image into a .jpg with the custom picture style. That way you can safely check that the camera will load the profile up and use it correctly. As I said, I can't imagine there will be any problems, but y'know, no harm in being cautious when loading custom files onto an expensive piece of equipment.

And yeah, I'm with you on the ColorChecker profiles. I used to swear by them and they are definitely more accurate than anything Canon makes, but I got tired of the extra effort years ago and I've been working mostly with just Neutral as the base profile in Lightroom or Fine Detail in DPP and in-camera. That's part of why I shoot Fuji alongside Canon and why I'm so keen on digging into custom profiles; if I can finally get the colour I want right in-camera, nice and quick, then I don't mind giving up a bit of technical quality. The less time I have to spend jerking colour around in Lightroom, the better.

What 10bit values. I can't follow.
... The ones I listed above and asked about. Y'know, the question which you responded to in your previous comment. The ones you already read. Technically, the ones that were talked about in this thread many years ago, 's just back then you were trying to dig it out of hex values and eyeballing with image analysis tools. (Page 7, Andy600's post about halfway down.) Well, now we've got the actual, factual numbers for that curve (which turn out to not be the same as the earlier thread concluded), and they seem completely out of place with everything else we know about these files. So to reiterate my question, I'd like to know if anybody can think of a reason why CineStyle would have been made this way instead of with 8-bit values like every other profile is, or, alternatively, if someone can spot something in those numbers that suggests an actual unique purpose that I've overlooked which could help us further 'decode' these profiles.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: October 26, 2020, 08:02:37 PM »
Their values aren't the same as what you had earlier in the thread, though. It also doesn't explain why, let alone how, they put in 10-bit values for an 8-bit system, which is what I'm asking about. We know what the end results are; I want to know what drove them to come up with that method to begin with.

And no, picture style doesn't come in "late in the chain"; it's throughout the whole chain, it is the chain. The first thing that happens is the cameras is checking to see if Standard or BW was chosen and then after that, assuming neither was, the image is rendered using the matrix specified in the picture style. It's coming before the gamma curve, before all the in-camera controls, and before the LAB curve. The different in-camera controls are implemented at different points between. That is not insignificant.

Also, as I said the last time you replied with something along those lines, there are more purposes at hand than just trying to get a neutral video to grade. That is one use case that is interesting to dig into, but ultimately we're talking about something that affects in-camera .jpgs, histograms, and of course can be used for all manner of creative purposes, especially for those who possess cameras that are not able to shoot in more technically-accurate, edit-friendly styles such as real log or 14-bit raw photo. (Or simply for people who don't want to or can't spend their life sat at a computer editing their photos or grading their videos, which is an increasingly large market thanks to vlogging taking over the planet and peoples' desires to get photos from camera to online ASAP.)

I'm not trying to suss this out because it's the end-all best workflow for professional video, I'm trying to suss it out because it's useful for all manner of purposes and because it's plain interesting and fun to dig into.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: October 26, 2020, 06:55:38 PM »
So a kind of reverse of what the Fine Detail profile does? (Which is the Standard hues but with the Neutral/Faithful contrast.) I'll see if I can cook it up. Might be a bit tricky because the Neutral profile is truly unique among all the profiles as its matrix is literally the most basic it can be; out of the entire 3x20 matrix grid, the only values are simply 1 in each of the initial red, green and blue channels, and nothing more. (See way below for an example table.) All the other profiles (update: except Standard, see following edit) have entries in every single position, accurate to six decimal places and don't actually use the tone curve inputs at all. Getting 60 values to work with the hue produced by just 3 might not be possible precisely, but I'll try to get as close as I can.
EDIT: I just checked. I don't think it'll be possible to copy the tone of Standard, and I've hit something very interesting. It seems the Standard and Monochrome profiles do not work the same as other profiles. There's almost nothing in them—the Standard matrix is the same basic one used by Neutral, shown way below—and instead it causes the software or camera to load up some kind of hidden 'default' rendering based on whatever profile is named at the top of the file. If you try to open up Standard you're basically just working with a drop-down list of the camera defaults. These are probably tailor-made for each camera model, compensating for the different sensitivity of different sensor models. This does not happen for other profiles; the drop-down list and named profile are ignored if anything other than Standard or BW (Monochrome) are chosen.

For example, if I change Standard's listed profile to Landscape, using that new profile just looks 100% like the Landscape profile even though I didn't touch the the matrix or curves; the look of Standard has been lost entirely. If I load up that new profile the matrix still is untouched, even though it's now producing the Landscape colour. But if I do the opposite, changing the Landscape profile's setting to Standard, it continues looking like Landscape and never gains the look of Standard.

In other words, Standard doesn't exist like the other profiles do. Standard is some kind of real firmware-level default, not something we can inspect.
Monochrome is the same deal, there's no way to extract it and it only exists in the same drop-down menu as Standard. Strangely, though selecting Standard in other profiles does not turn them into Standard, choosing 'BW' does turn any other profile into Monochrome but without any of the Monochrome-specific controls.

I expect there are many values still hidden to us that are controlling these things. I shall try to cook up a version of Neutral with contrast similar to Standard for you, but it looks like being able to copy Standard's contrast 1:1 is off the table.

Funnily enough it's those sorts of 'simple' adjustments that are turning out to be the hardest.

In the last month I've been messing with the matrix a lot, trying to 'decode' what each entry point refers to (given the matrix isn't labelled and the limited Japanese documentation with the software seems to only refer to the standard version of PSE), and I've found some ways NakedPSE can push beyond what regular PSE offers in order to create a variety of effects; some very useful, some... not so much.
Effects I've got mostly nailed now include negative/inverted colour and exposure, selective colour, single- and two-tone colourisation (separate from split toning), and posterisation. I've also mostly copied over Fujifilm's 'Classic Chrome' profile (about 90% accurate, stills needs some tweaking) and I'm going to start working on their 'Classic Negative'. (I did start copying Astia and Pro Neg S as well, but it turned out those were already 99% like Canon's own Portrait and Faithful profiles, so there wasn't much point bothering.)

... Then there's this thing, which I'm tentatively naming 'in-camera LSD'. That's what happens when you use NakedPSE to push regular PSE's 'specific colour' tool beyond its normal 30-180° limitations... in this case, typing in a hue range of 6000° and a hue shift of 4000°. Totally useless, but quite amusing to see working in live view/video and it goes to show just how free our in-camera colour choices are now. That one was an accident of mucking about, but I am now trying to work on a profile that keeps that has that sky gradient on purpose, without affecting the rest of the image. Just thought it'd be fun to have rainbow-coloured skies. Probably won't work, but we'll see.

Anyhoo, some more practical things I've picked up:

The Fine Detail profile that Canon added with the 5DS and the introduction of the .pf3 format can be converted to .pf2 and works perfectly fine in older cameras. There's nothing in its colour or contrast that the older cameras can't do. The only thing is you do of course lose the improved sharpening controls (old cameras still only get the one sharpening control, not the strength, fineness and threshold controls of newer cameras), but that's to be expected as that's not a system that is saved in any of the colour profiles. Or, in other words, Canon claiming Fine Detail was only made possible by the new tech and limiting it to .pf3 was always a lie. It's just a 3x20 matrix, same as all the others.

Canon's Video-X profile is the only one of Canon's own profiles that uses a non-linear RGB curve. The Video-X's curve lifts the shadows quite a lot and very slightly lowers mid tones. In fact Video-X is the only Canon profile that touches the curve inputs at all; every other profile simply has the curve set at 0-0, 255-255, with no other points between. The 'LAB' curve (which as noted earlier in this thread doesn't seem to actually be a LAB curve as it uses RGB values and the A and B axis are entirely ignored) is also plain linear in all profiles. (Video-X included.) All the contrast changes between the profiles other than Video-X is done entirely within the 3x20 matrix. Makes you wonder why Canon even bother having the curves at all.

Common wisdom for the last ~12 years, especially for video, is to create a faux-'log' look by selecting the Neutral profile and then setting both Contrast and Saturation to -4. Many books and sites will tell you that even the Neutral profile adds some slight contrast. You may also have heard that the Contrast and Saturation controls are applied before the picture style. Turns out, that's all gibberish. By doing a lot of back-and-forth with different custom matrices and seeing how each control behaves in response, I've deduced that Contrast and Saturation are applied after the profile's matrix and gamma but before the specific colour adjustments, then Tone and Sharpening are applied and finally the LAB curve is added. (Yes, after sharpening, bizarrely.) Neutral doesn't add any contrast to anything, it truly is as pure as a colour profile can be, as I detailed above. So setting Contrast and Saturation to -4 doesn't actually record more detail in anything with the stock profiles; if a stock profile's matrix is going to blow out a colour or crush a shadow, it will have already done so before those controls are applied. All that is happening with the Contrast control is it's applying a post-matrix, post-gamma brightness curve. Anything it appears to have 'saved' was in fact always there, just perhaps not entirely clear to you. (After all, how many people can tell the difference between a 100% white pixel and a 99% white pixel?) It also does definitively reduce detail as whatever space you make in the highlights and shadows is taken away from the mid tones. The Saturation control works the same way; it can't actually recover anything that has been genuinely blown, it's just enabling you to see more of the nuance in the most saturated parts of the image at the expense of losing nuance in the less-saturated parts. 
In practical terms this shouldn't really affect how anyone uses either the Contrast or Saturation controls, or the Neutral profile—after all, as long as your results look good to you that's all that matters—but it does mean anyone who previously said the Contrast control could actually expand dynamic range was simply, categorically wrong. Technically Neutral already has the fullest range possible, and changing Contrast or Saturation is just trading one part of the range for another.

Somewhat related, a question of my own: A lot of people used to rely on CineStyle, especially with with Contrast and Saturation turned down. That was, for years, 'the' way to film with a 5D2. CineStyle and many premium 'flat' profiles boasted about being able to give you more dynamic range and more colour nuance than any stock profile. Well, here's the CineStyle matrix: 

Look familiar? Yup, it's just the Canon Neutral profile. No change to initial recorded colour or contrast. Just like I outlined with the Contrast and Saturation controls, if something was being blown out (either a tone or colour) then that's it gone for good, nothing following it could save it; anything that seems 'saved' by CineStyle would also be equally saved by Neutral. It's not doing anything to save more data than the stock Neutral profile already does.

But what about the gamma curve? That certainly is different, but it's not doing quite what you might expect and this is where I want someone else to weigh in:
0 - 64
128 - 227
256 - 350
384 - 441
512 - 533
640 - 625
768 - 722
896 - 832
1023 - 1023

Given the curves in normal PSE work on just an 8-bit 0-255 scale, and Canon's own profiles all use an 8-bit 0-255 range, that sure is weird. It seems Technicolor found out (or Canon tipped them off) that they didn't have to just stick to the standard scale and used a 10-bit 0-1023 scale, even though 10-bit recording wasn't available at the time the profile was made, and then they... remap 90% of it to fit in a 64-940 video range/reference monitor's range? But then not the very top of the range? It's like they thought they were mixing HDR to SDR... before HDR existed... and thought the black point had to meet video standards but not the white point. It's bizarre.

This is, like I said, where I bow out and am asking for someone else to explain this one 'cause it's the one part of all my digging that has me stumped. I have been able to verify with lots of trial-and-error that it's not presenting any more 'recovery' than a simple 8-bit value, low-contrast curve can. It can't be for stills because .jpgs are 8-bit and this profile has no bearing on raw files. I know ML eventually got 10-bit video working on otherwise 8-bit SLRs, but that was well after CineStyle was made and released and neither Technicolor nor Canon would have been working on a profile on the assumption that a third party might make additional software to hack in extra functionality later.
So I just can't get my head around why they plotted the tone curve this way and what they hoped to achieve. Who would be working on an 8-bit system but use 10-bit values? And why would the cameras accept this when they can only apply it to 8-bit files? Anyone? Anyone? Bueller?

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: September 23, 2020, 05:52:06 PM »
Yeah most of the options in NakedPSE are for exporting in various formats, which doesn't help us really decontruct the colour profiles.

FWIW I've found that, like with the R, G and B curves, any changes to the 3x20 matrices in NakedPSE will be lost when opening the profile in PSE and some severe changes in either matrix results in my test EOS R crashing. Still trying to find out the exact limits but it's a lot of trial-and-error. For now it seems like editing the matrices is not advised.
The good news there is I've gone through many premium profiles—about thirty now—and none of them have used custom matrices anyway. It does seem like every profile people have been paying money for over the last ~14 years has in fact been made entirely using regular PSE with nothing truly custom or out of the ordinary. Good news for those of us who'd like to know what changes were being made and how, since it's all much simpler than it used to seem, but bad news for those who paid money for a profile thinking they were getting some actually new functionality.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: September 14, 2020, 09:05:30 PM »
On the other hand. Picture styles seems rather meaningless as they are applied to 8bit h264. Better just dial in some visibly wider range straight in cam imo.
If your goal is only to push the latitude a little further for video and your camera doesn't have log, sure.

But for creating specific looks, this is a big deal. For anyone who needs to run-and-gun and use either footage or .jpgs straight out of camera, being able to now easily edit R, G and B curve values in addition to all the other adjustments that can be made in PSE could be quite significant. I have to keep a Fuji system on me alongside Canon as although I have a better lens selection with Canon, sometimes I'm asked to take some behind-the-scenes snaps to post right away and Fuji's Classic Chrome and Classic Negative profiles have so far been my most-requested looks for that. They're also the two colour profiles that are really unique to Fuji and PSE couldn't replicate before due to the lack of shadow tinting. With this NakedPSE and a bit of time (and ideally some way to get .PF3s working), that could well no longer be the case. 

We're basically talking about bypassing any need to shoot flat for later grading and just getting the finished, graded look done in-camera instead. It may not be 100% of the quality on pixel-peeping inspection, but for times when speed matters more than quality, this is (potentially) extremely useful.

Reverse Engineering / Re: Reverse Engineering Picture Styles
« on: September 14, 2020, 05:59:49 PM »
Great timing. I read through this thread a year or so ago while looking for a way to access and hopefully edit the R, G, and B curves separately. Tried via hex as brought up earlier in this thread but couldn't get reliable results; probably more my limitation than the method itself, but anyway. Randomly got the itch to search again today and I'm glad I did. This NakedPSE gives quick access to the R, G and B curves separately. Signed up to the forum say thanks for bringing this to people's attention, at the very least.

What I've found so far is though profiles with edited curves work fine in-camera and in DPP, regular PSE doesn't reflect the alterations, and trying to edit the curves or most other 'hidden' things in NakedPSE and then editing the profile further in PSE causes the NakedRGB edits to be lost. Seems the only thing NakedPSE can change that PSE also maintains is the option to prevent a saved profile from being edited further or not. (And the basic sharpness, contrast, saturation and tone settings, but you don't need NakedPSE to change those.) Maybe I've missed something but it seems opening a .PF2 in PSE will essentially strip it of anything that wasn't edited within PSE itself. So you need to use NakedPSE to 'unlock' a profile, then edit it as much as possible in PSE, then open it up again in NakedPSE to mess with the 'hidden' parameters, or do it all in NakedPSE.
Additionally I've found that entering in any kind of negative or inverted curve, either in the RGB curves or the LAB curve, will cause DPP to crash when loading the profile; I don't know if a camera could take it but I'm not willing to risk it in a camera if DPP can't handle it. Seems every curve point has to be higher than the one preceding it. Whatever algorithms Canon uses to smooth out the curve can't handle it, I'm guessing. So no in-camera correction for scanning film negatives, I'm afraid. (Shame, since Nikon's editor can do that.)

Will be doing a lot of trial-and-error (since I don't speak Japanese) with every function for the next few days. Hope some of the more clued-in posters here can return to this thread and help 'decode' how much NakedPSE can do.

In any case, thanks again for finding that and bringing it to peoples' attention, this is potentially huge. Never thought I'd be able to get split-toned colour in-camera, but there it is.

Pages: [1]