MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[Danne]

Yes, and yet the adobe tag comes out like this when baked in adobe dng converter. They must run some template.
Unique Camera Model             : Canon EOS 750D

[a1ex]

Right.

Can you create a list of UniqueCameraModel names, with the exact spelling used by Adobe DNG converter?

[Danne]

They are quite consistent. Here is the code snippet from bouncyball and the correct names

void set_unique_camera_name(mlv_idnt_hdr_t *idnt_hdr)
{
    switch(idnt_hdr->cameraModel)
    {
        case 0x80000285:
            memcpy(idnt_hdr->cameraName, "Canon EOS 5D Mark III", 32);
            break;
        case 0x80000218:
            memcpy(idnt_hdr->cameraName, "Canon EOS 5D Mark II", 32);
            break;
        case 0x80000302:
            memcpy(idnt_hdr->cameraName, "Canon EOS 6D", 32);
            break;
        case 0x80000250:
            memcpy(idnt_hdr->cameraName, "Canon EOS 7D", 32);
            break;
        case 0x80000325:
            memcpy(idnt_hdr->cameraName, "Canon EOS 70D", 32);
            break;
        case 0x80000287:
            memcpy(idnt_hdr->cameraName, "Canon EOS 60D", 32);
            break;
        case 0x80000261:
            memcpy(idnt_hdr->cameraName, "Canon EOS 50D", 32);
            break;
        case 0x80000326:
            memcpy(idnt_hdr->cameraName, "Canon EOS 700D", 32);
            break;
        case 0x80000301:
            memcpy(idnt_hdr->cameraName, "Canon EOS 650D", 32);
            break;
        case 0x80000286:
            memcpy(idnt_hdr->cameraName, "Canon EOS 600D", 32);
            break;
        case 0x80000270:
            memcpy(idnt_hdr->cameraName, "Canon EOS 550D", 32);
            break;
        case 0x80000252:
            memcpy(idnt_hdr->cameraName, "Canon EOS 500D", 32);
            break;
        case 0x80000288:
            memcpy(idnt_hdr->cameraName, "Canon EOS 1100D", 32);
            break;
        case 0x80000331:
            memcpy(idnt_hdr->cameraName, "Canon EOS M", 32);
            break;
        case 0x80000346:
            memcpy(idnt_hdr->cameraName, "Canon EOS 100D", 32);
            break;
    }
}


list
Canon EOS 5D Mark III
Canon EOS 5D Mark II
Canon EOS 6D
Canon EOS 7D
Canon EOS 70D
Canon EOS 60D
Canon EOS 50D
Canon EOS 700D
Canon EOS 650D
Canon EOS 600D
Canon EOS 550D
Canon EOS 500D
Canon EOS 1100D
Canon EOS M
Canon EOS 100D

[dfort]

Cool trick. Here's one from a localized 700D. There isn't a trace of "700D" in the CR2 file.

Code: [Select]

exiftool -htmlDump IMG_1695.CR2 > rawexif.html


After running Adobe DNG Converter:

[/url]

I had a similar issue on the Assign lens focal length and name for non cpu lenses (currently broken until a commit message is fixed ) when I was discussing how lenses are automatically identified in ACR with an Adobe staff member. Seems that they use different EXIF tags for Nikon and Canon lenses so it isn't surprising that they use different tags for CR2 and DNG files.

[dmilligan]

Does everything still work if we use LocalizedCameraModel instead of UniqueCameraModel. If so I can make the switch.

[Danne]

How do you mean? Localized tags turned into uniquecameramodel tags won,t work.
The uniquecameramodel tag has to be a global/generic tag according to the list I wrote before.
It,s actually only a few cameras that have those localized tags. Check exiftool docs for which. I can do it later, not home...

[dfort]

Does everything still work if we use LocalizedCameraModel instead of UniqueCameraModel. If so I can make the switch.

I think the issue is that MLVFS puts the LocalizedCameraModel in the UniqueCameraModel EXIF tag:

Code: [Select]

exiftool -htmlDump M31-1256_000000.dng > rawexif.html


That should be "Canon EOS 700D". Note that Adobe DNG Converter writes both the LocalizedCameraModel and UniqueCameraMode tags and they are different on this camera.

[Danne]

Canon id tag is registered as a localized tag in magic lantern mlv files. Probably the only tag in there. The only tag needed is the unique one(used in ACR). That one can replace any of the other tags. Or not, doesn,t matter.
Isn,t a1ex looking into this anyway?

[dmilligan]

I think the issue is that MLVFS puts the LocalizedCameraModel in the UniqueCameraModel EXIF tag:

Righti know that, but it's the only thing to put there, which is why I asked the question. So I'll ask again, if you craft a DNG with LocalizedCameraModel and no UniqueCameraModel does it work correctly?

[Danne]

If you are asking to exclude the unique camera model tag from the DNG spec entirely that will not work since it,s the tag used in ACR. Well, I havn,t tested if ACR will rely on some other name tag if unique tag is excluded. Just erase it and see what happens.

[Danne]

I just erased the unique camera model tag in mlv_dump and ACR then reports the dng file as a DIGITAL NEGATIVE


The file still has the camera model name tag with the Camera Model Name               : Canon EOS REBEL SL1 tag but ACR doesn,t read that at all.
I didn,t put in the localized tag but I,m pretty confident it won,t work. I can see if I can put it in with exiftool.

[Danne]

Oops, I was wrong. The localized tag actually is read by ACR if the unique camera model tag is removed. Here is proof.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/100D_mv1080_frame_000003.dng

[Danne]

*Clearance
Even if the localized tag is shown in ACR this does not mean we can use the local tag itself with dcp profiles. We still are dependent on the camera-specific constants(list of camera model names). Even if it,s possible to technically switch out the unique tag with the localized tag it,s not the solution without correct camera model names.

[Andy600]

ACR will usually read a DNG regardless of which model tags are used but will default to an 'embedded' calibration if no tags are included or recognized. This means only the embedded meta is used. The embedded meta might or might not be correct depending on how the calibration was obtained and the type of calibration that was performed. Think of this as tier 3 calibration - i.e. there are typically one or 2 color matrices allowing ACR to find a good auto white balance if needed. Some ML apps also embed forward matrices which offer some degree of color tweaking to the white balanced image.

If a model tag other than UniqueCameraModel is recognized by ACR (i.e. it is from a camera that has a calibration hardcoded into ACR) then ACR will use its internally coded calibration for that specific camera model and embedded color matrices will be overridden. ACR will show something like ACR V9.x for recent cameras and ACR v4.x for older cameras depending on when the last update was for that model. Think of this as tier 2 - it is calibrated (by Adobe) but doesn't allow selection of camera specific color profiles. Forward matrices are used only if embedded in the DNG.

If the UniqueCameraModel tag is filled then ACR will let you select alternate profiles (Adobe Standard, Camera Standard, Camera Neutral etc) where available. Think of this as tier 1 with the most options and typically the best, most up-to-date calibration. Embedded calibration tags are overridden by the profile.

The UniqueCameraModel tag is used by most, if not all, cam manufacturers that have cams that capture to the DNG/CinemaDNG format (BMD,Digital Bolex, Convergent Design, Zenmuse etc). It is the most commonly used identifier and may or may not be important for accessing camera-specific calibration and certain features in some NLE and color apps.

Regional tags are best translated to uniqueCameraModel tags if possible/required. In terms of calibration and color, there are no internal differences between the US, Japanese and Euro versions of a camera. The main difference is usually the front badge.

The only time the UniqueCameraModel tag should be purposely omitted (or better still, changed to something unique for the purpose of identifying a specific camera) is when a specific sensor has been purposefully lab calibrated and where it's images will usually only be developed in ACR i.e. to ensure this very specific calibration is not over ridden in any way by ACR - this is usually only done for industrial or research reasons.

[Danne]

Thank you.

[Andy600]

You can find references to hard-coded calibrations by searching ACR plugin in notepad or other basic reader. Obviously, being code, most of what you see is random characters but the camera models that are supported show up in English. Try looking for a localized tag to see if it's supported.


EDIT:

Interestingly, if you reconvert your de-tagged DNG (or any detagged DNG) with DNG converter it adds a UniqueCameraModel tag (UCM) called 'Digital Negative'. This means the local tag is not sufficient enough to identify the camera to the converter or ACR (as I thought - else it would have added the usual UCM tag) but it likely needs the tag filled for validation (need to recheck DNG specs).

[Danne]

Cool findings Andy600(Digital Negative).
I downloaded my own file and renamed the localized tag to Canon EOS 5D mark III to see if it would work to use dcp profiles but only embedded option available. This verifies the need for the uniquecameramodel tag and it has to be set set to adobe dng standard naming.


In following example I added back the UniqueCameraModel tag and named it Canon EOS 5d Mark III and now all dcp profiles and alternate profiles works for a 5D mark III(even if the file has matrices for the eos 100D).
Linear dcp profile added

[PaulHarwood856]

Hello everyone,

     I apologize for the delay, here is a CR2 from my T3i on the latest Nightly Build, December 19th, 2016.

     https://drive.google.com/open?id=0Bz5n0qYxIkVOTno1M0NHZ0pSeXM

- Paul Harwood

[Danne]

I recently discovered you can actually alter the metadata tags inside the virtual dng files with http://www.exiv2.org/
I use this tool to rename camera model name via cr2hdr.app and also for auto white balancing. Now what I don,t understand is where the information is being stored when doing this? No MLD folders are created, it just writes, seemingly, straight to the virtual dng file? Some cached memory solution?
I run this command for instance.

Code: [Select]

find . -iname '*.dng' -print0 | xargs -0 -P 8 -n 1 exiv2 -M"set Exif.Image.UniqueCameraModel Ascii Canon EOS 100D"Check the movie file. 

[dmilligan]

That shouldn't be possible and is some crazy black magic. Debugging on the MLVFS side, write() is never called, and when data is requested from MLVFS via read(), MLVFS returns the original file data, yet the requesting process gets the overwritten data. Trying to do the same thing with a hex editor, results in an error when attempting to save changes as expected.


It looks like there's not going to be an easy and satisfactory solution to this issue. Note that this issue also means the cam matrices MLVFS embeds are also broken for "Rebels". I don't really like these being hardcoded either. So I think I'm just going to get rid of all of it, and make a way to have metadata tags defined in data files referenced by camera id. This will probably be a far amount of work.

[Danne]

black magic

My exact words to bouncyball when he tried doing the same on a windows machine and it didn,t work on his end.
I use it for altering default scale tag, AsShotNeutral and UniqueCameraModelTag so it really works for any tag seemingly.
http://www.magiclantern.fm/forum/index.php?topic=15108.msg177259#msg177259

[bouncyball]

Hi there

Really strange. In win, any time when altering dngs with exiv2, "altered" dng files are copied to MLD folder, but the content of files is identical to the original virtual file. Hexedit or any write operation surely ends with error (which is correct). "Definitely" apple did some sort of (black, gray, orange, whatever) magic on read/write operations .

bb

P.S. I should try this on linux.

[andrew_dotdot]

Hi,

Up until this summer, I used MLVFS on my mac *all the time* to work with MLV files from my 5D3 in Davinci Resolve, and it worked great. I could play back MLV in real time. This fall I did a project all in camera codec, so a break from raw, but I'm now working with .MLV files again -- and MLVFS is sooo slow!

I've tested my SSDs and un/re-installed MLVFS and updated Resolve and done all of the normal things I can think of to fix this. If I convert the MLV files on my PC, I can play those folders of DNG files back in Resolve in the mac at 40-something FPS, but the same source MLV under MLVFS is, maybe 7fps, and eventually bogs down and stops playing back.

Where can I look (some log file?) to help me get some clues to get my poor MLVFS back on track? I'm willing to work for this, but I don't even know where to start!

Thanks,

//Angus

[dmilligan]

Start by disabling any extra processing options you may have turned on. If it's still slow, check activity monitor and see what CPU usage looks like. Finally, you can download xcode and run MLVFS in the profiler (this is fairly easy and requires no coding skill).

[andrew_dotdot]

Thanks for the tips!

All the processing options are off. No stripes or dots or smoothing or anything like that. The processor shows that the mlvfs process stays around 100% while it works, which is just one of the 4 cores going like mad, I guess. Then Resolve's got either 26% or 126% at various times when I'm trying to render to Prores 422 HQ.

Sooo, I guess it's Xcode and profiler...which I've not used, but how hard can it be? We'll soon see..

//A