Messages

Very nice!
Just curious, is any focus pixel present on the pictures captured this way?

Hi
I think that shows the resolution of the embedded preview image.
I have a similar file, I took a picture from ExifPilot program
first is preview image(uncompressed RGB)
second is full size jpeg compressed raw(CFA)
both are embedded in the same DNG file

How does the DNG behave when it is opened in a raw processing program like RawTherapee?
Can you send samples?

I am at work now, if arrive at home i'll check (5 hours from now)
It's probably the version on ML Downloads/Utilities

export_from_compressed_mlv

tools used
dng frames and an mlv

@masc
I completely forgot mlvapp is capable of this, thanks for reminding

@Danne
Uploaded on "dng frames and mlv"
0_corrected_I_raw_corr_disabled.dng is that i made outside of mlvapp, the others are from the original, exported with different raw correction settings.
Mine does not look good as dng somehow, but i attached an mlv that contains the first frame of the round trip file. There is looks okay.

@adrjork
I have MLV App v1.11
Also i zipped the tools as "tools used"

[raw2mlv.exe *.dng -o output.mlv]

1-4. is correct
5. In MLVApp i used Default Naming Scheme.(I renamed the files manually in 7., i think it does not matter)
6. is correct
7. I used cr2hdr.exe app on windows. I have no knowledge about Switch.
Drag&dropping all dng onto cr2hdr.exe give me a "filename or extension too long" error, so i renamed the files. The cause however wasn't this, but the list of filenames is too long to drag&drop.
cr2hdr works on a single core/thread, and i have a skylake i5-6400/4core. I made a few directories, each containing the exe, and 250 dng files
I drag&dropped each directory's 250 file onto the EXE, i let 4 processes run at once, giving full cpu usage

cut&pasted the DNG files to a common single folder
The other app is raw2mlv.exe for making a single MLV from the DNGs
This one too shows "filename or extension too long" error on drag&drop
However, this can be worked around from command line. LINK
opening a command prompt in the dng folder, and running raw2mlv.exe *.dng -o output.mlv gave a single mlv file

MLVApp for compressing the original, raw correction off
MLVApp for extracting the frames as DNG, raw correction off
cr2hrd.exe for processing dual iso to single iso
raw2mlv for stiching the dng frames into a single mlv

btw this problem reminds me when i recorded dualIso anamorphic videos. MLVApp detected anamorphic video, and set the Transformation/Width Stretch to 3.0x
The frames exported this way were not recognizeable to cr2hdr, but when i set the stretch back to 1.0x, the exported frames could be processed. I think that too is stored somehow in the dng files

I took the 6GB file to experiment
Experiment 1
Opened in MLVApp, exported some frames as Uncompressed DNG.
RawTherapee saw the dng-s as raw files, but they were filled with 0 values.
Opened with my homemade tool i was able to see the values in the files. This tool ignores any metadata apart from resolution and raw values.
These dng-s cannot be processed with cr2hdr app directly. Converting with Adobe DNG converter usually solves this problem, but it faild to convert.

Experiment 2
The original file is uncompressed, made a compressed version with MVLApp. Exporting the compressed file's frames they could be successfully opened in RawTherapee. Valid values.
These DNG frames can be successfully processed with cr2hdr application. I processed them as group of 250 frames,4 times paralelled, default options
The processed frames can be stiched into a single MLV file with raw2mlv application.

I suspect there is some problem with the metadata/info fields in the original file. When frames exported, those info get tranferred into the dng, causing bad file.
Compressing the mlv file somehow solves this.
Done on Win10/x64

Here is the processed file
google drive link

I had some success with the .cr2 file. After dropping it into Adobe DNG converter, I've recieved a 16bit uncompressed .dng file.
After many tries, I've managed to open the data. Stangely the inner data width is 5360 opposed with 5202 that is reported by RawTherapee. The additional data is not black, but very dark rectangle on the left. RawTherapee somehow just crops it. Height is 3465.

I've overwritten the values with horizontal neighbour averages, and stuffed the data back into the dng. Black lines are gone, but there are visible artefacts, like above the eyes.

https://drive.google.com/file/d/1Wf6caNNeuPfo5uhd0Aak-s_PRiPGsAs5/view?usp=sharing




edits in italic

I've implemented vertical and horizontal spline interpolation in my monster program. Seems to work on dualiso dng too. My next step would be make it able to write back(or into copy) changes into dng.

Below picture is a difference between corrected and original values. No change is 0, on focus fixel coordinates it's something else.
Playing with exposure to make it visible to humans:

This was from 10bit anamorphic footage. Gradients everywhere. Maybe interpolation is the only sensible way to correct this.

Okay, false alarm about the 20000 focus pixels.
Turns out in MLVApp any kind of Raw Correction affects exported DNG.
Fix Focus Dots was off(the entire Enable Raw Correction off) for the single frame export.
Fix Focus Dots was on for the averaged frame export. It caused massive amount of pixels in this dual iso file.

thank you, done downloading
1 averaged frame would have been fine

@dfort Can i ask for the full length averaged mlv file of the beach scene?

I've mapped ~8700 pixels so far, these were quite visible.
I averaged the 4 frames you gave me, it added another ~20000 additional pixels to the map

Also, is there any way to see which fpm file is in use in MLVApp?

https://www.magiclantern.fm/forum/index.php?topic=20025.msg216138#msg216138
"- Drop focus pixel map files into app to install (except Linux AppImage)"

I've never managed to see it work, but it seems to be the correct way to intall them

Thank you @dfort
I've managed to seperate ISO fields, then seperate the color fields in all 4 frames.
I've noticed the lines are marching with every frame. It can be seen in MVLApp by zooming well into the top, disabled raw correction, and with "none" demosaic. Pressing next/prev frame will show the effect. Not sure if this is intended.
DDLLDDLL==>DLLDDLLD==>LLDDLLDD==>LDDLLDDL
Could it be the crop window is moving down? Even if so, the focus pixels stay in the same place across the frames.

I've made 2 .xcf files, they have the frames in them as layers.
levels adjusted
raw values

Btw just an idea, if we make a pixel map in this separated mode, and we make the "opposite" instructions in reversed order, we should get a pixel map for the original, unmodified picture.

Regular:
Load dng, deinterlace, use this image to make deinterlaced pixelmap. Load back created pixelmap, deinterlace-->profit

Dual iso:
Load dng, separate iso fields, deintelace top/bottom iso field, use this image to make deinterlaced pixelmap. Load back created pixelmap, intelace top/bottom iso field, make iso field alternating again--> profit

Maybe i am able to program pixelmap loading and exporting till weekend.
If these steps implemented without error, next steps would be experiment with values laying on pixelmap coordinates.
Maybe these values are just offset, maybe they are from another iso level.
If this turns out to be a dead end, maybe i'll try interpolating techniques, as last resort.

current program status



EDIT:
I also noticed lines in the overexposed areas in higher iso fields. They align perfectly in top/bottom direction, not sure what this is.

Hi again
I am finally able to open 14bit files, well, more like 16 bit. Basically, MLV App exports 10-14bit files as 16 bit so i can open all of them.

I started to experiment with dual iso files. I wrongly assumed row would be ordered like this:



Instead it looks like this:



I'd like to ask for a Dual ISO MLV/DNG sample, that was correctly opened and interpolated in MLVApp. I am pretty sure my camera settings were wrong, as i can see iso rows are turning opposite by every frame, DLLDDLLDDLLD and next next frame is LDDLLDDLLDDL(dark and light rows)



Flickr can't display this dng so it's a drive link: https://drive.google.com/file/d/1G8QzjHnQI2PRv1KnzVCr2Af-XsFvtDyZ/view?usp=sharing

Thank you @dfort

At the current state my program is only able to separate the fields(and comb them back) by reordering columns and rows. The methods used can be found in Form1.cs under DeinterlaceVertical(),DeinterlaceHorizontal(),InterlaceVertical(),InterlaceHorizontal() names.
They take a two dimensional integer array as input, and they overwrite the input after the transformation.
Yes, it is a Visual Studio 2017 C# windows application project.

It if easier for me to spot focus pixels this way, but it makes harder to make actual focus pixel files. Final coordiantes would be 2X+offsetX; 2Y+offsetY ==> 2X+0; 2Y+0 for my pixels in blue area, blue being 0;0 in my BGGR sensor pattern.

I've separated the testfiles from the sorce code and the build program and made it easier to read. They can be found in the shared folder. https://drive.google.com/drive/folders/1I9eSsfgxjZnuUc3sbdGCCVxIzhCa0MBX

Made some user interface too in p00_v2.zip.
My aim is to be able to import testfile3 and 4's data. They were shot today, with Danne's most recent firmware. TF3 is lossless 14bit, TF4 is uncomressed 14bit.

On my x64 Windows 5.5 RawTherapee makes kid of pop. Near the right corner of the building it is visible as yellow dots.

Neutral profile:
https://flic.kr/p/2fThTrf

And with .badpixel file:
https://flic.kr/p/2fXT1vn

I can make them more visible by separating B,G1,G2,R values. Here is an output from my program. The tiff is too dark by default, so i needed to adjust the level. I have focus pixels only in the blue data = even column, even row coordinates in the original.

https://flic.kr/p/2fXTtvV


I bought a 100D camera, and i made some ML videos a few weeks ago, mainly anamorphic. I'll try my luck with some exported dng from them.

Hello

I have a Xiaomi A1 smartphone, which is able to take .DNG raw files with firmware modification. This sadly also introduces 200k focus pixels in my pictures,that also affects phone generated .JPEGs.
I have a few thousand pictures taken already. RawTherapee does very good job removing them, also I made a .BADPIXELS file to average them out. It was made by following the forum thread. Many thanks for that.

I've decided to get rid of them by modifying the DNG's copy. I've made a simple C# program, that is able to extract the raw data from my dng-s to a greyscale 16bit tiff file without any loss of raw information. Quite sure it is not compatible with anything else. Also, filepaths are hardcoded.

When fiddling about with self made pixel maps, I've noticed that under the focus pixels, the raw data's histogram is veeery similar to the original picture's. My guess would be focus pixel values are just offset from the optimal values, by a constant or maybe a*b+c.
I am a bit excited, maybe it could help removing focus pixels from ML videos, by not averaging, but backtracking the original value, eliminating the averaging articacts or chroma smooth, which affects the whole picture.
I can continue next monday afternoon if it is true to ML expoted dngs.
My phone uses 16bit BGGR pattern.
https://drive.google.com/drive/folders/1I9eSsfgxjZnuUc3sbdGCCVxIzhCa0MBX?usp=sharing

Hello Danne

I don't think your post had aliasing.
Your picture seemingly consists 4 pixel(2 pattern) wide columns
I think every second column is offset by 8 pixels.
Here's what it looks like reordered:





The keyboard keys are a bit blurry, but the icons on the desktop look fine. Was this a handheld shot?
Is this usuable at all?

Apart from the center focus pixels in red and blue, i have noticed some repetetive pattern in both greens as well. Under the cat's head. Very faintly i see it in red and blue as well.

Hello

My aim was to help focus dots disappear and I've succesfully made a RawTherapee compatible badpixel file. My problem is I need to modify dng file datafields to make it work.

I have been fiddling with IDA_ML's raw files from post #1540
I used MLVPoducer to export 5 frames from it as CinemaDNG(uncompessed).

I opened the .dng with RawTherapee (debug enabled). I noticed in the console there is a badpixel related error. The strange thing is, it expects a badpixel file named "Canon EOS 100DMLVProducernonamenonameEF35mm f/2 IS USMCanon EOS.badpixels" to correct badpixels/focus dots.
Notice the "/" in the filename. Nor under windows nor linux let me name files like that.

I've used a program called Exif Pilot to modify/strip unwanted datafileds from the .dng-s.
Now it expects "Canon EOS 100DMLVProducer.badpixels", hurray. Focus dots in Rawtherapee are corrected.

My question is, if i use another program to extract the dng, do i still have this naming problem?
And another: My badpixel(text file) is in
X[space]Y
X[space]Y
format. Is there any known method to convert this into MLVPruducer compatible file?

I use Virtualdub deinterlacing on RT exported raw sensor data .tiff to get the correct coordinates.

https://drive.google.com/drive/folders/14VLaBNXRIe9EJJN-7VcZVAdGU8c3Eaed?usp=sharing

(sorry for the language butchering, native hungarian speaker here)