Dealing with Focus Pixels in raw video

[dfort]

There should be a way to eliminate the focus pixel map files that are currently being used in MLVFS and MLP. Although the EOSM/650D/700D have different raw image size capabilities they share the same raw frame buffer sizes and pixel patterns. The 100D has a different pattern but upon closer examination it turns out that there are many similarities especially in the mv1080 and mv720 modes where line skipping modifies the focus pixels pattern to a point where they become essentially the same pattern.

Here's a screenshot of the upper left corner of the mv720 pattern for the EOSM/650D/700D. Let's call this group pattern A because these cameras all share the same pattern and we'll call the 100D pattern B because it is a newer camera than the ones in the first group.

If you look closely you'll see that the pattern starts at position 72 horizontally, x axis and 290 through 465 in the vertical, y axis. The left 71 pixels is actually in the out of bounds area of the raw frame buffer so we can say that the pattern extends through the entire horizontal area of the picture area in all video modes. Vertically the difference between the patterns is simply that the B pattern occupies a larger area starting at 86 in the y axis and extending all the way to 669.

I wanted to see if it would be possible to create the mv720 raw buffer focus pixel map files with a simple bash script and this is what I came up with.

Code: [Select]

# focus pixel map files (.fpm)
# Proof of concept bash script

width=1808
height=727
output="output.txt"
pattern=A # pattern A for EOSM/650D/700D pattern B for 100D
if   [ $pattern == "A" ]; then fp_start=290; fp_end=465
elif [ $pattern == "B" ]; then fp_start=86;  fp_end=669
else exit 1
fi
for y in $(seq $((fp_start)) $((fp_end)) ); do
    if   (( (($y + 10)) % 12 == 0 )); then shift=2
    elif (( (($y +  9)) % 12 == 0 )); then shift=3
    elif (( (($y +  4)) % 12 == 0 )); then shift=6
    elif (( (($y +  3)) % 12 == 0 )); then shift=7
    else continue
    fi
    for x in $(seq 72 $((width)) ); do # 0-71 is out of bounds
        if (( (($x + $shift)) % 8 == 0 )); then
            echo -e "$x \t $y" >> "$output"
        fi
    done
done

This is just a first attempt of only one video mode and it could definitely be made better but the script does output a file that matches the current focus pixel map files. Using code instead of text files to map the focus pixels on the raw frame buffers would eliminate keeping an inventory of 15 text files that take up 6.2 MB of disk space.

A further refinement would be to bypass creating a map file of the full raw buffer and map only the cropped frame.

[Danne]

6.2mb isn,t much but if you can get it done without those files it is even better. Very interesting following this.

[DeafEyeJedi]

Wow! Would this not only save disk space but perhaps speed up the Automator workflow within MLP n MLVFS?

[dutchfilms]

Hello, so... I've read through this discussion and understood non of its technical (kudos to everyone who builds this firmware).  I've installed ML on my 700D, (yesterday) seen the pink dots and want to know how to remove them! what process do I need to follow to make these pesky dots go away?  Thanks.
Russ

[Walter Schulz]

Try MLP

[dmilligan]

I prefer having separate files even if they are duplicates. It's the simplest and most flexible solution, that is also capable of handling any current (or future) scenario, without making any assumptions about similarities between the cameras and how they operate (we engineers have a saying: KISS). It's also a tiny pittance of memory that is required. Because they are so redundant and also plain text, they compress really well too, so the current download size of MLVFS is only 1.5MB even with all those files included, amongst other things.

If you really want to save space we could keep the files in binary format. Since you need only 2 16 bit integers (4 bytes) to store each coordinate, the pixel file of EOSM et al would be: ~24KB, and the 100D would be ~400KB. But this would make it more of a challenge to edit the files and easily use them for other things.

Even a small amount of code is typically more difficult to maintain than a large data file in the long run => I make a living b/c this statement is true

ideas for supporting legacy RAW video files

This should eliminate the need to support the legacy RAW format.

[dfort]

Hello, so... I've read through this discussion and understood non of its technical (kudos to everyone who builds this firmware).  I've installed ML on my 700D, (yesterday) seen the pink dots and want to know how to remove them! what process do I need to follow to make these pesky dots go away?  Thanks.
Russ

Hi Russ--

This is a "General Development Discussion" topic so it tends to get a bit more into the techie details than than most users really need or want to know. I also started out with a camera that has the "pink dots" issue and my first foray in Magic Lantern code was to compile raw2dng with chroma smoothing turned on so I could deal with those pesky dots.

To answer your question, MLVFS and MLP have both integrated the focus pixel map files that were developed in the course of this forum discussion. MLV Producer for Windows has had pink dot removal before I even started this topic. PinkDotRemover tool 650D is the Grand Daddy of pink dot removal tools but it is no longer maintained.

Even a small amount of code is typically more difficult to maintain than a large data file in the long run => I make a living b/c this statement is true

Ha ha--how true. I guess that I should re-read "The Unix Philosophy" especially the part about storing data in flat text files. I'm not a programer but have been a weekend code hacker for a while and now that I'm retired I spent way too much time bashing out scripts.

I did find a maintenance issue with some of the raw buffer focus pixel map files. In the course of playing around with my scripts I discovered that some of the coordinates define areas that are beyond the edge of the frame. The reason was because one of my scripts would map a pixel at the width in resolution instead of width-1 because the coordinates should start at 0 instead of 1. Apparently it isn't causing any problems but I will fix them on my side.

By the way, you do know that none of this focus pixel fixing is going to work for 650D and 700D users until your mlv_rec fixes get into the nightly builds. I've been making comments on bitbucket for a while. The 100D builds nikfreak has posted for testers is also missing that commit. I had to build it myself to test out that camera. So that leaves only the EOSM nightly builds as the only platform that currently works with MLVFS and MLP focus pixel annihilation.

This should eliminate the need to support the legacy RAW format.

I'll get to testing mlv_lite and report on that topic but first back on this topic--more focus pixels!

I was wondering how the two different focus pixel patterns ended up being the same pattern with line skipping and here's the answer. They are basically the same only the sensor on the 100D has a lot more of them. Not sure if the extra focus pixels become inactive when line skipping is active. In any case, it does make for some interesting observations that adds little value to users who just want to get rid of those pesky dots!

[dfort]

Ok--as promised I fixed the focus pixel map files that had problems. These should be replaced in the applications that are using these files:

Code: [Select]

80000346_1872x1060.fpm
80000346_2592x1108.fpm

These are the mv1080crop and zoom video modes for the Canon SL1/100D. Somehow I slipped on a some lines and a few pixel coodinates were missing. Didn't discover it until I ran diff between the files I had posted and the files generated by my experimental script. So I guess that little scripting exercise wasn't a waste of time after all. Just for fun I made a branch of the mlv2badpixels.sh script that doesn't require the focus pixel map files.

The bitbucket repository is located here: https://bitbucket.org/daniel_fort/ml-focus-pixels

For convenience the fpm files are located in the download area as a zip package.

I also removed all the focus pixel related files from my Magic Lantern fork repository. Those files have already served their purpose.

[Danne]

Nice! Updated the fpm files in MLP.

*Updated MLP to your latest script. Small modification to exclude an error. Sent you an issue report at bitbucket. Very nice going from 6.2MB to 6kb. Nice scripting.

[DeafEyeJedi]

Damn that's some serious digging if you were able to accomplish to go from 6.2MB to 6kb ... Way to go, Dan!

[emoji106]

[dfort]

Actually there was some sort of issue with bitbucket. The commits I made yesterday didn't take and so I had to push them out again this morning. Good thing I still had an extra copy of what I worked on. So actually the script went from 5.4 KB to 7.9 KB. The two focus pixel map files that were fixed yesterday also needed to get updated again this morning. These are the crop mode files for the SL1/100D and are 1 and 1.5 MB files. That's a lot of focus pixels--151,200 on the zoom mode map. That's too much for bitbucket to display or show a diff.

[Macke1]

if someone can help me? i have a few shortmovies  and i cant get rid of of those dots, but i think its my knowledges that is the problem. https://drive.google.com/file/d/0B4KcdNcQPavBaTNkMnBmTmFKbWs/view?usp=sharing

[Danne]

@Macke1
Can you specify what camera and what movie mode you filmed in? Crop mode, regular mode? 720 mode?

[Danne]

Seems you filmed with a 700D and in regular mv1080 mode. When I select (02) ,click the picture below, from the menu all three clips gets the correct pixel maps from dforts script. See here as well.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg162922#msg162922

What is your problem?

[dfort]

@Macke1

I'm seeing the same non-issue as Danne.

Before focus pixel fixing


After focus pixel fixing

I'll admit that I did have some problems at first but after trying a couple of times it worked. Maybe was is because the shots were very short or maybe it was because I was trying to do one at a time? In any case this is a new feature in MLP.

Speaking of new features, guess the cat is out of the bag. I have been experimenting with that formula my brother wrote to take an image size and figure out the pan information so my mlv2badpixels script could work with v1.0 raw video files--I can see dmilligan rolling his eyes.

Allowing for the user to enter information can also be useful if the original MLV file has incorrect metadata, like the camera missing issue that was recently resolved or if the MLV file is missing but the DNG files for that shot are available. Danne was able to pull out some of the information from raw files so the user just needs to provide the camera and video mode.

Of course like most things in Magic Lantern this is a work in progress. The script got a little ugly with the added features but it does seem to be working. You've got to be organized and make sure you don't mix up cameras and video modes. I've also been very careful about making sure that the 5x zoom box is perfectly centered when using zoom mode. However, I've been doing some tests and so far it seems to be working even when the zoom box is moved around. That makes no sense and requires more testing but if it is true it certainly is good news.

[DeafEyeJedi]

Digging your amazing work on these humpy dots with some quick test files shot from SL1 ... Really nice actually and Thanks @Danne for implementing his work into MLP!

Pre:                                                                                                                 Post:

Dual-ISO's coming up next...   

[Macke1]

@dfort ok, now i tried using 02 and it worked like a charm, will try more later this day!

[Frank7D]

@dfort,

Is it just me, or does the "after" picture look a bit blurry compared with the "before?"

[dfort]

They are different frames and the camera was shaky so it wasn't exactly a before and after test. I was just showing that the focus pixels are being removed.

Here's a better example, RAW video shot with SL-1/100D in mv1080 mode. This camera shows lots of focus pixels and all those colored focus pixels makes it look like there's a color shift.

As it comes out of the camera

After focus pixel removal

Here's a blow up A+B comparison for dramatic effect. It seems like resolution isn't being affected. Note that there are still a few bright pixels in sharp high contrast areas. I'm not sure if those were caused by the "badpixels" interpolation in dcraw or the debayering algorithm. Most likely it is aliasing artifacts. Still, quite an improvement.

P.S. if we get this working and create all the maps, we can also add it to mlv_dump.

Is it working? 

[Danne]

This is amazing.

[DeafEyeJedi]

Add this script into mlv_dump and it'll be golden!!!

[dfort]

Actually it is the focus maps and the pixel interpolating code that is needed for mlv_dump. The script I wrote is just for quick prototyping, though it is playing nicely with dcraw via MLP.

The script uses mlv_dump to extract the metadata from MLV files and exiftool for DNG files. There's nothing that I know of for RAW though as a test I did hack raw2dng so it displays just the file information without creating the DNG files. I'd like to be able to figure out how to read the MLV header and RAW footer with a simple bash (or lua, python, perl) script so it won't require these dependencies--and to learn how to work with binary files too.

There's a lot more that can be done with focus pixels in post production. For example the Adobe DNG specifications has a FixBadPixelsConstant and a FixBadPixelsList (Opcode ID 4 and 5) in the documentation but I have not seen any examples of how this can be used.

[dfort]

Since this is a General Development topic maybe someone can help me figure out how to read MLV headers and RAW footers in order to find the information needed to deal with focus pixels?

Even though the raw format may be on its last legs I thought I'd start there because there's only one thing to find, image size. The footer of a raw file is fairly simple. Here's one that's easy to find because it has some blank space before the footer.

Running xxd on the raw file results in this:

Code: [Select]

49dffd0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
49dffe0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
49dfff0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
49e0000: 5241 574d 0005 d002 00a0 1800 3000 0000  RAWM........0...
49e0010: 0100 0000 a45d 0000 0300 0000 e84a 4fff  .....].......JO.
49e0020: 0100 0000 0081 7906 a604 0000 1007 0000  ......y.........
49e0030: 5c0c 0000 a873 3900 0e00 0000 ff07 0000  \....s9.........
49e0040: 983a 0000 0000 0000 0000 0000 c806 0000  .:..............
49e0050: 8604 0000 1c00 0000 4800 0000 a204 0000  ........H.......
49e0060: 1007 0000 0000 0000 0000 0000 0001 0102  ................
49e0070: 0100 0000 ca19 0000 1027 0000 b7fc ffff  .........'......
49e0080: 1027 0000 55fc ffff 1027 0000 88ee ffff  .'..U....'......
49e0090: 1027 0000 aa30 0000 1027 0000 c708 0000  .'...0...'......
49e00a0: 1027 0000 31fc ffff 1027 0000 f707 0000  .'..1....'......
49e00b0: 1027 0000 0418 0000 1027 0000 4b04 0000  .'.......'..K...

The stuff on the right is probably meaningless in this situation. The best explanation I found how to read this is in a post about how to replace a file footer. It starts with a snippet from Magic Lantern / modules / lv_rec / lv_rec.h

Code: [Select]

#ifndef _lv_rec_h_
#define _lv_rec_h_

#include "raw.h"

/* file footer data */
typedef struct
{
    unsigned char magic[4];
    unsigned short xRes;
    unsigned short yRes;
    unsigned int frameSize;
    unsigned int frameCount;
    unsigned int frameSkip;
    unsigned int sourceFpsx1000;
    unsigned int reserved3;
    unsigned int reserved4;
    struct raw_info raw_info;
} lv_rec_file_footer_t;

#endif
where each char is 1 byte (2 hex characters), short is 2 bytes, and int is 4 bytes, that way you can crossreference the struct definition with the number's you are editing in hex.

the raw hex stores these numbers in "little endian" which means that if you want to obtain the real value of an integer from raw hex you would have to write it in reverse( raw hex: "52 41 57 4D 80 07 38 04 00 5F 37 00 F4 21 00 00 ..."  is 00 00 21 F4 or just 0x21F4)
I used windows calculator in programmer mode to convert from hex to dec quickly (f4 f5)

I used footers from raw videos that were correct to calculate how many frames i had using the framesize field which is the size of a frame in bytes.  I took the file size and divided by the frame size to get a rough rounded down frame count that i would translate into hex, into little endian and write into the raw hex in that location in the footer.

I can't quite follow that. Does he mean to start at the end of the file and go through it backwards and suddenly it will become clear? Is this like Leonardo DaVinci's writing that you have to hold up a mirror to read his notes?

Looking at the raw2dng source code is this how it it is done?

Code: [Select]

                case 0x100: /* ImageWidth */
                    printf("width: %d\n", data);
                    raw_info.width = data;
                    break;
                case 0x101: /* ImageLength */
                    printf("height: %d\n", data);
                    raw_info.height = data;
                    break;
I'm a bit lost. Can someone help me with this? Remember I'm just trying to do this with a simple shell script.

[Walter Schulz]

When it comes to MLV you better stick with g3gg0 directly because MLV is his brainchild.
And everyone involved in developing MLV converters should know, of course.

[dfort]

When it comes to MLV you better stick with g3gg0 directly because MLV is his brainchild.

Yes, of course, here's the MLV topic I've been trying to understand: http://magiclantern.fm/forum/index.php?topic=7122.0

It is sort of like g3gg0 wrote a novel and I'm still learning the alphabet.