Dealing with Focus Pixels in raw video

Started by dfort, October 22, 2015, 11:09:10 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

dmilligan

To understand the RAW footer you need to look at the struct definitions. To write the data to the file we simply dump the struct's in memory representation to file.


typedef struct
{
    unsigned char magic[4];
    unsigned short xRes;
    unsigned short yRes;
    unsigned int frameSize;
    unsigned int frameCount;
    unsigned int frameSkip;
    unsigned int sourceFpsx1000;
    unsigned int reserved3;
    unsigned int reserved4;
    struct raw_info raw_info;
} lv_rec_file_footer_t;

Notice it references struct raw_info. So basically just insert raw_info struct at the end:

struct raw_info {
    uint32_t api_version;           // increase this when changing the structure
    #if INTPTR_MAX == INT32_MAX     // only works on 32-bit systems
    void* buffer;                   // points to image data
    #else
    uint32_t do_not_use_this;       // this can't work on 64-bit systems
    #endif
   
    int32_t height, width, pitch;
    int32_t frame_size;
    int32_t bits_per_pixel;         // 14

    int32_t black_level;            // autodetected
    int32_t white_level;            // somewhere around 13000 - 16000, varies with camera, settings etc
                                    // would be best to autodetect it, but we can't do this reliably yet
    union                           // DNG JPEG info
    {
        struct
        {
            int32_t x, y;           // DNG JPEG top left corner
            int32_t width, height;  // DNG JPEG size
        } jpeg;
        struct
        {
            int32_t origin[2];
            int32_t size[2];
        } crop;
    };
    union                       // DNG active sensor area (Y1, X1, Y2, X2)
    {
        struct
        {
            int32_t y1, x1, y2, x2;
        } active_area;
        int32_t dng_active_area[4];
    };
    int32_t exposure_bias[2];       // DNG Exposure Bias (idk what's that)
    int32_t cfa_pattern;            // stick to 0x02010100 (RGBG) if you can
    int32_t calibration_illuminant1;
    int32_t color_matrix1[18];      // DNG Color Matrix
    int32_t dynamic_range;          // EV x100, from analyzing black level and noise (very close to DxO)
};


Each field in the struct is stored one after another, and each field's size depends on the data type, so here's the sizes you need to know:
int, unsigned int, or int32_t: 4 bytes
short or unsigned short: 2 bytes
int32_t array[n]: 4 * n bytes
char array[n]: n bytes

a union is sort of like saying a particular memory location can have different names and data types that use up the same space. To simply things lets just leave out the unneeded extra interpretations, and merge these two definitions in the final resulting definition:

typedef struct
{
    unsigned char magic[4];
    unsigned short xRes;
    unsigned short yRes;
    unsigned int frameSize;
    unsigned int frameCount;
    unsigned int frameSkip;
    unsigned int sourceFpsx1000;
    unsigned int reserved3;
    unsigned int reserved4;
    uint32_t api_version;           // increase this when changing the structure
    uint32_t do_not_use_this;
    int32_t height, width, pitch;
    int32_t frame_size;
    int32_t bits_per_pixel;         // 14
    int32_t black_level;            // autodetected
    int32_t white_level;            // somewhere around 13000 - 16000, varies with camera, settings etc
    int32_t jpeg.x;
    int32_t jpeg.y;           // DNG JPEG top left corner
    int32_t jpeg.width;
    int32_t jpeg.height;
    int32_t active_area .y1;
    int32_t  active_area x1;
    int32_t  y2;
    int32_t  x2;
    int32_t exposure_bias[2];       // DNG Exposure Bias (idk what's that)
    int32_t cfa_pattern;            // stick to 0x02010100 (RGBG) if you can
    int32_t calibration_illuminant1;
    int32_t color_matrix1[18];      // DNG Color Matrix
    int32_t dynamic_range;          // EV x100, from analyzing black level and noise (very close to DxO)
};

So, the first 4 bytes will be 'magic' (an identifier) and they are supposed to be set to "RAWM"
next is 2 bytes xRes, and 2 bytes yRes, 4 bytes frameCount, and so on and so fort.

It's important to note that the data is little endian, which means the bytes come in the backwards order that would probably make sense to you. The LSB is first. So a 4 byte int that appears in the file like: 0x010203040 that value is actually: 0x40302010 = 1076895760. Your hex editor probably has an option to display the little endian value of whatever you have selected.

MLV works the same way except the data starts at the beginning of the file. See mlv.h for the struct definitions.[/code]

dfort

Wow--thank you very much. Now I've something to chew on.

dfort

Looks like it wasn't all that difficult to read the raw footer in a shell script but it isn't nearly as elegant as dmilligan demonstrated in C. Maybe it is time to re-write mlv2badpixels in C?

raw_footer.sh
# The RAW file footer is 192 bytes long so we need to
# back up 192 bytes (hexadecimal c0) from the end of the file.
# The -s is the seek offset and the hex number is negative
# because we're seeking from the end of the file.
# The -g 1 option groups the output in single bytes to make
# it slightly more human readable.

printf "\nRAW footer:\n"
xxd -s -0xc0 -g 1 "$1"

# Note the 4-byte 'magic' RAWM at the start of the footer.
# We're interested in the frame size of the raw file
# and the two bytes following RAWM, 188 bytes (hex bc)
# from the end is the image width size in pixels.
# We want just those two bytes so we'll suppress everything
# else (-p) so grouping no longer can be applied.

printf "\nWidth in hexidecimal:\n"
xxd -s -0xbc -l 2 -p "$1"

# Print this out as a decimal number doesn't work because
# it is "little endian" and the bytes need to be read in
# reverse order. Note the "0x" to tell printf that this is
# a hex number.

printf "\nNormal hexadecimal to decimal conversion doesn't work on little endian encoding:\n"
printf "%d\n" 0x`xxd -s -0xbc -l 2 -p "$1"`

# The solution is to put the hex number in a variable and
# print it out in reverse order.

printf "\nConverted from little endian encoding:\n"
w=`xxd -s -0xbc -l 2 -p "$1"`
printf "${w:2:2}${w:0:2} \n"

# Finally print out the image width from the raw footer.

printf "\nImage width taken from the raw footer:\n"
printf "%d\n" 0x"${w:2:2}${w:0:2}"

# Take the next two bytes and do the same thing for the image height.

printf "\nImage height taken from the raw footer:\n"
h=`xxd -s -0xba -l 2 -p "$1"`
printf "%d\n" 0x"${h:2:2}${h:0:2}"


Here's a sample output:
./raw_footer.sh 700D_zoom.RAW

RAW footer:
3d90000: 52 41 57 4d 00 05 d0 02 00 a0 18 00 28 00 00 00  RAWM........(...
3d90010: 01 00 00 00 ac 5d 00 00 03 00 00 00 e8 4a 4f ff  .....].......JO.
3d90020: 01 00 00 00 00 81 79 04 24 04 00 00 50 07 00 00  ......y.$...P...
3d90030: cc 0c 00 00 b0 fc 34 00 0e 00 00 00 ff 07 00 00  ......4.........
3d90040: 98 3a 00 00 00 00 00 00 00 00 00 00 08 07 00 00  .:..............
3d90050: 08 04 00 00 1c 00 00 00 48 00 00 00 24 04 00 00  ........H...$...
3d90060: 50 07 00 00 00 00 00 00 00 00 00 00 00 01 01 02  P...............
3d90070: 01 00 00 00 ca 19 00 00 10 27 00 00 b7 fc ff ff  .........'......
3d90080: 10 27 00 00 55 fc ff ff 10 27 00 00 88 ee ff ff  .'..U....'......
3d90090: 10 27 00 00 aa 30 00 00 10 27 00 00 c7 08 00 00  .'...0...'......
3d900a0: 10 27 00 00 31 fc ff ff 10 27 00 00 f7 07 00 00  .'..1....'......
3d900b0: 10 27 00 00 04 18 00 00 10 27 00 00 4b 04 00 00  .'.......'..K...

Width in hexidecimal:
0005

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
5

Converted from little endian encoding:
0500

Image width taken from the raw footer:
1280

Image height taken from the raw footer:
720


I'll update mlv2badpixels.sh so the user doesn't need to manually enter the image size for a v1.0 raw video file then I'll start reading some C programming tutorials.

Thanks again dmilligan!

Danne


DeafEyeJedi

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Since it was possible to read the RAW footer with a bash script I thought I'd try it with an MLV header and it works. I'll update mlv2badpixels.sh so that it isn't dependent on mlv_dump. It doesn't really improve the script and it adds a dependency on xxd but it is sort of a proof of concept that working with RAW and MLV files is possible with rather simple shell scripts. Of course C is still a better choice.

mlv_header.sh
# Some more command line tricks this time
# for reading MLV headers.

# Most of the information seems to be in
# a fixed offset location and is rather
# easy to find.

# Note that little-endian words need to
# be read right to left.

printf "resolution width: "
h=`xxd -s 0x44 -l 2 -p "$1"`
printf "%d\n" 0x"${h:2:2}${h:0:2}"

printf "resolution height: "
h=`xxd -s 0x46 -l 2 -p "$1"`
printf "%d\n" 0x"${h:2:2}${h:0:2}"

printf "raw buffer height: "
h=`xxd -s 0x50 -l 2 -p "$1"`
printf "%d\n" 0x"${h:2:2}${h:0:2}"

printf "raw buffer width: "
h=`xxd -s 0x54 -l 2 -p "$1"`
printf "%d\n" 0x"${h:2:2}${h:0:2}"

printf "camera information: "
h=`xxd -s 0x01dc -l 6 -p "$1"`
printf "%s\n" 0x"${h:6:2}${h:4:2}${h:2:2}${h:0:2}"

# However, the crop and pan is recorded for each
# frame so we need to find the 'magic' that
# identifies that block. We're not going to attempt
# to support dynamic panning for now so let's just
# grab the first frame.

# Find the first "VIDF" block, "56 49 44 46" in hex
# the 1!N; append the next line when not on the last
# line in case "VDIF" is split by a newline. Keep
# a large enough chunk to include where the panX
# and panY information is located.

# For example, here's a typical first VDIF match:
#
# 0000290: 56 49 44 46 f0 5d 0a 00 b4 39 07 00 00 00 00 00  VIDF.]...9......
# 00002a0: 00 00 00 00 f0 02 ee 01 e9 02 ee 01 b0 0f 00 00  ................
#                                  ^^ ^^ ^^ ^^ This is what we need

k=`xxd -p "$1" | sed -n '1!N; /56494446/{p;n;p;q;}' | tr -d '\n' |  grep -Eo -m 1 '56494446.{48}'`

# Next find the panX and cropX and print them
# to the terminal:

printf "panX: "
printf "%d\n" 0x"${k:50:2}${k:48:2}"
printf "panY: "
printf "%d\n" 0x"${k:54:2}${k:52:2}"

Roberto Mena

I cannot find a decent way of dealing with these damn focus pixels shot on a T5i/700D using a Mac and Adobe CC workflow. Most of the apps recommended on here and online are all PC based like pinkdotremover and I cannot get them to work on my Mac. Does anyone have solutions that doesn't involve coding or something that extreme? I keep reading something about using "chroma smoothing" to fix the issue but I find no info how to do this online either or what it actually is and I am an AV tech. Please, any Mac user out there with the similar issue, help me!

Thanks.

dmilligan


dfort

@Roberto_Mena

There are several tools that work on Mac. dmilligan pointed out his MLVFS which has been working on Mac and Linux for quite a while and more recently on Windows. What is great about MLVFS is that the DNG files are created on the fly with the focus pixels removed. In fact if you would have started with MLVFS you might not even know there is an issue with focus pixels on your T5i/700D.

Another fine tool is MLP. It can work in conjunction with MLVFS or via mlv_dump which is included in the MLP.workflow package. Note that when using mlv_dump the DNG files will still have the focus pixels but they are removed in the MOV files.

Finally, though a Windows program MLV Producer can run under WINE on the Mac and it too can remove the focus pixels.

I have also written some shell scripts that can be used with dcraw if you want to roll your own application. This script is also used by MLP.

Roberto Mena

Thanks for this new info! I will try it out!

Roberto Mena

I finally got MLVFS and FUSE to work and got rid of my focus pixel issue! Thank you soooo much dfort and dmilligan for all your help you guys are awesome! You can't imagine how much this means to me to finally get this working!

Roberto Mena

Hi guys. So my latest test attempts with MLVFS do get rid of the "focus pixel" issue associated with the T5i/700D but I am now getting different weird artifacts here and there. One of the artifacts looks like purple/pinkish dots that seem to sparkle in and out of the image. The other type of artifact looks like spots of my video are turning into old school video game graphics like old pixels that kind bleed into small spots of my video. Most of these artifacts are not on my image and they are not stable like the "focus pixel" issue I was getting before. Any ideas how to fix this and what settings should be set to 'On' with MLVFS (i.e. Bad Pixel Fix, Vertical Stripes Fix, etc.)? Thanks!

dmilligan

Sounds like aliasing. Upload a sample DNG, all processing options should be off.

Roberto Mena

I am certain its not aliasing because Im familiar with what that looks like. How big of a DNG sample size do you need? Like one DNG frame? A couple of seconds? Thanks in advance.

dfort

One DNG would be good for a start but even posting a jpeg here would help. Best of all would be a short MLV file that we can download and examine. That would take out the possibility that the issue was introduced in your post process.

Roberto Mena

I tested MLVFS with only one setting turned on which was Chroma Smoothing at 5x5 and it seems to have fixed the issues. I'll keep you updated. Thank you.

Welles

Hi, I have a 100D/SL1, and by default the raw video is unusable, with a pattern that resembles the one described earlier in this thread.

Thanks to your amazing work, using MLVFS, almost all of these evil dots are gone, and the footages are now so much better. However, I still have some dots left (the most obvious is two lines near up and down) are some artifacts. I wonder if I'm the only one, or if this is a problem that you are aware about.

Here is an example of a 1440×602 raw video, with various settings enabled. I don't know in if I was in mv1080, mv1080crop. The only modified settings is RAW VIDEO, 1440×602.

No fix at all : https://picload.org/image/rglowipw/nofix.png

With bad pixel on : https://picload.org/image/rglowipi/bad_on.png
Aggressive : https://picload.org/view/rglowiiw/agressive.png.html

Bad + 2×2 Chroma: https://picload.org/image/rglowipa/bad_chroma22.png
Bad + 3×3 Chroma: https://picload.org/image/rglowipl/bad_chroma33.png

Aggressive + 5×5 Chroma: https://picload.org/image/rglowipr/agressive_chroma55.png





100D.100B / EF-S 10-18mm IS STM / EF-S 18-55mm IS STM / EF-S 55-250 mm IS II / EF 40mm STM

dmilligan

I'm a little confused, with MLVFS the focus pixels should be gone even with nothing "turned on". If you have to turn on bad pixel fix or chroma smooth, then focus pixel fix is not working.

Welles

Oh! Too bad... Yes, in the provided "nofix.png" picture, nothing is turned on, using the mounted MLV.

I use a fresh Windows 10 Pro with Dokan 0.8.0 (redist version). I copied the directory containing all the unzipped files of "MLVFS_x86.zip" in "c:\Program Files (x86)". The user (me) executing mlvfs.exe is admin class, as it seemed to work, but cmd itself is "not run as admin". I run mlvfs.exe in this directory.

mlvfs.exe d:\mnt --mlv_dir=d:\mlv

Am I doing something that can cause the malfunction? What can I test that might help?
100D.100B / EF-S 10-18mm IS STM / EF-S 18-55mm IS STM / EF-S 55-250 mm IS II / EF 40mm STM

Welles

OK, solved! "Run as Admin" cmd is mandatory in order to get all the dots away with nothing turned on.

I still get some artifacts on the white cars and power cables I can't seem to get rid of, is there anything I can do about that?

https://picload.org/image/rglcpccd/ok.png
100D.100B / EF-S 10-18mm IS STM / EF-S 18-55mm IS STM / EF-S 55-250 mm IS II / EF 40mm STM

Frank7D

@ Welles, looks like aliasing. No perfect solution, but there have been a lot of suggestions for reducing or preventing it if you look around. Crop mode nearly eliminates it (but has its own drawbacks).

dfort

Crop mode does help with aliasing but this looks like an issue with interpolating values to fill in the focus pixels. Don't know if I got the terminology correct but basically if the area surrounding a focus pixel has fine detail and high contrast it is difficult to come up with a value that will hide the focus pixel. This is more of a problem with dual iso. Overall it looks pretty good.


Upon closer examination the problem becomes apparent.


Have you tried adding some chroma smoothing? Don't do the bad pixel fix, just chroma smoothing.

Greg

Someone compared FRSP vs LV raw crop frame?

If the hybrid AF is like a dual pixel, one photodiode is composed of two :
http://www.imaging-resource.com/ee_uploads/news/4486/hybrid-cmos-af-iii.jpg

There is a possibility that in the photo mode, data of halves focus pixel are averaged. What is the result, focus pixel works as normal pixel.

So maybe just change the ADTG/CMOS registry to get a image without focus pixels.

DeafEyeJedi

Quote from: dfort on May 20, 2016, 11:25:45 PM
Have you tried adding some chroma smoothing? Don't do the bad pixel fix, just chroma smoothing.

This should help a bit. But this is to be expected when shooting Dual-ISO videos from our DSLR's.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Quote from: Greg on May 21, 2016, 12:48:35 AM
Someone compared FRSP vs LV raw crop frame?

There are no focus pixel issues with FRSP.

Quote from: Greg on May 21, 2016, 12:48:35 AM
There is a possibility that in the photo mode, data of halves focus pixel are averaged. What is the result, focus pixel works as normal pixel.

So maybe just change the ADTG/CMOS registry to get a image without focus pixels.

I've said several times that the ideal way to deal with focus pixels is to do it in camera but that's way over my head. Exactly how would you do it?