Raw video interlaced for SD camera

[otherman]

SD camera has big limit in resolution for raw recording, and interlaced video can be a good work-around. 1080i can be converted in a good-enough 1080p with 3D deinterlacing magic from avisynth and his filter, and this tecnique can bring fake-4k raw to CF camera too!
It's not the real thing, but 3D deinterlacing can bring some amazing resoult. I'll definitely pay for such a feature!

[a1ex]

Proof please.

Record a few seconds of full-resolution raw, discard odd/even lines to make it interlaced, then reconstruct it with your proposed method and compare the result with the original.

If your results are convincing, I'll implement it.

If not, I'll ask the moderators to move this to Duplicate Questions

[Audionut]

imagesource("x:\00%d.tif", use_DevIL=true)
separatefields()
selecteven()
weave()
converttoyv12()
QTGMC(preset="slow")

source
interlaced frame
deinterlaced frame

Don't know why someone hasn't suggest interlacing earlier, the broadcast industry has been using it since the begging of time. 

[a1ex]

For deinterlacing, you need at least two source frames, otherwise it's just vertical interpolation. Also, if the scene is static (or a pan from a static scene), reconstructing the original frame is very easy (and you may as well take a still picture).

For our purpose, the source frame has to be be RAW (Bayer), not RGB. So, you will need RG lines from one frame and GB lines from another. Or, you may group lines by 2, as with dual iso, to keep full color info in each frame.

So, keep trying

[Audionut]

Yes, my source was 10 frames.

separatefields()  <- separate even lines into half height frame, same with odd lines.
selecteven()  < discard the resulting odd frames from above
weave()  < combine the even frames to produce original resolution frame
QTGMC()  < deinterlacer.

We should not be worried about the deinterlacing.  This is well tested (and accepted) thanks to the broadcast industry.   
Having to deinterlace double line interlacing, or separate RG - GB fields, this will be more complicated.  I'll see what I can do with some dual ISO frames.

[otherman]

I made the very same test of Audionut, so no point to upload it
I'm going to make some test with 2 lines ('couse I've not a clue to how to manage the  "RG and GB" thing)

[Audionut]

Yeah I don't think we should be worrying about deinterlacing, per se, if some other deinterlace(s) can perform the required action.  Reinventing the wheel

The issue will arise, if none of the currently available deinterlacers, can handle dual line interlacing, in any fashion.

[otherman]

For now, I'm tryng to make an avisynth script to make "2 lines" interlaced" script (to deinterlace after). Sugestion?

[Audionut]

Can't think of an easy way with avisynth.

I was going to use some dual ISO frames with "cr2hdr --debug-blend", which will output the separated frames.  Then just weave the bright, or the dark frames together.   

[otherman]

I manage to get this frame to experiment, and I believe we have to find a way to manage the "RG and GB" thing  : 2 lines don't seems to work.


And after some test with "RG and GB", I know that if it's possible, I don't kow how to do 

[otherman]

If somebody want to try something, this is a 5 frames clip interlaced 2by2

[otherman]

On RG and GB: we could reconstruct blue channel for the first field by motion estimation interpolation of the second field, and viceversa for the red channel, using full temporal resolution green channel for good motion estimation. Yes, I think this is the way; "2 rows" deinterlace lose too much details.

[Audionut]

a1ex, can you describe how to make cr2hdr output original resolution, bright and dark DNG?

Currently, it looks like it is interpolating the bright and dark exposure to as shot resolution.

[a1ex]

The closest thing is with --debug-amaze. However, before reaching this step, it will try to identify bright and dark fields, and in this case it will fail (so you need to plug some fake values instead of the autodetection).

I suggest loading the raw data in Octave (or Matlab if you have a license). If there is interest, I can make a tutorial, covering how to load raw files, how to apply some filters, how to export the output, and maybe even some basic debayering and a very simple converter for dual ISO.

This could be a good starting point for anyone who wants to research raw image processing algorithms (including FPN reduction, denoising, pink pixel fixing, super-resolution or whatever else).

[otherman]

I'm so near to get it! (the avisynth part )
I've sinthetized the new red channel for GB rows , but I'm getting a ghosts of near frames too. I'm sure it's just for my non-existent coding-skills.

Audionut, can you help me? This is my post on avisynth forum where I explain what I did and the problem.

[Audionut]

If there is interest, I can make a tutorial, covering how to load raw files, how to apply some filters, how to export the output, and maybe even some basic debayering and a very simple converter for dual ISO.

Registering interest.   

Audionut, can you help me? This is my post on avisynth forum where I explain what I did and the problem.

I never got into motion estimation with mvtools.  Each color channel doesn't include full lines of resolution, only pixels.  So you may need to do an vertical separation also.  See page 5 of this PDF.

But then you only have quarter resolution detail for those channels that you want to interpolate., so it may not be useful.  It may be best to show a1ex that deinterlacing two fields source is possible, first. 

Once the theory is shown to be sound, I'm sure he will be more interested in taking it to the next level.   

[otherman]

I already did it. 
It's frustating to know to be so close

[Audionut]

Ah sorry, I meant vertical separation.  Check my edit.

[otherman]

from my experiment, with deinterlacing two fields source you get quarter resolution too 

[Audionut]

How did you determine that?  The source (dual ISO) is half resolution interlaced, just like all other interlaced content.  Simply scanned differently.  A regular deinterlacer that expects single line fields, will struggle.

Donald Graft seems to think it will be extremely easy.  Just need to provide them with an accurate source I guess.

[otherman]

I'l try again.

[otherman]

I don't unterstand: in a theoretical interlaced frame we have all red point (not half) of a time A, and all blue point of a time B. Is it not possible debayer it normaly, getting full red channel of time A (but no of time B), and full blue channel of time B (but no of time A)? Maybe have half green point for time A, and half green point of time B will get wrong output?

Untill some days ago, I thought any pixel had all 3 channels, I didn't know all debayer thing so... sorry for the stupid questions.

[a1ex]

If you debayer in camera, it's no longer RAW, it's 8-bit YUV.

You had YUV422 recording and silent pictures before raw was found, but nobody was impressed by that.

[otherman]

I was thinking of debayer on pc and deinterlace after that: am I getting it wrong?

[otherman]

and... whait a minute: uncompressed YUV422 video? I want that!

[a1ex]

Yes, that's the only way you can do it.

But:

- if you skip one line from the raw stream, you will have only RGRGRG lines in half of the frames, and GBGBGB lines in the other half
- if you skip two lines from the raw stream, you have RGB info in each frame, but debayering each frame separately may get you into trouble on the skipped lines

I can create a modified raw2dng that creates interlaced DNG from a normal raw stream, so you can get proper test data.

Uncompressed 422 frames are larger than raw video ones, so I doubt anyone would ever find a good reason to resurrect it. To check the quality, install the stable 2.3 and try the silent picture option (that one is working for sure). You can also compile the old lv_rec module, but you may need to use the codebase right before raw_rec was created. Or just get a HDMI recorder.

[otherman]

well, in my case larger=useless

skipping one line takes problems in debayering? Audionut said it take resolution down to 1/4: if it's true, it is useless too.

[Audionut]

I can create a modified raw2dng that creates interlaced DNG from a normal raw stream, so you can get proper test data.

Yes please, I would like an accurate source stream before moving to the next step.  My original attempt was to use dual ISO 100/200 on a raw stream.  This way, I can take the ISO 100 exposure from 1 frame, and the ISO 200 exposure (darkened) from the next frame (repeating).  But extracting the original (half) resolution bright and dark frames, is more difficult then I expected.

skipping one line takes problems in debayering? Audionut said it take resolution down to 1/4: if it's true, it is useless too.

Well, when you debayer a normal image, you have 4 pixels for the debayering. 

At a closer look, the ISO alternates every two lines, so the pattern looks like this: l, H, H, l, l, H,
H, l, l. You may say it would have been better to alternate every single line, right?

Well, not quite. With ISO alternating every two lines, you get complete RGGB cells at ISO 100
and complete RGGB cells at ISO 1600. With ISO alternating every single line, you would get the
RG pixels at ISO 100 and the GB pixels at ISO 1600. Good luck interpolating that

By using single lines in an interlaced video manner, the issue isn't one where the other line may be overexposed, the issue is one where the motion is probably different.  But we can't simply motion interpolate the 2 frames, as each frame doesn't even contain the full RGB data, and it is very likely, that the RG pixels don't contain accurate information for the GB pixels, and vice versa (motion difference).

When I mentioned quarter resolution, this was in regards to the data you were using for the interpolation.

Each color channel doesn't include full lines of resolution, only pixels.  So you may need to do an vertical separation also.

But then you only have quarter resolution detail for those channels that you want to interpolate., so it may not be useful.

Of course, doing both vertical and horizontal separation of the image, leaves only 1/4th of the data. 

[otherman]

Uhm, what about this:
in mlv a tag say the stream is interlaced
Raw2dng reed the tag, than separate fields and create new fake fields (gb for the rg, and viceversa) with value=0. This way we can have a normal debayer of rg fields of a time A, without interference by gb fields of the time B, am I right? (The other way around is to write a new specific debayer algorithme... But it seems pretty complicated!)
If I'm correct, than we have full temphoral resolution on Green channel to use in motion estimation that would be sufficent, i hope, to reconstruct well missing information.

[a1ex]

Patch for raw2dng to output interlaced frames, filled with 0 where the data is missing:

Code: [Select]

diff -r 7e8078716621 modules/lv_rec/raw2dng.c
--- a/modules/lv_rec/raw2dng.c Wed May 28 08:59:17 2014 +0300
+++ b/modules/lv_rec/raw2dng.c Wed May 28 10:15:10 2014 +0300
@@ -43,6 +43,8 @@
 void find_and_fix_cold_pixels(int fix, int framenumber);
 void chroma_smooth();
 
+void interlace_experiment(int frame_number);
+
 int fix_cold_pixels = 1; //1=fix cold pixels, 0=disable
 
 #define EV_RESOLUTION 32768
@@ -136,13 +138,15 @@
         char fn[100];
         snprintf(fn, sizeof(fn), "%s%06d.dng", prefix, framenumber);
 
-        fix_vertical_stripes();
-        find_and_fix_cold_pixels(fix_cold_pixels, framenumber);
+        //~ fix_vertical_stripes();
+        //~ find_and_fix_cold_pixels(fix_cold_pixels, framenumber);
 
         #ifdef CHROMA_SMOOTH
-        chroma_smooth();
+        //~ chroma_smooth();
         #endif
 
+        interlace_experiment(framenumber);
+
         dng_set_framerate(lv_rec_footer.sourceFpsx1000);
         save_dng(fn, &raw_info);
     }
@@ -1158,3 +1162,18 @@
     free(aux2);
 }
 #endif
+
+void interlace_experiment(int frame_number)
+{
+    int group_lines = 1; /* or 2 */
+    int i;
+    for (i = 0; i < raw_info.height; i++)
+    {
+        if ((i/group_lines) % 2 == frame_number % 2)
+        {
+            /* fake interlacing by filling half of the lines with 0 on even frames, and the other half on odd frames */
+            void* row = (void*)raw_info.buffer + i * raw_info.pitch;
+            memset(row, 0, raw_info.pitch);
+        }
+    }
+}


[Audionut]

I added lv_rec to the end of the module build queue, but getting lots of errors around EDmac.

Code: [Select]



    Building module lv_rec...
    REBUILDING
    [ README   ]   module_strings.h
    Warning: 'Author/Authors' tag is missing. You should tell the world who wrote your module ;)
    Warning: 'License' tag is missing. Under what conditions we can use your module? Can we publish modified versions?
    Warning: 'Summary' tag is missing. It should be displayed as help in the Modules tab.
    [ CC       ]   lv_rec.o
    lv_rec.c: In function 'lv_rec_vsync_cbr':
    lv_rec.c:374:9: error: implicit declaration of function 'ConnectWriteEDmac' [-Werror=implicit-function-declaration]
             ConnectWriteEDmac(data->dmaChannel, data->dmaSourceConn);
             ^
    lv_rec.c:375:9: error: implicit declaration of function 'PackMem_SetEDmacForMemorySuite' [-Werror=implicit-function-declaration]
             err = PackMem_SetEDmacForMemorySuite(data->dmaChannel, data->chunkData.memSuite, data->dmaFlags);
             ^
    lv_rec.c:376:9: error: implicit declaration of function 'PackMem_StartEDmac' [-Werror=implicit-function-declaration]
             PackMem_StartEDmac(data->dmaChannel, 0);
             ^
    lv_rec.c:389:13: error: implicit declaration of function 'PackMem_PopEDmacForMemorySuite' [-Werror=implicit-function-declaration]
                 PackMem_PopEDmacForMemorySuite(data->dmaChannel);
                 ^
    lv_rec.c:401:9: error: implicit declaration of function 'ConnectReadEDmac' [-Werror=implicit-function-declaration]
             ConnectReadEDmac(data->dmaCopyChannel, data->dmaCopyConn);
             ^
    lv_rec.c: In function 'lv_rec_save_footer':
    lv_rec.c:497:5: error: implicit declaration of function 'fps_get_current_x1000' [-Werror=implicit-function-declaration]
         footer.sourceFpsx1000 = fps_get_current_x1000();
         ^
    lv_rec.c: In function 'lv_rec_start':
    lv_rec.c:608:9: error: implicit declaration of function 'PackMem_RegisterEDmacCompleteCBRForMemorySuite' [-Werror=implicit-function-declaration]
             PackMem_RegisterEDmacCompleteCBRForMemorySuite(data.dmaCopyChannel, &complete_cbr, 0);
             ^
    lv_rec.c:609:9: error: implicit declaration of function 'PackMem_RegisterEDmacPopCBRForMemorySuite' [-Werror=implicit-function-declaration]
             PackMem_RegisterEDmacPopCBRForMemorySuite(data.dmaCopyChannel, &pop_cbr, 0);
             ^
    lv_rec.c:770:9: error: implicit declaration of function 'DeleteMemorySuite' [-Werror=implicit-function-declaration]
             DeleteMemorySuite(data.memCopySuite);
             ^
    In file included from lv_rec.c:21:0:
    lv_rec.c: At top level:
    ../../src/module.h:8:39: error: redefinition of '__module_strings_lv_rec'
     #define MODULE_STRINGS_PREFIX         __module_strings_
                                           ^
    ../../src/module.h:216:82: note: in definition of macro 'MODULE_STRINGS_START__'
     #define MODULE_STRINGS_START__(prefix,modname)                  module_strpair_t prefix##modname[] MODULE_STRINGS_SECTION = {
                                                                                      ^
    ../../src/module.h:214:65: note: in expansion of macro 'MODULE_STRINGS_START_'
     #define MODULE_STRINGS_START()                                  MODULE_STRINGS_START_(MODULE_STRINGS_PREFIX,MODULE_NAME)
                                                                     ^
    ../../src/module.h:214:87: note: in expansion of macro 'MODULE_STRINGS_PREFIX'
     #define MODULE_STRINGS_START()                                  MODULE_STRINGS_START_(MODULE_STRINGS_PREFIX,MODULE_NAME)
                                                                                           ^
    lv_rec.c:788:1: note: in expansion of macro 'MODULE_STRINGS_START'
     MODULE_STRINGS_START()
     ^
    ../../src/module.h:8:39: note: previous definition of '__module_strings_lv_rec' was here
     #define MODULE_STRINGS_PREFIX         __module_strings_
                                           ^
    ../../src/module.h:216:82: note: in definition of macro 'MODULE_STRINGS_START__'
     #define MODULE_STRINGS_START__(prefix,modname)                  module_strpair_t prefix##modname[] MODULE_STRINGS_SECTION = {
                                                                                      ^
    ../../src/module.h:214:65: note: in expansion of macro 'MODULE_STRINGS_START_'
     #define MODULE_STRINGS_START()                                  MODULE_STRINGS_START_(MODULE_STRINGS_PREFIX,MODULE_NAME)
                                                                     ^
    ../../src/module.h:214:87: note: in expansion of macro 'MODULE_STRINGS_PREFIX'
     #define MODULE_STRINGS_START()                                  MODULE_STRINGS_START_(MODULE_STRINGS_PREFIX,MODULE_NAME)
                                                                                           ^
    ./module_strings.h:23:1: note: in expansion of macro 'MODULE_STRINGS_START'
     MODULE_STRINGS_START()
     ^
    cc1: some warnings being treated as errors
    make[3]: *** [lv_rec.o] Error 1



[a1ex]

Yeah, that's why I said you will need a codebase right before raw_rec was created.

[Audionut]

Oh, I didn't consider that, as I assumed that was directly related to the uncompressed 422 discussion.  Which it probably was........  Nevermind 

After getting all of the dependencies in order, I produced this (for anyone else interested).
edit:  removed

I haven't had a chance to test it yet.

[a1ex]

May be a good idea to rename it to something like raw2dng-interlace.exe, so people won't use it by mistake on normal footage.

[Audionut]

Doesn't appear to be working.

Processing a normal ISO file.

https://dl.dropboxusercontent.com/u/34113196/ML/interlaced/000000.dng

And a dual ISO file.

https://dl.dropboxusercontent.com/u/34113196/ML/interlaced/000001.dng

edit:  Wait, should I be processing files from the old lv_rec?

[a1ex]

The first DNG looks OK to me: RG lines were replaced with 0, so the picture only has green and blue components.

Zoom at 100% in ufraw, for example, with AHD demosaicing. Or, process the image with "dcraw -4 -E -T 000000.dng" and then zoom in.

The VNG 4-color algorithm seems to remove these lines very well.

[Audionut]

In that case, I'll link raw2dng again, for someone else to look at.

https://dl.dropboxusercontent.com/u/34113196/ML/Builds/raw2dng-interlaced.exe


I was expecting it to skip two lines, and produce half resolution images.  I leave it for otherman or someone else to produce results from here (for now).

[a1ex]

See the comments for what to modify for two lines.

[otherman]

Thank you, guys

[Audionut]

I have created a 21 frame sequence using the raw2dng build above.

https://www.dropbox.com/sh/e9jyoryn77xog2i/AACE0eEK75xNJpOFnCeRlrN-a


Separatefields() will remove the 0's, leaving only valid RG or GB data.

[otherman]

Sorry for the absence, preatty busy weeks! I'm going to continue my tests