Author Topic: uncompressed 14-bit RAW video recording  (Read 755591 times)

Michael Zöller

  • Administrator
  • Senior
  • *****
  • Posts: 304
  • 5D Mark II
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #150 on: May 07, 2013, 09:33:56 AM »
I've moved the posts regarding compilation and TCC to a new thread.
neoluxx.de
EOS 5D Mark II | EOS 600D | EF 24-70mm f/2.8 | Tascam DR-40

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #151 on: May 07, 2013, 11:18:41 AM »
I made a quick comparison (with my 6D)  test in AE using linear wipe: 1080p h.264 IPB encoder VBR bitrate 10, qp 6 24fps (bitrate about 80 mbit/s) vs 1080p yuv422 24fps processed with 422toimg -single file yuv to jpeg sequence - quality 100%.
IMHO there is almost no visible difference, maybe very very small improvement only visible on 400% crop. Did you guys make such comparisons?
www.pilotmovies.pl   5D Mark III, 6D, 550D

3pointedit

  • Senior
  • ****
  • Posts: 263
  • 550D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #152 on: May 07, 2013, 12:51:29 PM »
A good way to detect artefacts is to mix between identical shots with a subtraction function. This way the only image data left will be the difference.
550D on ML-roids

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3149
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #153 on: May 07, 2013, 12:59:22 PM »
thanks for your comparison.
i think JPEG might be a bad choice when you want to compare image quality, even with factor set to 100 :)
i am no video guy, but maybe you should compare fine details with high contrast. like grass or trees.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #154 on: May 07, 2013, 03:45:54 PM »
Low QPs are supposed to be close to "lossless".. but I see a difference in the 4:2:0 to 4:2:2 right away. I got MZ sync'd on 600D for 24P/30P and even works when 720P is overwritten to 24 frames... I still get shifted frames tho :(

Weird... I think burst + modules use the same EVF sync... when I tried MZ sync for evf sync it produced bad burst pics.. but normally it does not... so wtf is happening to the YUV frames if burst frames are coming out perfect. Almost 160 YUV frames on 720P today... I'm hoping when raw recording is decodable it won't be doing this. I think frame sizes are similar so the yield should be too.


Ok, double weird.... if you look at the sequences with moving subjects... after 21 frames the corruption starts, first at the bottom. Then it appears after a few frames the sequence is repeating. i.e I see the same truck that passed me in a good frame in a corrupt frame further down the line. WTF, indeed.

Issue remains, sraw trick or not.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3149
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #155 on: May 07, 2013, 04:51:42 PM »
huh?
when recording with lv_rec you get corrupted frames?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #156 on: May 07, 2013, 05:06:14 PM »
Yea, on 600D. Just like the guy posted. After 21 frames starts to look like mis-aligned old silent pic and repeats images. 6D I haven't noticed any issues yet. Someone was complaining about something similar with 5dII, maybe affects digic IV only?

minimimi

  • Freshman
  • **
  • Posts: 68
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #157 on: May 07, 2013, 05:16:29 PM »
memory copy test results(without memSuite)
I'm checked it but chunk copy is fast . It's misteirous.
and, I don't know uint32_t timer counter is how many cycled.So I don't know the result is correct or not.


The code is here, and these codes located at  run_test()
Code: [Select]
    void *m1 = malloc(20*1000*1000);
    void *m2 = malloc(20*1000*1000);
    void *sm1 = shoot_malloc(20*1000*1000);
    void *sm2 = shoot_malloc(20*1000*1000);
    void *ss1 = shoot_malloc_suite(20*1000*1000);
    void *ss2 = shoot_malloc_suite(20*1000*1000);

    uint32_t t1 = tskmon_get_timer_reg();
    uint32_t t2,tmp;
    int i,off;

    console_printf("malloc-ed 20M memcpy\n");
    memcpy(m1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc-ed 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(m1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc-ed 20M dma_memcpy\n");
    dma_memcpy(m1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc-ed 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(m1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M memcpy\n");
    memcpy(sm1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(sm1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M dma_memcpy\n");
    dma_memcpy(sm1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc-ed 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(sm1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc to shoot_malloc-ed 20M memcpy\n");
    memcpy(sm1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc shoot_malloc-ed 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(sm1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M memcpy\n");
    memcpy(m1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            memcpy(m1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            t2 = ~tmp - t1 + t2;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);





    t1 = tskmon_get_timer_reg();
    console_printf("malloc to shoot_malloc-ed 20M dma_memcpy\n");
    dma_memcpy(sm1,m2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("malloc shoot_malloc-ed 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(sm1+off,m2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M dma_memcpy\n");
    dma_memcpy(m1,sm2,20*1000*1000);
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);

    t1 = tskmon_get_timer_reg();
    console_printf("shoot_malloc to malloc 20M dma_memcpy 2m each\n");
    off=0;
    for(i=0;i<20;i++)
        {
            dma_memcpy(m1+off,sm2+off,2*1000*1000);
            off = off + 2*1000*1000;
        }
    t2 = tskmon_get_timer_reg();
    if(t2<t1)
        {
            tmp= 0;
            tmp = ~tmp - t1;
            t2 = t2 + tmp;
        }
    else
        {
            t2 = t1 -t2;
        }
    console_printf("%d\n",t2);



And result is:  (printf has no %u so I used %d(signed))
Code: [Select]
malloc-ed 20M memcpy
-485806
malloc-ed 20M memcpy 2m each
-71147
malloc-ed 20M dma_memcpy
-272364
malloc-ed 20M dma_memcpy 2m each
-524067
shoot_malloc-ed 20M memcpy
-484381
shoot_malloc-ed 20M memcpy 2m each
-81456
shoot_malloc-ed 20M dma_memcpy
-768859
shoot_malloc-ed 20M dma_memcpy 2m each
-527604
malloc to shoot_malloc-ed 20M memcpy
-561515
malloc shoot_malloc-ed 20M memcpy 2m each
-76210
shoot_malloc to malloc 20M memcpy
-494179
shoot_malloc to malloc 20M memcpy 2m each
-77837
malloc to shoot_malloc-ed 20M dma_memcpy
-266705
malloc shoot_malloc-ed 20M dma_memcpy 2m each
-513931
shoot_malloc to malloc 20M dma_memcpy
-268545
shoot_malloc to malloc 20M dma_memcpy 2m each
-537939

Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1857
  • Have you tried turning it off and on again?
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #158 on: May 07, 2013, 06:50:24 PM »
Sorry for the dumb question. I have no problem working with DNGs but I haven't a clue how to convert YUV files. I downloaded 422ToImage but can't get it to work. Are there any tuts?
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

AlexVakulov

  • New to the forum
  • *
  • Posts: 3
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #159 on: May 07, 2013, 06:58:12 PM »
Sorry for the dumb question. I have no problem working with DNGs but I haven't a clue how to convert YUV files. I downloaded 422ToImage but can't get it to work. Are there any tuts?
Hello! Change your file extension from YUV to 422 and use the same program.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #160 on: May 07, 2013, 07:07:49 PM »
This is what is happening to 600D sequences:

http://imgur.com/a/fNUme#0

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12258
  • Emergencies only
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #161 on: May 07, 2013, 07:12:45 PM »
I've noticed the same distortion on 5D3 with raw mode (480p at 12fps). First frames are pretty much OK, then it loses sync.

mindogas

  • New to the forum
  • *
  • Posts: 21
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #162 on: May 07, 2013, 07:16:33 PM »
Hello! Change your file extension from YUV to 422 and use the same program.
Bad answer. *.YUV extension is for YUV422 videos and *.422 for still pictures. *.YUV files contains some extra information about its frames so  this extra data is like trash for .422 conversion. converting *.YUV files is quite easy but you must know that this is experimental feature. ML and 422ToImage sources changes everyday so you should use the last versions then you experimenting with YUV videos.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3149
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #163 on: May 07, 2013, 07:46:52 PM »
hmm, interesting.
made a few videos in .422 video mode and got no distortion at all.
this looks like there is some data lost in the beginning of frame number 21.

will look into it tonight.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

deleted.account

  • Member
  • ***
  • Posts: 154
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #164 on: May 07, 2013, 08:57:45 PM »
Bad answer. *.YUV extension is for YUV422 videos and *.422 for still pictures. *.YUV files contains some extra information about its frames so  this extra data is like trash for .422 conversion. converting *.YUV files is quite easy but you must know that this is experimental feature. ML and 422ToImage sources changes everyday so you should use the last versions then you experimenting with YUV videos.

Is it really a bad answer? I rename the .422 files to .yuv, import into avisynth, rescale chroma as the source yuv is JFIF chroma over full 8bit range and encode to h264 without creating any intermediate image sequences at all. :-)

We already know images sizes for .422 and can specify them in Avisynth so no big deal.

Its raw 8bit yuv, whether we call it image or video, whether we name it .422 or yuv.

Its down to how the host application handles it?

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #165 on: May 07, 2013, 09:47:25 PM »
Haven't seen any LV_rec issues on 6D yet... but I can't play back the raw :( (yea i know, bitch bitch bitch ) :)

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #166 on: May 07, 2013, 10:14:57 PM »
Sorry for the dumb question. I have no problem working with DNGs but I haven't a clue how to convert YUV files. I downloaded 422ToImage but can't get it to work. Are there any tuts?

Do you have the newest 1.9 version? it handles single movie files *.yuv very well, i tried today and had no problem with converting yuv to jpeg.
ver 1.8 didn't have such a feature
www.pilotmovies.pl   5D Mark III, 6D, 550D

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3149
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #167 on: May 07, 2013, 10:31:07 PM »
weird. tried now with 1700 frames again and no problem.
maybe its due to a slow card?
i have a lexar 1000x. do you have a slower one?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

mindogas

  • New to the forum
  • *
  • Posts: 21
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #168 on: May 07, 2013, 10:31:46 PM »
Is it really a bad answer? I rename the .422 files to .yuv, import into avisynth, rescale chroma as the source yuv is JFIF chroma over full 8bit range and encode to h264 without creating any intermediate image sequences at all. :-)

We already know images sizes for .422 and can specify them in Avisynth so no big deal.

Its raw 8bit yuv, whether we call it image or video, whether we name it .422 or yuv.

Its down to how the host application handles it?
Your post should start from last line i think. The thing is that question was about 422toimage software and for it the answer is clear - renaming files extension messing things up.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #169 on: May 07, 2013, 10:36:10 PM »
6D, no issues SD card
600D, same make/model sd = issues.

theoretically raw on 6D should be really out of sync if its bad on 5d3 with "slower" card.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3149
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #170 on: May 07, 2013, 11:07:05 PM »
about the EDMAC performance:
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

deleted.account

  • Member
  • ***
  • Posts: 154
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #171 on: May 07, 2013, 11:16:43 PM »
Your post should start from last line i think. The thing is that question was about 422toimage software and for it the answer is clear - renaming files extension messing things up.

Of coarse you're correct as the author of the application. It 'messes things up' due to the way 422toimage handles the source, my apologies.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D

shuggyrasklat

  • New to the forum
  • *
  • Posts: 38
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #173 on: May 08, 2013, 05:26:18 AM »
why cant we get this for the mkii?!?!?

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3149
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #174 on: May 08, 2013, 07:50:28 AM »
Oh come on. How often did we explain that?
Let us finish the stuff so it is working at least on one model before we port it to the other models.
....
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!