Magic Lantern Forum

General Discussion => General Chat => Topic started by: reddeercity on December 11, 2016, 07:27:31 AM

Title: FFmpeg-Play with MLV Raw Support
Post by: reddeercity on December 11, 2016, 07:27:31 AM
Always searching for faster and better ways to work with MLV raw for fast turn around when time is limited .
I know this has been talk about before back a few years ago that ffmpeg started to support magic lantern raw video.
With this in mind I downloaded the latest version for windows (http://ffmpeg.org/download.html#build-windows) and in the package there a app called "ffmpeg play"
so I just drop a .mlv file on top of ffmpeg play and it open up a command window & a viewer and starts playing
the files with audio in sync as far as I can tell . There a issue with black level decoding (Green Cast) be on that
it a fast why of checking the Video Stream just like mlrawviewer but ffmpeg play seems to support bit reduction raw video (10-12)
I tested 14bit & 10bit there seem to play the same on my SSD . Not Sure what there are using to decode maybe DCraw?
If the right black levels could be obtained this would be very handy indeed .

(https://c7.staticflickr.com/1/525/31194175830_d64cd7d95a.jpg) (https://flic.kr/p/PwwiFL)
FFmpeg Play With MLV+Audio (https://flic.kr/p/PwwiFL) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://c1.staticflickr.com/1/512/31194175880_24ef9b4e85.jpg) (https://flic.kr/p/PwwiGC)
FFmpeg Play Command Window (https://flic.kr/p/PwwiGC) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr

(https://c5.staticflickr.com/1/365/31194175900_a051346829.jpg) (https://flic.kr/p/PwwiGY)
FFmpeg Play With MLV+Audio_ScreenShot (https://flic.kr/p/PwwiGY) by RedDeerCityTV (https://www.flickr.com/photos/67942440@N06/), on Flickr
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: Danne on December 11, 2016, 07:55:10 AM
Greenish cast seems not so easy to fix. There,s code out there you could find and maybe even post about it on the ffmpeg forum.
The faster workaround seems to use mlvfs and open up dng sequences in mlrawviewer.
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: reddeercity on December 11, 2016, 08:16:32 AM
Quote from: Danne on December 11, 2016, 07:55:10 AM
Greenish cast seems not so easy to fix. There,s code out there you could find and maybe even post about it on the ffmpeg forum.
The faster workaround seems to use mlvfs and open up dng sequences in mlrawviewer.
that's 2 steps   :)  mlrawviewer does not support 10-12 mlv raw unless you use mlvfs , I could do the same with
MLVProductor . Just looking to check files with audio & for corruption .
Just looking to expand my magic lantern raw video tool set because you never know , it my come in handy e.g.
view files on set with director ( yes I know I could use the player in the camera which now is pretty good near real time)
I like to have many options  ;D 
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: Danne on December 11, 2016, 08:23:49 AM
There are settings for different levels and maybe even piping could achieve a non green output. I might take a look at a piping command. Me and dfort was sitting with this a year ago, I even think he tried to get some attention at ffmpeg forum back then.
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: reddeercity on December 11, 2016, 08:55:57 AM
I down loaded the source code and found the decoder mlvdec.c file.
From what I can tell mlv file is being extracted to 16bit file and looks like it's reading black & white level
form the rggb , cfa pattern ? not sure

            avio_skip(pb, 8 + 16 + 24); // black_level, white_level, xywh, active_area, exposure_bias
            if (avio_rl32(pb) != 0x2010100) /* RGGB */
                avpriv_request_sample(avctx, "cfa_pattern");
            avio_skip(pb, 80); // calibration_illuminant1, color_matrix1, dynamic_range
            vst->codecpar->format    = AV_PIX_FMT_BAYER_RGGB16LE;
            vst->codecpar->codec_tag = MKTAG('B', 'I', 'T', 16);
            size -= 164;
        } else if (ast && type == MKTAG('W', 'A', 'V', 'I') && size >= 16) {
            ret = ff_get_wav_header(avctx, pb, ast->codecpar, 16, 0);
            if (ret < 0)
                return ret;
            size -= 16;
        } else if (type == MKTAG('I','N','F','O')) {
            if (size > 0)
                read_string(avctx, pb, "info", size);
            continue;
        } else if (type == MKTAG('I','D','N','T') && size >= 36) {
            read_string(avctx, pb, "cameraName", 32);
            read_uint32(avctx, pb, "cameraModel", "0x%"PRIx32);
            size -= 36;
            if (size >= 32) {
                read_string(avctx, pb, "cameraSerial", 32);
                size -= 32;
            }
        } else if (type == MKTAG('L','E','N','S') && size >= 48) {
            read_uint16(avctx, pb, "focalLength", "%i");
            read_uint16(avctx, pb, "focalDist", "%i");
            read_uint16(avctx, pb, "aperture", "%i");
            read_uint8(avctx, pb, "stabilizerMode", "%i");
            read_uint8(avctx, pb, "autofocusMode", "%i");
            read_uint32(avctx, pb, "flags", "0x%"PRIx32);
            read_uint32(avctx, pb, "lensID", "%"PRIi32);
            read_string(avctx, pb, "lensName", 32);
            size -= 48;
            if (size >= 32) {
                read_string(avctx, pb, "lensSerial", 32);
                size -= 32;


Source code for mlv decoder "mlvdec.c" file in my dropbox link
https://www.dropbox.com/s/xw1bu5gggnzz4wm/mlvdec.c?dl=0

Edit: even reads the canon picture style

} else if (type == MKTAG('S','T','Y','L') && size >= 36) {
            read_uint32(avctx, pb, "picStyleId", "%"PRIi32);
            read_uint32(avctx, pb, "contrast", "%"PRIi32);
            read_uint32(avctx, pb, "sharpness", "%"PRIi32);
            read_uint32(avctx, pb, "saturation", "%"PRIi32);
            read_uint32(avctx, pb, "colortone", "%"PRIi32);
            read_string(avctx, pb, "picStyleName", 16);
            size -= 36;


Metadata from ffmpeg play
Input #0, mlv, from 'C:\Users\Toshiba i7\Downloads\ffmpeg-20161210-edb4f5d-win64
-static\ffmpeg-20161210-edb4f5d-win64-static\bin\M09-1750-10bit-crop.MLV':
  Metadata:
    guid            : 0xd4ab7eb25b0dcb6b
    picStyleName    : Landscape
    shutterValue    : 20549
    wbs_ba          : 0
    isoMode         : manual
    isoValue        : 800
    isoAnalog       : 96
    digitalGain     : 0
    focalLength     : 50
    focalDist       : 138
    aperture        : 400
    stabilizerMode  : 0
    autofocusMode   : 3
    flags           : 0
    lensID          : 169
    cameraName      : Canon EOS 5D Mark II
    cameraModel     : 2147484184
    cameraSerial    : #######
    lensName        : 50mm
    wb_mode         : 9
    kelvin          : 3700
    wbgain_r        : 461
    wbgain_g        : 1024
    wbgain_b        : 590
    wbs_gm          : 0
    picStyleId      : 131
    contrast        : 0
    sharpness       : 4
    saturation      : 0
    colortone       : 0
    time            : 2016-12-09 17:51:18
  Duration: 00:00:22.69, start: 0.000000, bitrate: 557724 kb/s
    Stream #0:0: Video: rawvideo (BIT[16] / 0x10544942), bayer_rggb16le, 2144x10
76, 23.98 tbr, 23.98 tbn, 23.98 tbc
    Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, 2 channels,
s16, 1536 kb/s

Title: Re: FFmpeg-Play with MLV Raw Support
Post by: Danne on December 11, 2016, 04:16:03 PM
Is FFmpeg/FFplay outputting the raw stream?
Stream #0:0: Video: rawvideo (BIT[16] / 0x10544942), bayer_rggb16le,
Can we apply color matrix or colorformat to this stream? Maybe piped to another ffmpeg/ffplay output?


Here,s a sort of a dead end but also some good info.
http://lists.ffmpeg.org/pipermail/ffmpeg-user/2014-June/022073.html
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: Danne on December 11, 2016, 06:39:27 PM
Here is a little quick test with a 3D correction lut created in DaVinci resolve. I just played with this manually, no math, could be much better. Command goes like this.

3D lut download
https://drive.google.com/file/d/0B4tCJMlOYfiranRiRHhJMTA2Sjg/view?usp=sharing

FFplay -vf lut3d=drag/3d/lut/here your_file.MLV

With 3D correction lut
(https://s23.postimg.org/ggh83mnx7/Screen_Shot_2016_12_11_at_18_32_34.png)

Without 3D corerction lut
(https://s23.postimg.org/4dbwg2cuz/Screen_Shot_2016_12_11_at_18_32_49.png)
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: dfort on December 11, 2016, 06:57:57 PM
Wow, interesting. When we looked at this a while back (FFMPEG now officially supports Magic Lantern Video (http://www.magiclantern.fm/forum/index.php?topic=11566.0)) it seemed that ffmpeg wasn't debayering MLV files properly. Something to do with the extra green pixels in the RGGB Bayer Pattern. Looks like you can "correct" it out.
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: reddeercity on December 11, 2016, 11:25:22 PM
Not to bad @Danne , could be a temporary work around . Look like you loss the highlight maybe fine tuning the lut a bit more ?

Quote from: dfort on December 11, 2016, 06:57:57 PM
it seemed that ffmpeg wasn't debayering MLV files properly. Something to do with the extra green pixels in the RGGB Bayer Pattern.

I being looking over the "bayer_template.c"  code and it seems the raw @16bit is being converted to RGB24 (8bit)
plus there seems to be 2 different Bayer & CFA pattern so maybe a issue?

RGGB  CFA pattern R=0 , G=1 , B=2 (0,1,2)
GBRG  CFA pattern R=2 , G=1 , B=0 (2,1,0)
Normally I think mlv convertors use RGGB if I'm not mistaken

It will take some time to get my head around all this , I believe the solution is with this code

/*
* Bayer-to-RGB/YV12 template
* Copyright (c) 2011-2014 Peter Ross <[email protected]>
*
* This file is part of FFmpeg.
*
* FFmpeg is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* FFmpeg is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with FFmpeg; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/

#if defined(BAYER_BGGR) || defined(BAYER_GBRG)
#define BAYER_R       0
#define BAYER_G       1
#define BAYER_B       2
#endif
#if defined(BAYER_RGGB) || defined(BAYER_GRBG)
#define BAYER_R       2
#define BAYER_G       1
#define BAYER_B       0
#endif

#if defined(BAYER_8)
#define BAYER_READ(x) (x)
#define BAYER_SIZEOF  1
#define BAYER_SHIFT   0
#endif
#if defined(BAYER_16LE)
#define BAYER_READ(x) AV_RL16(&(x))
#define BAYER_SIZEOF  2
#define BAYER_SHIFT   8
#endif
#if defined(BAYER_16BE)
#define BAYER_READ(x) AV_RB16(&(x))
#define BAYER_SIZEOF  2
#define BAYER_SHIFT   8
#endif

#define S(y, x) BAYER_READ(src[(y)*src_stride + BAYER_SIZEOF*(x)])
#define T(y, x) (unsigned int)S(y, x)
#define R(y, x) dst[(y)*dst_stride + (x)*3 + BAYER_R]
#define G(y, x) dst[(y)*dst_stride + (x)*3 + BAYER_G]
#define B(y, x) dst[(y)*dst_stride + (x)*3 + BAYER_B]

#if defined(BAYER_BGGR) || defined(BAYER_RGGB)
#define BAYER_TO_RGB24_COPY \
    R(0, 0) = \
    R(0, 1) = \
    R(1, 1) = \
    R(1, 0) = S(1, 1) >> BAYER_SHIFT; \
    \
    G(0, 1) = S(0, 1) >> BAYER_SHIFT; \
    G(0, 0) = \
    G(1, 1) = (T(0, 1) + T(1, 0)) >> (1 + BAYER_SHIFT); \
    G(1, 0) = S(1, 0) >> BAYER_SHIFT; \
    \
    B(1, 1) = \
    B(0, 0) = \
    B(0, 1) = \
    B(1, 0) = S(0, 0) >> BAYER_SHIFT;
#define BAYER_TO_RGB24_INTERPOLATE \
    R(0, 0) = (T(-1, -1) + T(-1,  1) + T(1, -1) + T(1, 1)) >> (2 + BAYER_SHIFT); \
    G(0, 0) = (T(-1,  0) + T( 0, -1) + T(0,  1) + T(1, 0)) >> (2 + BAYER_SHIFT); \
    B(0, 0) =  S(0, 0) >> BAYER_SHIFT; \
    \
    R(0, 1) = (T(-1, 1) + T(1, 1)) >> (1 + BAYER_SHIFT); \
    G(0, 1) =  S(0,  1) >> BAYER_SHIFT; \
    B(0, 1) = (T(0,  0) + T(0, 2)) >> (1 + BAYER_SHIFT); \
    \
    R(1, 0) = (T(1, -1) + T(1, 1)) >> (1 + BAYER_SHIFT); \
    G(1, 0) =  S(1,  0) >> BAYER_SHIFT; \
    B(1, 0) = (T(0,  0) + T(2, 0)) >> (1 + BAYER_SHIFT); \
    \
    R(1, 1) =  S(1, 1) >> BAYER_SHIFT; \
    G(1, 1) = (T(0, 1) + T(1, 0) + T(1, 2) + T(2, 1)) >> (2 + BAYER_SHIFT); \
    B(1, 1) = (T(0, 0) + T(0, 2) + T(2, 0) + T(2, 2)) >> (2 + BAYER_SHIFT);
#else
#define BAYER_TO_RGB24_COPY \
    R(0, 0) = \
    R(0, 1) = \
    R(1, 1) = \
    R(1, 0) = S(1, 0) >> BAYER_SHIFT; \
    \
    G(0, 0) = S(0, 0) >> BAYER_SHIFT; \
    G(1, 1) = S(1, 1) >> BAYER_SHIFT; \
    G(0, 1) = \
    G(1, 0) = (T(0, 0) + T(1, 1)) >> (1 + BAYER_SHIFT); \
    \
    B(1, 1) = \
    B(0, 0) = \
    B(0, 1) = \
    B(1, 0) = S(0, 1) >> BAYER_SHIFT;
#define BAYER_TO_RGB24_INTERPOLATE \
    R(0, 0) = (T(-1, 0) + T(1, 0)) >> (1 + BAYER_SHIFT); \
    G(0, 0) =  S(0, 0) >> BAYER_SHIFT; \
    B(0, 0) = (T(0, -1) + T(0, 1)) >> (1 + BAYER_SHIFT); \
    \
    R(0, 1) = (T(-1, 0) + T(-1, 2) + T(1, 0) + T(1, 2)) >> (2 + BAYER_SHIFT); \
    G(0, 1) = (T(-1, 1) + T(0,  0) + T(0, 2) + T(1, 1)) >> (2 + BAYER_SHIFT); \
    B(0, 1) =  S(0, 1) >> BAYER_SHIFT; \
    \
    R(1, 0) =  S(1, 0) >> BAYER_SHIFT; \
    G(1, 0) = (T(0, 0)  + T(1, -1) + T(1,  1) + T(2, 0)) >> (2 + BAYER_SHIFT); \
    B(1, 0) = (T(0, -1) + T(0,  1) + T(2, -1) + T(2, 1)) >> (2 + BAYER_SHIFT); \
    \
    R(1, 1) = (T(1, 0) + T(1, 2)) >> (1 + BAYER_SHIFT); \
    G(1, 1) =  S(1, 1) >> BAYER_SHIFT; \
    B(1, 1) = (T(0, 1) + T(2, 1)) >> (1 + BAYER_SHIFT);
#endif

/**
* invoke ff_rgb24toyv12 for 2x2 pixels
*/
#define rgb24toyv12_2x2(src, dstY, dstU, dstV, luma_stride, src_stride, rgb2yuv) \
    ff_rgb24toyv12(src, dstY, dstV, dstU, 2, 2, luma_stride, 0, src_stride, rgb2yuv)

static void BAYER_RENAME(rgb24_copy)(const uint8_t *src, int src_stride, uint8_t *dst, int dst_stride, int width)
{
    int i;
    for (i = 0 ; i < width; i+= 2) {
        BAYER_TO_RGB24_COPY
        src += 2 * BAYER_SIZEOF;
        dst += 6;
    }
}

static void BAYER_RENAME(rgb24_interpolate)(const uint8_t *src, int src_stride, uint8_t *dst, int dst_stride, int width)
{
    int i;

    BAYER_TO_RGB24_COPY
    src += 2 * BAYER_SIZEOF;
    dst += 6;

    for (i = 2 ; i < width - 2; i+= 2) {
        BAYER_TO_RGB24_INTERPOLATE
        src += 2 * BAYER_SIZEOF;
        dst += 6;
    }

    if (width > 2) {
        BAYER_TO_RGB24_COPY
    }
}

static void BAYER_RENAME(yv12_copy)(const uint8_t *src, int src_stride, uint8_t *dstY, uint8_t *dstU, uint8_t *dstV, int luma_stride, int width, int32_t *rgb2yuv)
{
    uint8_t dst[12];
    const int dst_stride = 6;
    int i;
    for (i = 0 ; i < width; i+= 2) {
        BAYER_TO_RGB24_COPY
        rgb24toyv12_2x2(dst, dstY, dstU, dstV, luma_stride, dst_stride, rgb2yuv);
        src  += 2 * BAYER_SIZEOF;
        dstY += 2;
        dstU++;
        dstV++;
    }
}

static void BAYER_RENAME(yv12_interpolate)(const uint8_t *src, int src_stride, uint8_t *dstY, uint8_t *dstU, uint8_t *dstV, int luma_stride, int width, int32_t *rgb2yuv)
{
    uint8_t dst[12];
    const int dst_stride = 6;
    int i;

    BAYER_TO_RGB24_COPY
    rgb24toyv12_2x2(dst, dstY, dstU, dstV, luma_stride, dst_stride, rgb2yuv);
    src  += 2 * BAYER_SIZEOF;
    dstY += 2;
    dstU++;
    dstV++;

    for (i = 2 ; i < width - 2; i+= 2) {
        BAYER_TO_RGB24_INTERPOLATE
        rgb24toyv12_2x2(dst, dstY, dstU, dstV, luma_stride, dst_stride, rgb2yuv);
        src  += 2 * BAYER_SIZEOF;
        dstY += 2;
        dstU++;
        dstV++;
    }

    if (width > 2) {
        BAYER_TO_RGB24_COPY
        rgb24toyv12_2x2(dst, dstY, dstU, dstV, luma_stride, dst_stride, rgb2yuv);
    }
}

#undef S
#undef T
#undef R
#undef G
#undef B
#undef BAYER_TO_RGB24_COPY
#undef BAYER_TO_RGB24_INTERPOLATE

#undef BAYER_RENAME

#undef BAYER_R
#undef BAYER_G
#undef BAYER_B
#undef BAYER_READ
#undef BAYER_SIZEOF
#undef BAYER_SHIFT

#if defined(BAYER_BGGR)
#undef BAYER_BGGR
#endif
#if defined(BAYER_RGGB)
#undef BAYER_RGGB
#endif
#if defined(BAYER_GBRG)
#undef BAYER_GBRG
#endif
#if defined(BAYER_GRBG)
#undef BAYER_GRBG
#endif
#if defined(BAYER_8)
#undef BAYER_8
#endif
#if defined(BAYER_16LE)
#undef BAYER_16LE
#endif
#if defined(BAYER_16BE)
#undef BAYER_16BE
#endif

Title: Re: FFmpeg-Play with MLV Raw Support
Post by: Danne on December 12, 2016, 02:30:57 PM
Yes, could be the place for a fix. Since it,s overwashed with green I,d start checking green numbers. I tried swapping som defined numbers and there was a shift towards magenta. What all numbers do in code is hard to say for me. Not sure where to proceed from here. Maybe start nagging the moderators on ffmpeg forum ;).
Title: Re: FFmpeg-Play with MLV Raw Support
Post by: Danne on December 12, 2016, 11:06:00 PM
I can also recommend https://mpv.io/
This player also runs MLV files and is also open source.