Compressed Raw mlv_rec_lj92

Started by reddeercity, April 15, 2017, 07:17:46 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

reddeercity

Just for kicks I thought I would try to update source with compressed_raw branch & mlv_rec_lj92 branch
and see if I could get compressed raw video , and it work for the most part (didn't crash  :D )
I recorded 4 mlv's file @ 1856x1044 and it didn't seem to be less data rate (70MB/s)
so when I check them with, mlvfs I go dng with right metadata but at 64K , yea I know there's a
special mlv_dump and run them thought there and it stop at
[ERROR] VIDF: File ends in the middle of a block
Processed 0 video frames


This is what the output was with mlv_dump -v

C:\raw_compressed\New1>mlv_dump -v --dng m14-2100.mlv

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'm14-2100.mlv'
   - Verbose messages
   - Enforcing 14bpp for DNG output
   - Using output path 'm14-2100_' for DNGs
   - Convert to DNG frames
   - Output into 'm14-2100_'
File m14-2100.mlv opened
File m14-2100.m00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 3450200333653317349
    FPS         : 23.976000
    File        : 0 / 0
    Frames Video: 364
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 183.912000 ms
    Res:  1856x1044
    raw_info:
      api_version      0x0001A95C
      height           1610612755
      width            -8295772
      pitch            309868
      frame_size       0xFFB2D048
      bits_per_pixel   268501006
      black_level      1791
      white_level      15000
      active_area.y1   18
      active_area.x1   160
      active_area.y2   1268
      active_area.x2   2040
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: INFO
  Offset: 0x000000e8
    Size: 16
    Time: 185.967000 ms
Block: RTCI
  Offset: 0x000000f8
    Size: 44
    Time: 0.001000 ms
     Date:        14.04.2017
     Time:        21:00:09 (GMT+0)
     Zone:        ''
     Day of week: 32
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000124
    Size: 40
    Time: 0.002000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    21348 microseconds (1/46.84)
Block: LENS
  Offset: 0x0000014c
    Size: 96
    Time: 0.003000 ms
     Name:        'EF24-70mm f/2.8L USM'
     Serial:      ''
     Focal Len:   70 mm
     Focus Dist:  107 mm
     Aperture:    f/3.20
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x000000E6
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 0.004000 ms
     Camera Name:   'Canon EOS 5D Mark II'
     Camera Serial: '4EB9D103'
     Camera Model:  0x80000218
Block: WBAL
  Offset: 0x00000200
    Size: 44
    Time: 0.005000 ms
     Mode:   9
     Kelvin:   4800
     Gain R:   461
     Gain G:   1024
     Gain B:   590
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 0.006000 ms
     picStyle:   33
     contrast:   -4
     sharpness:  0
     saturation: -2
     colortone:  0
Block: RTCI
  Offset: 0x00000260
    Size: 48
    Time: 391.506000 ms
     Date:        14.04.2017
     Time:        21:00:09 (GMT+0)
     Zone:        ''
     Day of week: 32
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000290
    Size: 48
    Time: 393.020000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    21348 microseconds (1/46.84)
Block: LENS
  Offset: 0x000002c0
    Size: 96
    Time: 393.052000 ms
     Name:        'EF24-70mm f/2.8L USM'
     Serial:      ''
     Focal Len:   70 mm
     Focus Dist:  107 mm
     Aperture:    f/3.20
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x000000E6
     Flags:       0x00000000
Block: WBAL
  Offset: 0x00000320
    Size: 48
    Time: 393.067000 ms
     Mode:   9
     Kelvin:   4800
     Gain R:   461
     Gain G:   1024
     Gain B:   590
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x00000350
    Size: 3394768
    Time: 428.784000 ms
   Frame: #0000
    Crop: 160x128
     Pan: 160x121
   Space: 3824
[ERROR] VIDF: File ends in the middle of a block
Processed 0 video frames
Done


link to my dropbox with 9 dng's from mlvfs and a 450MB .mlv file
https://www.dropbox.com/sh/ckv102q37wkd3ff/AABW-9FOgnIUouWeLJr969kIa?dl=0
https://www.dropbox.com/s/ulqojk8uyn1wr5o/M14-2103.MLV?dl=0

This was just a quick test to see what  would happen , I didn't spend any time with the code just updated & merged

Edit: FYI : Magic Lantern Nightly.2017Apr15.5D2212 (7a9b6805756c (mlv_rec_lj92))
# Built on 2017-04-15 02:48:39 UTC by ml@ml-pc

edit 2: seems to be able to be extracted on raw2cdng version 1.7.9

a1ex

Click ;)

...


      api_version      0x0001A95C
      height           1610612755
      width            -8295772
      pitch            309868
      frame_size       0xFFB2D048
      bits_per_pixel   268501006


5D2 is much harder to unbrick (not joking). This can be a sign of memory corruption - you are playing with fire.

The LJ92 branch will not compress files while recording - the codec is implemented only in mlv_dump. Only the compressed_raw branch (and the crop_rec_4k branch, which includes compressed_raw) will compress on the camera, and only with mlv_lite.

reddeercity

@A1ex --Will I think I might have soft brick my 5d2  :-[ , after I played around with lj92 compression It seemed that there no Raw Video buffer to access .
It just displays 0x0 for the raw video resolution instead of e.g. 1856x1044 etc..  it does works find on normal nightly builds and even works with 10-12bit
experimental builds that I compile . Problem is with any of my custom image buffers I was testing in my 3k/UHD thread , in fact just before I tried this compressed raw build
I could get  3584x1068 but after trying my compressed raw build I could no longer . But like I said I still can use nightly's without issue . I guess I need to fix this now if I what to continue developing the 3k/uhd thread.
Where would I start ? I do have my 2 Rom's backup from before I started developing , It seems I may have some corruption in the ram is there a way to verify this ?
I the mean time , I do some searching on the forum for any info on this . Thanks for any advice .



a1ex

Quote from: reddeercity on June 25, 2017, 08:42:22 AM
It just displays 0x0 for the raw video resolution instead of e.g. 1856x1044 etc..  it does works find on normal nightly builds and even works with 10-12bit experimental builds that I compile.

That means, for some reason, raw_update_params() failed. There are some consistency checks done, in particular on the optical black area - so if the image has e.g. some lines of garbage at the bottom, these tests will fail.

To skip these checks, I just hardcode the black level value temporarily (in autodetect_black_level, write some reasonable values for both black_level and black_stdev_x100, and comment out the autodetection).

I don't think it has anything to do with ROM contents. More likely, if the raw buffer does not update completely (read: the EDMAC transfers fewer bytes than declared raw buffer size), the last lines from the buffer will contain whatever it happened to be in memory when allocating it (and that's non-deterministic). The above patch will let you preview incomplete buffers (both in raw_diag and in the raw recorder's "grayscale" preview).

Regarding bricking issues, currently 5D2 can be emulated in QEMU, so if it happens, should be recoverable (unless you erase the bootloader somehow). A nice research project would be emulating the flash ROM routines, then jumping to random addresses (only in QEMU, never on real camera!!!) and checking what would happen (whether you can reflash parts of the ROM by mistake, and then, how to prevent these actions). See this PR, but it's really just scratching the surface.

reddeercity

Thanks A1ex I'll do as you suggested

reddeercity

I'll continue in this the  3K/UHD 5D2 Raw development and Other Digic IV Cams  so other can follow my progress/mistakes .