Topics

Hey guys I have strange behavior (channel gain or channel black/white level changes in time, within some range and then stops) when recording in 3K crop_rec mode (5D3 firmware 1.1.3). Here is the sample MLV link.

Reproduce like this:

Use crop rec with raw recording (I used 2880 width with 2.35 aspect for continuous recording). Record small clip, check, all must be fine. Then use 10x magnification to focus on something. Switch back to 1x mode (ML slow preview) and record again about a 30 seconds.

@a1ex: I guess this footage can not be recovered fully right?

Hi guys,

Last two weeks I've been investigating lossles JPEG (92) specs, LJ92Lib from Andrew Baldwin and MLV files produced by both a1ex's lovely crop_rec_4k/mlv_lite and by mlv_dump from lj92 branch, also DNGs produced by MLRAWviewer.

It seems that Andrew Baldwin's implementation is somewhat weird. It encodes raw data as two images each with its own headers, one scan and one component in it. That means that encoded data has two SOI (0xFFD8, Start of image) and EOI (0xFFD9, End of image) markers and each image has its own SOF3 (0xFFC3, Start of frame header), DHT (0xFFC4, Huffman table) and SOS (0xFFDA, Start of scan header) with its own one image component with full width and half height. Also baldand chose the prediction method 6 in his encoder because of reasons explained here. If someone interested the original document is here.

Meanwile the raw data encoded (as configured by a1ex) by Canon HW compressor fully follows the specs and has one SOI/EOI, which accordingly have DHT, SOF3 and SOS in it. SOS consists of two components with full width and half height. Predictor used is number 1.

This is the reason why LJ92lib can not decode ML files out of the box despite it supports all 7 predictors while decoding (not only number 6). The component number should be correctly taken into account.

While studying all these I came across Syoyo Fujita's github account with his modified header file based on baldand's LJ92lib. ( which also was pointed out by @martinherring )

In short Syoyo correctly modded the main loop in parseScan function to support components and changed decoder struct and added 'components' field to it. Also he added multiple Huffmann table support to the lib which is really unnecessary at least for ML compressed data. So I edited lj92.c accordingly. Unfortunatelly encoder still needs some work. However for MLVFS encoder is unnecessary

Using all this info I pathched MLVFS and it works nicely with all its bells and whistles (on the fly raw processing). I want to thank Danne as usual for testing/support and MAC binary.

David Milligan merged changes to MLVFS repo.
Download latest MAC binary here and Windows x64 binary here.
The latest 'DokanSetup_redist.exe' installer is here.

I also modified mlv_dump-on-steroids but unfortunatelly it's not ready for prime time yet. Needs more time. Always time . Now supports all latest stuff plus some features that none of the mlv_dump versions have. Binaries uploaded.

Note: mlv_dump-on-steroids merged in ML repository to crop_rec_4K branch.

regards
bb

Hi there,

First of all I want to admit: this is not the first time when someone modified mlv_dump. I also did it once not so long while ago but this time it's little bit different. This one based on latest vanilla MLVFS source files edited and changed by me.

Also, I can say that digging into MLVFS sources is a real joy . I love it starting from the idea and ending with the very cool multiplatform implementation. I want to thank David Milligan for this gem, special thanks to Danne (he always supports my development attempts), also a1ex and g3gg0 for all this thing called ML project moving forward and all active maintainers and devs of other ports and all the community around the project. Thank you guys!

Now about the subject:

UPDATE:
UPDATE2:
UPDATE3 (2018):
UPDATE4 (last!): It is merged to official ML's crop_rec_4k branch: Link

( hopefully ) it:
1. Saves 16bit any bit depth Cinema DNG files. e.g. 10/12/14 bits compressed or uncompressed, "--no-bitpack" exports 16 bit uncompressed DNGs.
2. Supports any bit depth raw data as input compressed or uncompressed
3. Has additional or reworked switch options:

Code Select Expand


---
  --no-audio          for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV
---
  --no-fixfp          do not fix focus pixels
  --fixcp2            use aggressive method for revealing more bad pixels
  --no-stripes        do not fix vertical stripes in highlights
  --force-stripes     compute stripe correction for every frame
  --is-dualiso        use dual iso compatible horizontal interpolation of focus and bad pixels
  --is-croprec        generate focus map for crop_rec mode
  --save-bpm          save bad pixels to .BPM file
  --fixpn             fix pattern noise
  --deflicker=value   per-frame exposure compensation. value is target median in raw units ex: 3072 (default)
  --no-bitpack        write DNG files with unpacked to 16 bit raw data
  --show-progress     show DNG file creation progress. ignored when -v or --batch is specified
                      also works when compressing MLV to MLV and shows compression ratio for each frame
  --fpi <method>      focus pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 1
  --bpi <method>      bad pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 0

---
  -c                  compress video frames using LJ92. if input is lossless, then decompress and recompress again.
  -d                  decompress compressed video and audio frames using LZMA or LJ92
  -p                  pass through original raw data without processing, it works for lossless or uncompressed raw

"--force-stripes" forces stripe detection for each frame separately. Slow but more effective on some footage
"-p" option replaces "-c -c" and passes through unmodified data for both uncompressed or lossless compressed raw.
"--no-audio" for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV, it's forced when "-f" used
"--no-fixfp" option deactivates fixing of focus pixels"
"--is-croprec" option  generates focus map for crop_rec mode"

4. Supports focus pixels the same way MLVFS does, also has ability to override and load '.fpm' manually if file has the same name as input MLV. Automatically fixes focus pixels for losless or uncompressed MLVs without special maps. If could not fix some dots for some special case map file can be specified as usual to override auto fixing. Auto detects crop_rec mode
5. Saves/Loads '.bpm' (bad pixel map) files. Name of the file same as input MLV
6. Can fix pattern noise
7. Has deflicker option of MLVFS
8. --show-progress prints every operation's done while saving DNGs, also works when compressing MLV to MLV and shows compression ratio for each frame
9. Implemented advanced WAV header with iXML for demanding NLEs
10. Includes 'crop_rec_4k' branch goodies/changes
11. Correctly fills up 'Unique Camera Model' tag in dng header and has additional camera matrixes
12. With "-b" switch you can convert bit depth and related black/white levels and export to that bit depth DNG (uncompressed only)
13. Chroma smooth code updated to the more advanced one
14. Added pixel interpolation method from raw2dng
15. Introduced two command line switches: "--fpi" and "--bpi" (see usage), by default, for focus pixels RAW2DNG method (correct edge pixel processing added) and for bad pixels MLVFS method is used
16. Supports new focus pixel maps w/header, generated by fpmutil (in addition to original simple .fpm), 'fpmutil' binaries can be downloaded here
17. Well... maybe I forgot something to mention

'fpmutil' also updated and can be downloaded from the links above as usual

Any feedback is welcome!

Here are links to the fork and downloads.
Here is a branch with simplified changeset to understand, with less effort, how 'mlv_dump.c' was modified and what has changed.

Now it's merged to Magic Lantern repository: Link

The best regards
bb

[Danne]

So what is that all about:

I've modified raw2dng code and added the fully compatible (hopefully ) MLV output with all required info blocks and metadata.

Code Select Expand


usage:

raw2dng file.raw [prefix|--mlv [sidecar]]

  prefix    will create prefix000000.dng, prefix0000001.dng and so on.
   --mlv    will output MLV with unprocessed raw data and the same name as input.
sidecar    if needed specify (prerecorded or any) MLV file to override meaningless
            metadata values in IDNT, EXPO, LENS and WBAL blocks


It fully supports multiple file chunks in DNG and MLV mode. Produced MLV files tested and working with mlv_dump, mlvfs, MlRawViewer. Compiled for Linux and Win32. OSX binaries kindly provided by Danne.

I needed this tool and I've done it for myself, but if our _master_ devs or anyone feel like this is useful addition it's a honor to contribute at least something to the great ML project community.

Download latest source and binaries or compile it from ML source (it's merged to Unified branch).



Update:

Here is a small tool which automatically sets proper frameCount value to MLV file header (development raw_rec, raw v1.1 mlv lite case where it's zero not any more, patched).

Code Select Expand


usage:

mlv_setframes file.mlv [--set]

   --set    if specified actually writes frameCount to file
            otherwise just outputs the information

   Extra testing option:
   --set0x00000000    sets zero frameCount to any mlv file



Download: Source, Binaries

The binary looks for proper MLV/MXX file not by extension but a content of a file, makes sure the file has to be changed and only after that alters the value if additionally --set option specified.

You can safely run it on any folder with any mixed files in it like this: "for file in *.*; do ./mlv_setframes "$file"; done".
However *.M* wildcard is better performance wise.

If --set is not specified it changes nothing - just outputs a few info about processed files. With --set0x00000000 you can go back to original state.
Note: It does not alter file modification time. Which I guess essential at least for me.



Update 2:

Here is a mlv_dump version from Dmilligan's magic-lantern repo. He did substitute original dng handling (chdk-dng.c) with his own library (dng.c) which is a base of mlvfs dng handling code. dng.c from latest mlvfs is quite different from the version of this branch, so I merged some sweet code from latest file and it seems worked out ok.

What we got after that:
1. Very similar to MLVFS dng output except it's 14bit and there are no forward matrixes in the header.
2. Fully works (hopefully): white balance, timecode, default scale.
3. Some fixes to mlv_dump to work properly including crash when using chroma smooth option.

I want to thank David for his great work on dng.c and mlvfs in general and Danne for his ideas, pointing out to this great reworked version of mlv_dump and helping out to test the program during development process. As well as all ML leading devs here and all the community.

Latest source is here.



Update 3:

There is more advanced and up to date version called "mlv_dump on steroids"



Update 4:

Now "mlv_dump on steroids" merged to official ML's crop_rec_4k branch Link


Regards
bb

Hi,

I have some questions about the subject:
1. The histo is a little bit different for LV and Quick Review mode right after the shot. Difference is about 0.4-0.7 stop. QR mode always shows brighter. For example: LV E 0.2 before shot - QR E -0.5 after shot. Is it normal for the current implementation of the raw histogram? 0.7 stop is not a small diff btw. I can assure that the lite condition is not changing during exposure.

2. As I understand RAW histo math heavily depends on Canon's ExpoSim mode?! (I have no enough experience in coding to examine the sources for the algo). Then why on my 5D3 raw histo operates even with ExpoSim mode off (and shoes wrong value) and only shows "ExpoSim off" icon when this mode set to "when DOF button pressed".

3. In some dark but contrasty situation (quite dark wall with some quite bright highlights on it) ETTR looses control e.g. does not show messages "whoops" or "exposure limits reached" and shows ETTR hint for example E 1.5 while severe RGB clips are present like R30 G34 B28.

4. I've been experimenting a little and looking at raw histo with lens cap on (max darkness). Aperture did not make any sense only shutter and iso. I thought that raw algo in this situation mesures just a noise level but I guess I was wrong. When:
a) shutter 1/8000 and iso 100 ettr hint shows E 9.2
b) shutter 1/8000 and iso 25600 ettr hint shows E 11.9 (higher ISOs than this make no sense)
I don't understand those values while knowing that dynamic range is larger and noise lower for iso 100 (11.5 stop) than for iso 25600 (7 or something) with significantly more noise.

TIA
bb

Hello everybody

Please someone explain to me why the ML CF card benchmark results are so different in photo (so called playback) mode and video mode?

photo mode (not in live view)


video mode


Regards
bb