Hi there,
First of all I want to admit: this is not the first time when someone modified mlv_dump. I also did it once not so long while ago but this time it's little bit different. This one based on latest vanilla MLVFS source files edited and changed by me.
Also, I can say that digging into MLVFS sources is a real joy :D. I love it starting from the idea and ending with the very cool multiplatform implementation. I want to thank David Milligan for this gem, special thanks to Danne (he always supports my development attempts), also a1ex and g3gg0 for all this thing called ML project moving forward and all active maintainers and devs of other ports and all the community around the project. Thank you guys!
Now about the subject:
UPDATE:
UPDATE2:
UPDATE3 (2018):
UPDATE4 (last!): It is merged to official ML's crop_rec_4k branch: Link (https://bitbucket.org/hudson/magic-lantern/src/0075d0fa70f229ec972d7831aa756c4a9280e625/modules/mlv_rec/?at=crop_rec_4k)
( hopefully ;) ) it:
1. Saves 16bit any bit depth Cinema DNG files. e.g. 10/12/14 bits compressed or uncompressed, "--no-bitpack" exports 16 bit uncompressed DNGs.
2. Supports any bit depth raw data as input compressed or uncompressed
3. Has additional or reworked switch options:
---
--no-audio for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV
---
--no-fixfp do not fix focus pixels
--fixcp2 use aggressive method for revealing more bad pixels
--no-stripes do not fix vertical stripes in highlights
--force-stripes compute stripe correction for every frame
--is-dualiso use dual iso compatible horizontal interpolation of focus and bad pixels
--is-croprec generate focus map for crop_rec mode
--save-bpm save bad pixels to .BPM file
--fixpn fix pattern noise
--deflicker=value per-frame exposure compensation. value is target median in raw units ex: 3072 (default)
--no-bitpack write DNG files with unpacked to 16 bit raw data
--show-progress show DNG file creation progress. ignored when -v or --batch is specified
also works when compressing MLV to MLV and shows compression ratio for each frame
--fpi <method> focus pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 1
--bpi <method> bad pixel interpolation method: 0 (mlvfs), 1 (raw2dng), default is 0
---
-c compress video frames using LJ92. if input is lossless, then decompress and recompress again.
-d decompress compressed video and audio frames using LZMA or LJ92
-p pass through original raw data without processing, it works for lossless or uncompressed raw
"--force-stripes" forces stripe detection for each frame separately. Slow but more effective on some footage
"-p" option replaces "-c -c" and passes through unmodified data for both uncompressed or lossless compressed raw.
"--no-audio" for DNG output WAV not saved, for MLV output WAVI/AUDF blocks are not included in destination MLV, it's forced when "-f" used
"--no-fixfp" option deactivates fixing of focus pixels"
"--is-croprec" option generates focus map for crop_rec mode"
4. Supports focus pixels the same way MLVFS does, also has ability to override and load '.fpm' manually if file has the same name as input MLV. Automatically fixes focus pixels for losless or uncompressed MLVs without special maps. If could not fix some dots for some special case map file can be specified as usual to override auto fixing. Auto detects crop_rec mode
5. Saves/Loads '.bpm' (bad pixel map) files. Name of the file same as input MLV
6. Can fix pattern noise
7. Has deflicker option of MLVFS
8. --show-progress prints every operation's done while saving DNGs, also works when compressing MLV to MLV and shows compression ratio for each frame
9. Implemented advanced WAV header with iXML for demanding NLEs
10. Includes 'crop_rec_4k' branch goodies/changes
11. Correctly fills up 'Unique Camera Model' tag in dng header and has additional camera matrixes
12. With "-b" switch you can convert bit depth and related black/white levels and export to that bit depth DNG (uncompressed only)
13. Chroma smooth code updated to the more advanced one
14. Added pixel interpolation method from raw2dng
15. Introduced two command line switches: "--fpi" and "--bpi" (see usage), by default, for focus pixels RAW2DNG method (correct edge pixel processing added) and for bad pixels MLVFS method is used
16. Supports new focus pixel maps w/header, generated by fpmutil (https://bitbucket.org/bouncyball/mlv-tools/overview) (in addition to original simple .fpm), 'fpmutil' binaries can be downloaded here (https://bitbucket.org/bouncyball/mlv-tools/downloads/)
17. Well... maybe I forgot something to mention
'fpmutil' also updated and can be downloaded from the links above as usual
Any feedback is welcome!
Here are links to the fork and downloads (https://bitbucket.org/bouncyball/magic-lantern/downloads/).
Here is a branch with simplified changeset to understand, with less effort, how 'mlv_dump.c' was modified and what has changed.
Now it's merged to Magic Lantern repository: Link (https://bitbucket.org/hudson/magic-lantern/src/0075d0fa70f229ec972d7831aa756c4a9280e625/modules/mlv_rec/?at=crop_rec_4k)
The best regards
bb
Only option I'm missing, since 10/12 bit raw recording ;D, is the option for different bit depths when creating dng's.
I know it's crazy, but I keep the dng's as backup and throw away the MLV files :P
So an option for 10, 12 or 14 bit dng output, according to my recording bit depth in the original MLV, would be killer future 8).
On steroids I say. It,s beautiful rework and addons. Don,t forget to mention white balance support as well as darkframe and flatframe inclusion. I,ll check deeper into this once I have some time. Here is a binary for mac meanwhile.
Great work.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_macOS
@Levas:
Yeah I understand, but the whole purpose of using mlvfs universal implementation is to convert any bit depth sources to 16 bit and then do raw processing on 16 bit buffers (downconverting again is a bad idea). Maybe some day experimental hardware raw compression from silent pictures goes to the mlv_rec/mlv_lite who knows.
@Danne:
Hey! Thanx for the binary. Uploaded.
Hello, bouncyball. It's a incredibly great.
Why you don't a "Hero member" yet? Yes, that message counters just nonsense...
Ok, on topic: Ironically, I just finished to transcode a couple of TB with old gold mlv_dump.
Actually, due to this issue
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173295#msg173295
i used 2 versions of mlv_dump: one for the dual iso shots, and another(which corrupts dual iso) for the rest -
because that one handles vertical stripes better, to my eye.
As far as i know, that modification finally was removed:
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173394#msg173394
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
and i had some persisted stripes on panned shots with MLVFS.
What is your approach here?
Best regards, and really great thanks for your contribution.
@Levas: I keep DNG too. I compress it with Adobe DNG converter.
Virtually there is no difference in size between 14 and 16 bit versions after compression.
And, i guess, 10 and 12 bit will have no benefit in size after compression.
Quote from: mothaibaphoto on February 17, 2017, 06:38:56 AM
So, the first question - what version of vertical stripes fix code do you use?
Is it possible to have both via swich or something like that?
And, second: there was a discussion in MLVFS thread, that MLVFS does not analyse every frame for vertical stripes for performance reason,
Well, It's exact MLVFS stripe removal code not from raw2dng however I guess technically they are quite similar.
What do you mean by "possible to have both via swich"?
The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.
Quote from: bouncyball on February 17, 2017, 07:57:52 AM
What do you mean by "possible to have both via swich"?
I mean with this:
https://bitbucket.org/hudson/magic-lantern/commits/ab24965b3f4d45e4cba10214bc0061f36b440730
and without
Quote from: bouncyball on February 17, 2017, 07:57:52 AM
The stripes analyzed on zero frame or 1st frame of a range, they could be analyzed on every frame though. Can be implemented via switch.
It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.
Ok understood. MLVFS doe not include that fix from a1ex. Also, I did not port the dualiso stuff from mlvfs anyway.
Will think about --stripes2 switch.
Tested some clips with fpm files and it works great just like in mlvfs. Tried the load .bpm file but it doesn,t seem to come through. The bpm files loads but the pixels aren,t patched. In your older .map code which I,m using now the dots are gone but the same coordinates don,t apply here. Maybe the zeroes at the end are disturbing?
My test files here: https://drive.google.com/file/d/0B4tCJMlOYfira2hKTEhIdmwzZ3c/view?usp=sharing
Note that this file won,t work with mlvfs fpm files since it,s a 700D crop_rec test file(experimental build from dfort). It,s the same as for eosm so it works with map files and it also loads the .bpm file in your steroid version but focus pixels are still present.
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: '/Users/Daniel/Desktop/test/700D_crop_rec_test.MLV'
- Convert to DNG frames
- Output into '/Users/Daniel/Desktop/test/700D_crop_rec_test_'
File /Users/Daniel/Desktop/test/700D_crop_rec_test.MLV opened
File /Users/Daniel/Desktop/test/700D_crop_rec_test.M00 not existing.
Processing...
Using bad pixel map: '/Users/Daniel/Desktop/test/700D_crop_rec_test.bpm'
39060 pixels loaded
Vertical stripes correction: 'UNNEEDED'
1.00000 1.00000 1.00014 1.00079 0.99968 0.99924 0.99899 1.00012
Extracting frames...
Current frame : '/Users/Daniel/Desktop/test/700D_crop_rec_test_000001.dng'
Reached end of chunk 1/1 after 9 blocks
Processed 2 video frames
Done
@mothaibaphoto
Thanks for the tip for using Adobe DNG converter, will try it out.
@Danne: I'll take a look
Sorry for offtopic, but:
@Levas:
Keep in mind that after DNG compressor your DNG are not CDNG anymore. This means you loose FPS tag at the very least and may have some problems with your post software.
No problem, using RawTherapee for editing DNG's, export as TIFF sequence and use these in DaVinci Resolve 8)
Quote from: mothaibaphoto on February 17, 2017, 08:36:03 AM
It really matters for some shots.
As long as performance not so important when pre-converting, it would be very useful to have this switch.
Had no time until today. Implemented --force-stripes switch. Please report the result.
For mac
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_steroids_1.0
Cool, but I have no footage to test on now :(
Hi guys,
I updated downloads on the first post. Latest binaries (linux/mac/win) with lot of internal changes and bug fixes.
I hope someone's using it, really need some beta tester's feedback :P
bb
Thanks for this @bouncyball and just downloaded. Will test and report back my findings after trying it out with a DF Avg process test. 8)
*edit*
Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?
hmh, maybe i missed the point.
but why develop a separate branch that grows too far from the main tree?
@DeafEyeJedi: Thanks for your help :)
Quote from: DeafEyeJedi on March 06, 2017, 08:52:02 PM
Actually probably due to my idiotic moment but I notice the mlv_dump (for Mac OS) file I downloaded isn't a binary -- did I miss a step in this?
Hmm... strange. The binary compiled/linked and running well under my virtual Sierra. I just stripped off the debug info by strip command to make it smaller. Did you rename it to mlv_dump and make it executable?
@g3gg0: Do you mean I gotta make pull request? :D
Quote from: g3gg0 on March 06, 2017, 11:13:02 PM
but why develop a separate branch that grows too far from the main tree?
The point is that there were a lot of things to do (remove very mlvs specific code, add my functions and make some optimizations/bugfixes, still want to add some more stuff) and at that moment I hesitated to do PR. Right now I think it's possible.
I've started it as a fork of unified, then included goodies from 10/12 branch, then just merged it to that branch and it's there now. It even merges to the unified cleanly at least the way as 10/12bit branch does.
regards
bb
@deafeyejedi
chmod u=rwx mlv_dump
Enter
@Danne: Yup. Right :)
Quote from: Danne on March 07, 2017, 07:54:07 AM
@deafeyejedi
chmod u=rwx mlv_dump
Enter
That did it. Thanks!
Hi,
In Windows 7 Service Pack 1 64-bit, program mlv_dump not working.
mlv_dump crash (https://drive.google.com/open?id=0B5jIPDo4rZ60OTB3VFFhTHBpTDA)
Problem signature:
Problem Event Name: APPCRASH
Application Name: mlv_dump.exe
Application Version: 0.0.0.0
Timestamp applications: 589fb28f
The name of the module with an error: ntdll.dll
The module version with the error: 6.1.7601.23455
The timestamp of the module with an error: 573a5463
Exception code: c0000005
Moving the exception: 00012c22
OS Version: 6.1.7601.2.1.0.256.1
Locale ID 1045
Additional information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
Read our online privacy statement:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0415
If the principle of confidentiality of information online is not available, please read our privacy statement offline:
C: \ Windows \ system32 \ en-US \ erofflps.txt
Raw Video (https://drive.google.com/drive/folders/0B5jIPDo4rZ60ZHpGNEd6bVN0TWM?usp=sharing)
Is there mlv_dump_steroids_1.0 version for Windows?
Yay! Feedback at last! :)
Well in reality this is the problem of your .IDX xref file. It's somehow wrong or corrupted (file size is wrong and block_size is wrong after XREF header). In all versions of mlv_dump it's gonna dump the core (will crash).
Just delete this .IDX file and if you really need it, build one with the command 'mlv_dump -x your.mlv' and you'll see the difference in sizes: Your's 5980 bytes vs Generated 4504 bytes.
Regards
bb
I just finished a cr2hdr_steroids.app(MAC) build for testing which utilize the steroid version of mlv_dump. Download here
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg
Source code for this version is inside the dmg.
It works the same as cr2hdr.app you start out with a main menu and from there you head over to the mlv_dump settings menu by selecting (m).
https://bitbucket.org/Dannephoto/cr2hdr/overview
Note that you reach darkframe and flatframe automation from the main menu, not from mlv_dump menu.
Also updated to the latest focus pixel(dfort) script with the -n switch
https://bitbucket.org/daniel_fort/ml-focus-pixels
Main menu
(https://s29.postimg.org/egr7nupmv/Screen_Shot_2017_03_07_at_15_20_52.png)
mlv_dump menu
(https://s29.postimg.org/pf2h61e87/Screen_Shot_2017_03_07_at_15_20_29.png)
Hey man! Great news, thank you :)
Uploaded a new cr2hdr_steroids version. Some small fixes around darkframe automation.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/cr2hdr_steroids.dmg
All tests below with eosm in crop_rec
Tested darkframe and flatframe handling and confirmed working. Like with g3gg0/a1ex 10/12 bit mlv_dump darkframe subtraction works good with 14 and 12 bit files but with 10bit not so very nice. Instead of subtracting color noise it seems to add to it a little with 10bit. Good to know if recording 10bit footage.
A not so very nice problem is later chroma smooth(dualiso code) vs raw2dng chroma smooth code. As cs 2x2 is great with raw2dng code and for cameras in need of it like eosm the cs code in dualiso code can,t really cope with focus pixels. Have a look.
Also about focus pixels it seems not all of them are taken care of. Check to the right on the picture. Maybe the -n switch needs some checking into @dfort?
Chroma smooth artifacts
(https://s10.postimg.org/t20rp2yw9/Screen_Shot_2017_03_07_at_19_10_48.png)
Focus pixels
(https://s3.postimg.org/8wjxkshb7/Screen_Shot_2017_03_07_at_19_14_50.png)
Darkframe testing. Last two picture flatframe averaging
14bit original
(https://s3.postimg.org/6sncwc403/01_14bit.jpg)
14bit Darkframed
(https://s3.postimg.org/pdbl3epfn/02_14bit.jpg)
12bit original
(https://s3.postimg.org/qyquv7znn/01_12bit.jpg)
12bit Darkframed
(https://s3.postimg.org/z9co31v7n/02_12bit.jpg)
10bit original
(https://s3.postimg.org/p5jct5p37/01_10bit.jpg)
10bit Darkframed
(https://s3.postimg.org/9padwm9tv/02_10bit.jpg)
Original (not flatframed)
(https://s3.postimg.org/ij1adpwsj/01_flat.jpg)
Flatframed
(https://s3.postimg.org/dphj8v0ar/02_flat.jpg)
Loving the update so far @Danne and I really dig the new mlv_dump menu within your app. It's so funny that 10-bit looks worse after DF avg process. Seems 12-bit may be the 'sweet spot' after all. Thanks for sharing and will test thoroughly before I report anything.
Yes. CS issue exacly as in mlvfs. I guess the solution is: http://www.magiclantern.fm/forum/index.php?topic=7122.msg179319#msg179319 (http://www.magiclantern.fm/forum/index.php?topic=7122.msg179319#msg179319).
Think I got something working with the -n swith in the focus pixel list.
This:
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
if [[ "$cropXY" == no ]]; then
((x = $i))
((y = $j))
echo -e "$x \t $y" >> "$output"
else
((x = $i - $cropX))
((y = $j - $cropY))
if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
if [[ "$format" == fpm ]]; then
echo -e "$x \t $y" >> "$output"
else echo -e "$x \t $y \t 0" >> "$output"
fi
fi
fi
fi
fi
done
}
instead of:
##
# output_row() function
# Write a row of focus pixel coordinates to the output file.
#
output_row() {
for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
if [[ "$cropXY" == no ]]; then
((x = $i))
((y = $j))
else
((x = $i - $cropX))
((y = $j - $cropY))
fi
if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
if [[ "$format" == fpm ]]; then
echo -e "$x \t $y" >> "$output"
else echo -e "$x \t $y \t 0" >> "$output"
fi
fi
fi
fi
done
}
Nicely done! :)
output_row() {
for i in $(seq 72 $((raw_width)) ); do # 0-71 is out of bounds
if (( (($i + $shift1)) % x_rep == 0 )) || (( (($i + $shift2)) % x_rep == 0 )); then
if [[ "$cropXY" == no ]]; then
((x = $i))
((y = $j))
+ if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
+ if [[ "$format" == fpm ]]; then
+ echo -e "$x \t $y" >> "$output"
+ else.
+ echo -e "$x \t $y \t 0" >> "$output"
+ fi
+ fi
else
((x = $i - $cropX))
((y = $j - $cropY))
if (( "$x" >= 0 )) && (( "$y" >= 0 )); then
if (( "$x" < "$width" )) && (( "$y" < "$height" )); then
if [[ "$format" == fpm ]]; then
echo -e "$x \t $y" >> "$output"
else.
echo -e "$x \t $y \t 0" >> "$output"
fi
fi
fi
fi
fi
done
}
Just a small correction. Now this does not break dcraw style output (zero at the end of line) and makes sure there're no negative coordinates (x,y >= 0).
Loving the collaborations from you two! [emoji56]
That extra dcraw else function isn,t really working. I think the if [[ "$format" == fpm ]] Needs an actual fpm file to be fed to the script else it won,t recognize outputting fpm and it will take it for a dcraw output anyway. At least that,s what my tests shown.
I think like this. The -n switch will probably only be used for the mlvfs focus pixel code in your steroid version and otherwise we run the script without the -n switch and then we,re back to the extra 0.
*Unless I specify the -f for fpm file, huh, just noticed that little detail. Will test again.
Ok, your little extras are working since I specified the -f switch as well. Then I tested the old focus pixel code with the -f switch but that din,t work, go figure. I will add your extra lines to dforts script. Great.
Updated cr2hdr_steroids.app with the latest fpm.sh script.
Yup that -f is there from the beginning :)
-f -n does what I really was after when adding -n.
bb
can we partially merge the goodies from that branch into the main branch?
i am afraid that this will end in both branches to develop in their own context and finally get un-merge-able.
Hi g3gg0 thanx for asking.
Quote from: g3gg0 on June 14, 2017, 03:21:03 PM
can we partially merge the goodies from that branch into the main branch?
Sure we can talk about it more deeply. I always wanted to do this but my version is quite different (different from dmilligans version as well) and I kinda thought to ask you and a1ex to just add it as a separate branch (I agree: not the best idea). Actually, to the "mlv_dump.c" itself, I always tried to make changes as subtle as possible. I think I succeeded more or less and all big changes are only regarding DNG handling and WAV advanced header. All processing of the RAW data before saving DNG is taking place outside of the mlv_dump.c.
Quote from: g3gg0 on June 14, 2017, 03:21:03 PM
i am afraid that this will end in both branches to develop in their own context and finally get un-merge-able.
I did some important commits last two days (now it supports any bit depth compressed/uncompressed DNG output) and did not test merge-ability yet. However, usually I'm very carefull about it and always merge any new changes you make.
BR,
BB
Thanks for latest changes. Being able to output native bit 10/12/16 with mlv_dump stripes, chroma smooth etc added is magic. Also working with compression with all bits after final strip byte fix to match dng checksum.
Also looking forward to some sort of merge story involving mlv_dump unified as g3gg0 points out. How that would be done idk but where's there competence there's possibilities. Always been missing white balance and some of the matrices.
Insights around the strip byte tag would perhaps make it possible to calculate the correct strip byte tag for when exporting compressed 10bit/12bit dng files with crop_rec_4k build now?
Great work Bouncyball.
well, its already such a huuuuuuge changeset that it is almost impossible to review and understand.
if it was asked to merge, i would have no clue what (side) effects merging would bring with.
also not sure if i want to have e.g. deflicker in mlv_dump.
(don't get me wrong, it's for sure a good piece of software, but mlv_dump wasn't meant as post processing tool.
it fixes all the very camera specific issues and is made for a solid mlv-to-other-tools converter)
just having a PR for this branch is very unlikely to get merged.
you would have to split the changes up.
Indeed. more mlvfs like approach. And hey, I only scratch my head in amazement when I look into code in mlv_dump. Good inspiration.
Hi g3gg0,
I really understand your point. Huge changeset is not gonna get merged in any case and that is why I hesitated to make PR. Even 1st changes were big enough (but review-able) because of "chdk_dng.c" been replaced by "dng.c" to support CDNG and more...
I think you are familiar with dmilligan's dng lib more or less (actually I modified it but the idea is the same) and if you accept the fact that in order to merge goodies from this version chdk_dng.c should be replaced, then I will try my best to split changes and start merging them to crop_rec_4k branch.
Quote from: g3gg0 on June 17, 2017, 11:40:31 PM
also not sure if i want to have e.g. deflicker in mlv_dump.
Well :) it's the smallest and least important part of changes and takes place outside of mlv_dump.c. Can be removed anytime.
regards
bb
Hey guys!
Finally I updated the binaries to the latest version. Lots of changes are made from last time. Look at the 1st post of this thread. Download links are the same.
And... any feedback is welcome as usual!
br,
bb
Hey Bouncyball
I am interested in trying out --force-stripes. I am having issues with Vertical Stripes in a lot of my footage.
Could you post an example how I would write the command ? I am not sure where to put in --force-stripes.
@Kharak
Command like this:
mlv_dump --dng --force-stripes --show-progress INPUT.MLV
should do the trick.
Also have a look at flat field method as g3gg0 mentioned here (http://www.magiclantern.fm/forum/index.php?topic=19300.msg186057#msg186057)
@ BB
Thank you, sorry for the late reply. I have been so busy.
The --force-stripes does a very good job. But it is very slow as you said.
I just tried converting a compressed mlv without --force-stripes.
mlv_dump --dng --show-progress INPUT.MLV
And it said "Vertical stripes correction: 'NEEDED'" and then proceeded to show the numbers under it. Does this mean it did perform a "normal" Vertical Stripe Fix or that I need to do it. I tried without --force-stripes hoping I could achieve same results as Danne with his CR2HDR app; he got straight out of the box with an older Vertical Stripe fix code, identical results to the --force-stripes. I don't know if CR2HDR app with that (old) Vertical Stripe Code is a slow process. But I am wondering why this old stripe fix is working so well and I have to use (a seemingly new code?) --force-stripes to remove mine? Is it a Mac and Win thing? :)
I also tried the same MLV:
mlv_dump_s --dng --fixpn --show-progress input.MLV
It said "Vertical Stripe Correction: Unneded". Yet extremely slow, maybe slower than --force-stripes (did not take time). But half way through the conversion I could see I was just adding pattern noise, I was hoping maybe this --fixpn could attack the vertical stripes, i mean in reality Vertical Stripes are Pattern Noise, no?
And last but not least, is there any way to batch process with you mlv_dump or for some form of automation?
Thank you.
@Kharak
Quote from: Kharak on June 25, 2017, 03:07:55 AM
And it said "Vertical stripes correction: 'NEEDED'" and then proceeded to show the numbers under it. Does this mean it did perform a "normal" Vertical Stripe Fix or that I need to do it.
It's just an information, that after evaluating of the 1st frame there is a real need to correct stripes. No intervention required from user. Normal vertical stripes correction is done according to 1st frame analysis.
Quote from: Kharak on June 25, 2017, 03:07:55 AM
he got straight out of the box with an older Vertical Stripe fix code, identical results to the --force-stripes. But I am wondering why this old stripe fix is working so well and I have to use (a seemingly new code?) --force-stripes to remove mine? Is it a Mac and Win thing? :)
Good question :) (no mac or windows thing). This mlv_dump uses all the same raw processing algorithms what used in MLVFS but it does not mean these algos are better than some other ones used in other code. Not only stripe fix but also chroma smooth and bad/focus pixel interpolation is different and yields different results. Investigation of this subject is on my schedule, just had low priority until recently. Thanks for sharing experience with both stripes code. Are you certain when you say that older normal stripes removal gives same results as --force-stripes? Can you show some images illustrating this?
Quote from: Kharak on June 25, 2017, 03:07:55 AM
I was hoping maybe this --fixpn could attack the vertical stripes, i mean in reality Vertical Stripes are Pattern Noise, no?
Pattern noise removal method is not the same as stripes removal even if it helps to make stripes less visible.
Quote from: Kharak on June 25, 2017, 03:07:55 AM
And last but not least, is there any way to batch process with you mlv_dump or for some form of automation?
It's a command line utility. Of course it's intended to use for batch processing in shell out of the box. Under windows, unfortunatelly, there is no any user friendly, ready to use, automator like Danne's cr2hdr.app.
bb
I fired up windows under virtual box so been fiddling some windows commands today. Who knows where that ends :P
http://www.magiclantern.fm/forum/index.php?topic=10526.msg186219#msg186219
By the way. How does --force-stripes work? Is it calculating every dng file individually or only the first?
Hey Danne!
Hahaha welcome to winworld :). Well... there is a rudimentary support of bash in Win10 :P, very, VERY limited to use with full blown scripts. If you someday, somehow decide to port your great work you're gonna need to use powershell (can be even made GUI app with it).
Regarding "--force-stripes": it has force prefix because it means analyze every single frame and correct it. That's why it takes ages :)
bb
Gleaned on bash in windows 10. Will check out power shell definetaly :).
Seems there is need for a second vert stripe option, "always on" to be able to overide automation code and at the same time avoid forcing it on every dng.
Well... it "always on" when no stripe related switch is specified:
no switch - analyze 1st frame, process all frames according to gathered data (code decides and prints out whether correction 'NEEDED' or 'UNNEEDED')
--no-stripes - do not correct stripes at all
--force-stripes - analyze every frame, process current frame according to gathered data (code decides and prints out whether correction 'NEEDED' or 'UNNEEDED')
Did you mean something else? :)
Always on, sure about that? Kharak is reporting bad results when force stripes isn´t selected? I think he needs to put up some demo pics here...
Yes normal correction is on by default. Look at output with --show-progress. There was a serious bug in there, stripe correction was done only to 1st frame. Sorry about that... will upload new fixed binaries soon. Thanks to Danne for the hint :)
Sure :) I already asked him about it.
Fixed binaries are uploaded. Link is the same @ 1st post.
You still need examples?
I jumped to the new script and mlv_dump Danne posted here:
http://www.magiclantern.fm/forum/index.php?topic=10526.new#new
No thank you :)
Well. What would be good is to compare latest vertical stripes code in unified and crop_rec_4k versions of mlv_dump to a1ex older vertical stripes code. If there are a lot of footage that doesn´t get the stripe treatment we need to know. Could you check and compare Kharak and see what you come up with? You can simply do this test by exchanging mlv_dump in batch_mlv.
Well good idea. I don's have havily striped footage at the moment to test with.
@Kharak: If you do the test anyway test this mlv_dump as well. Also cut original mlv file(s) to have 1st 5-10 frames and put up them somewhere please.
Sure can do, but how do I cut the MLV's?
I used to use MLV_Diag from Chmee to cut the files, but it does not work with the lossless MLV's.
mlv_dump -f 5 -o OUTPUT.MLV INPUT.MLV
Beware output name must be different from input name or you end up with a 0 byte file.
okey, thank you.
I am uploading 3 Folders with DNG's from 3 separate mlv_dumps all containing 6 DNG's each, hope it enough. Bouncyball's mlv_dump, Danne's dump and latest Crop_rec dump (june 19th). All on the same mlv. The one I posted jpeg examples of in the crop_rec 4k thread. Good to have a standard.
@Danne,
I can not use your latest mlv_dump, it requires cygwin1.dll or tells me to "please reinstall the program". You mentioned something about win 10, I am on win 8.1 maybe that is why. So I used the previous version that I was using before.
This crazy connection. Download 4-5 kb/s and upload 130-150 kb/s. Thought I had to postpone these uploads 10 days from now.
I send you PM's when it is done.
Fixed the mlv_dump issue Kharak.
Can you post examples of the difference between vertical stripe code among the mlv_dump versions? A few close ups here would do the trick. Also link to your dng files. Would be great to also have the shortened MLV files. If we need to do some retests we could easliy recheck against the same files.
Yes I can, but I can not post the originals I sent you, publicly. I could send the DNG/mlv's to the relevant Devs, but that is not very open source.
So I have some other examples like this one I posted already in crop 4k thread once, much more visible stripes too. https://mega.nz/#!8YIHGIJB!hzewH2JBX55vELsy2MPdp3gPSo2mTzlNKe_CQIVqu_o
But maybe this particular issue has more to do with highlight stripes. The other examples I gave you were stripes in the shadows. anyways, yours and Bouncyballs mlv_dumps fixed these stripes aswell, so if you like, I can send this. I got a very shady connection at the moment, but I will try and upload dng's and mlv's.
I have to cut the mlv in the middle somewhere to get this example in a 6 frame MLV, some code? suggestions?
This example has no flaring the first few seconds and that made, according to a1ex, the crop_rec4k mlv_dump to not remove it. (yours do though).
You can cut anywhere in the MLV file thanks to g3gg0 indexing code. For example:
mlv_dump -f 25-30 -o output.MLV input.MLV
Only took 50 attempts! but here it is.
Experimental Crop_rec4k MLV_dump - DNG's (June 19th Build):
https://mega.nz/#F!pURzVaJQ!jGGhzZAWtHXRm-gX7LEfhQ
Danne's Batch MLV_dump - DNG's:
https://mega.nz/#F!hZREiJZK!2j1SfZGur31XZJ0p2gvsbA
Bouncyball's MLV_dump on Steroids - DNG's:
https://mega.nz/#F!9dZm3QSA!Q2XhjUCzWSWR5nE8NkEc4A
The cut MLV:
https://mega.nz/#!cF43DDqQ!z1J-HlaQscli88b0BeCzxfBRrV6HC6X-i1L1o88Kob0
I hope these can help. Let me know if the Scene should be different somehow. This example was my initial foré in to Vertical Stripe fix with the experimentals, but later I found out that I was having severe issues in the shadows aswell.
@Kharak: Thank you. Downloaded MLV.
Thanks K. Will check asap.
I am posting another example showing Vertical Stripes in the shadows, no matter what MLV_dump I use they are there. FYI: of the bat in this case I see no difference in the shadow stripes between MLV_dump from Bouncyball/Danne or A1ex's Experimental MLV_Dump (june 19th). I did not pull all of them together in to and editor to check differences. No matter, stripes are there.
On a static scene the Vertical Stripes are well hidden, but as soon as movement is going on, either camera movement or subject movement across shadows, they become very visible.
Look Camera Right, the shadow on his cheek and throat, MLV Example:
https://mega.nz/#!sAZCABKK!XHWF4aGHue6Uu1356gufLS7_7NALDx5D5ucQG3iZPJs
Set MLRawViewer Curve to Log-c add a punchy LUT of your choosing. Preferably one that just barely crushes blacks and raises highlights to almost blow out. Raise the EV to +1.51 (or there abouts) to emulate the Cinelog-C Workflow, where you increase the Signal Strength, this should highlight the stripes. And ofcourse play at 24 FPS.
These stripes are not just visible to the Pixel Peepers but for the "normal" audience too. And hiding the stripes in blacks is not a workable solution, way too much DR is lost to crush them out.
The problems persists after Noise Reduction as the algorithms "think" it's important information, atleast noise dithers the stripes somewhat, but I don't like 5D noise in those medium dark shadows, and stripes are still visible with or without NR. I took some screenshots with NR and full grade, but on the screenshots they disappear because no movement, I hope you can replicate, if not let me know.
When playing the converted DNG's in MLRawViewer and having Vertical Stripe fix ON, the Vertical Stripes that MLV_dump could not get rid of, disappear. At first it looks perfect! But at a closer look I see it introduces Horizontal stripes in its place instead, not as severe as the Vertical Stripes, but Horizontal Stripes are there. MLRawViewer will most of the time remove the Vertical Stripes that MLV_dump could not, but at total random will leave chunky Vertical Lines through the frame where any Black or high contrast source is in an image or these Horizontal small lines in medium to dark shadows.
Unfortunately I can't export Prores from MLRawViwer, that just stopped working for me many months ago.
Let me know if I can do something else.
http://www.magiclantern.fm/forum/index.php?topic=13152.msg170924#msg170924
This is called pattern noise (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/#patternnoise), and it's unrelated to what we call "vertical stripes". If a dark frame doesn't fix it, there is an experimental algorithm in MLVFS, although, for best results, you may have to adjust things directly in the source code.
Try exposing properly next time. Since the highlights are not critical in this case, you should have increased the exposure by 2 stops (which would make a big difference in shadows and midtones, but will also require some highlight recovery capabilities from your raw processor). To clip the skin tones (near the window), you would need +3.5 stops on top of your current exposure level, so there's plenty of headroom.
To recap:
- pattern noise in shadows -> use a dark frame
- vertical stripes in highlights (sharp, not noisy) -> use a flat frame
- both kind of defects -> use both calibration frames.
For exposure:
- clouds, sky, buildings, out of focus backgrounds: clipping 1 or 2 channels should be OK
- skin tones: do not clip anything
(unfortunately Auto ETTR doesn't have face detection)
For existing footage, you should be able to record the calibration frames afterwards.
fyi
Pattern noise setting is ported to mlv_dump for steroids.
Thanks for pointing out - will definitely experiment with this option as soon as my PC will be able to play videos again.
Cool.
And @Kharak. Also try out chroma smooth settings. 2x2 with darkframes works wonders on eosm for instance.
@Danne.
What are the dis/advantages of using Chroma Smoothing? I must have read something about it years ago, that it crushes details or something, because I've been avoiding it like the plague. Besides, if Chroma Smoothing is a fix for these issues and has no obvious caveats, shouldn't it be On as standard for every converter? For me, anything that destroys detail I avoid, especially on 5D3 because the perceived resolution is so small as is.
And how do I do FPN fix in mlv_dump? Do I need to make FPN Frames like Darkframes ?
@a1ex
Okey thank you, so its FPN. So now the silly questions. MLRawViewer manages to get rid of them with its vertical Stripe fix, but introduces artifacts like horizontal stripes and Chunky vertical stripes in black/dark spots and/or high contrast sources instead. Is it in anyway possible to combine the best of both worlds? It is my understanding that MLRawViewer uses OpenGL shaders with combination of different coding languages, ty @bouncyball, but it does a fine job at removing these vertical FPN except for generating other artifacts. Thoughts?
I find the exposure to be fine for my style, I like creamy soft highlight roll-offs and ETTR'ing too much usually ruins that, especially blowing out two channels, that is just not recoverable. This shot could have been +0.5 EV higher, maybe +1, that's a big maybe. But for my shooting style it's good enough. The shadows hold a lot of information underneath the noise, something Temporal NR is good at extracting. I just can't get rid of the FPN.
Quick grade: https://mega.nz/#!FVwXSSpI!4lyl105BGrLCU7fcPEJVxqoX1jVxVaGZpl5MzhU2gbY .jpeg
Or maybe I need to learn to Pull exposure better in post.
This FPN in the shadows was not a problem for me with this shooting style in Vanilla 24p 3x3 1080p. I think the sensor is pushed too much in 60p mode, introducing more FPN than normal. Just a hunch I have.
btw bit off topic, a line above in the link you posted:
QuoteThe read noise can be isolated by taking a "black frame" image, an exposure with the lens cap on and the highest available shutter speed; there are thus no photons captured, and only the electronic noise from reading the sensor remains.
Is this the actual proper way of doing a Darkframe ? Setting shutter speed to the highest available?
And yet again, everything points to Darkframe+Flatframe for the ultimate image. But for any kind of production that is not possible as is, on Windows atleast. Its just way too much manual work, decompressing, sorting ISO's, sorting lenses, reprossesing for Darkframe+Flatframe, to me it just doesn't seem feasible. Not the Flatframe thing atleast. I have some Vintage lenses that have quite big variations in the Vignetting at the different F-stops, from 1.4 up to 5.6 and they are all Manual. Here's crossing fingers for Danne's venture in to Windows Darkframe automation. ;)
As always, let me know what else I can help with.
Try a darkframe. Probably gonna work best. Chroma smooth 2x2 is fine in my world but it might not be needed.
When doing a darkframe film with exact same setting as the footage, lens cap on, shoot 2-3 seconds.
Creating a storage folder with darkframes for batch_mlv shouldn't be too hard. Just need some time.
You need to decompress your footage too before doing the mlv_dump averaging. If it's too many steps just upload a vanilla darkframe and I test it for you.
Quote from: Kharak on July 01, 2017, 03:16:29 AM
MLRawViewer manages to get rid of them with its vertical Stripe fix, but introduces artifacts like horizontal stripes and Chunky vertical stripes in black/dark spots and/or high contrast sources instead. Is it in anyway possible to combine the best of both worlds? It is my understanding that MLRawViewer uses OpenGL shaders with combination of different coding languages
Unfortunately, MLRawViewer just locks up my current machine (bringing down the entire operating system), so it's not straightforward for me to play with this code. Otherwise I'd probably maintain this program, to some extent.
Quote
This shot could have been +0.5 EV higher, maybe +1, that's a big maybe.
To see how it would have looked at +1 EV, do this:
exiftool M08-1044-cut_000000.dng -WhiteLevel -BlackLevel
White Level : 16200
Black Level : 2047
exiftool M08-1044-cut_000000.dng -WhiteLevel=9123
How did I come up with this number? (2047 + 16200) / 2.
Proof that taking the average of black and white is equivalent with increasing the exposure 1 stop (regarding overall look and highlight clipping point, but not noise levels) if left as an exercise for the reader.
Result...
octave:1> a = read_raw('M08-1044-cut_000000.dng');
octave:2> prctile(a(:), [1 5 10 50 90 99 99.9 99.99 100])'
ans =
2054.0 2066.0 2076.0 2176.0 3285.0 6054.0 8156.0 8828.6 9113.0
That means, not a single pixel would have been clipped in this frame by increasing the exposure by 1 stop (which means, no information would have been lost). You should be able to grade this file identically to the original (please try).
To simulate 2 stops, try white level 5585, and for 1.5 stops, 7050. Math is left as exercise.
octave:3> prctile(a(1:2:end,1:2:end)(:), [1 5 10 50 90 99 99.9 99.99 100])'
2050.0 2059.0 2067.0 2126.0 2767.0 4212.5 5019.0 5236.2 5364.0
That means, with 2 stops, no pixel on the red channel would have been clipped (therefore, no highlights should be clipped to pure white).
My JPEGs (0, +1, +1.5, +2 from white level, +3, +2, +1.5, +1 from exposure slider; for other settings, change extension to .ufraw):
(https://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M08-1044-cut_000000.jpg) (https://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M08-1044-cut_000000+1ev.jpg)
(https://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M08-1044-cut_000000+1.5ev.jpg) (https://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M08-1044-cut_000000+2ev.jpg)
The highlights in the last two (where some channels are clipped) are desaturated (they do not retain the original colors, but they are not clipped to white either). That's how highlight recovery works in ufraw. To my knowledge, many commercial raw processing programs perform a lot better in this area (curious to see how Danne would render this).
Quote
btw bit off topic, a line above in the link you posted:
> The read noise can be isolated by taking a "black frame" image, an exposure with the lens cap on and the highest
> available shutter speed; there are thus no photons captured, and only the electronic noise from reading the sensor remains.
Is this the actual proper way of doing a Darkframe ? Setting shutter speed to the highest available?
Not sure how exactly you reached this conclusion; this page (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/) (next time, make sure the article you are talking about can be identified, e.g. with a link) mentions the highest shutter speed as a way to analyze the read noise component (without e.g. thermal noise, which depends on exposure, or PRNU, which depends on the number of photons captured), not as the best way to capture dark frames.
I'd take the dark frames under the same conditions as the original clip. Probably some settings (like shutter speed) won't matter much, but I didn't experiment with it.
QuoteI have some Vintage lenses that have quite big variations in the Vignetting at the different F-stops, from 1.4 up to 5.6
Flat frame would correct the vignetting in those lenses as well - but you will need one averaged flat MLV at each aperture. Repeat for each ISO and some usual shutter speeds, and you'll probably end up with a 3D matrix of correction frames. That would definitely require some sort of automation.
Very informative, educative. Will try to find more time do some more testing around it. Just started vacay...
Hey guys! Thanks for lots of very interesting talking here :)
Quote from: a1ex on July 01, 2017, 09:07:47 AM
Otherwise I'd probably maintain this program, to some extent.
I'm sure every community member would be extremely thankful for that :)
Quote from: a1ex on July 01, 2017, 09:07:47 AM
Flat frame would correct the vignetting in those lenses as well - but you will need one averaged flat MLV at each aperture. Repeat for each ISO and some usual shutter speeds, and you'll probably end up with a 3D matrix of correction frames. That would definitely require some sort of automation.
as proposed in early MLV times, we could use FLAT and DARK frame types instead of VIDF.
a "correction.mlv" then would contain (sorted by block time):
MLVI, RAWI, RAWC, LENS, (misc info), EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, ...
where EXPO and FLAT+DARK are a series of correction frames for the EXPO settings.
(yeah DARK isn't possible as of now so we would omit it, but maybe somewhen we can do that)
possible correction frames: FLAT, DARK, BIAS
(see http://deepskystacker.free.fr/english/faq.htm )
An idea right out of my ass. What about creating one for all isos darkframe with rec trigger? Record a sequence with iso 100 then pause, set iso 200 record a sequence pause, iso 300 etc, then average this all iso MLV file to one hell of a darkframe?
yes, thats what i wrote with the sequence above ;D
EXPO <= ISO, exposure time
LENS <= lens, aperture, focal length
typedef struct {
uint8_t blockType[4];
uint32_t blockSize; /* total frame size */
uint64_t timestamp; /* hardware counter timestamp for this frame (relative to recording start) */
uint32_t isoMode; /* 0=manual, 1=auto */
uint32_t isoValue; /* camera delivered ISO value */
uint32_t isoAnalog; /* ISO obtained by hardware amplification (most full-stop ISOs, except extreme values) */
uint32_t digitalGain; /* digital ISO gain (1024 = 1 EV) - it's not baked in the raw data, so you may want to scale it or adjust the white level */
uint64_t shutterValue; /* exposure time in microseconds */
} mlv_expo_hdr_t;
typedef struct {
uint8_t blockType[4];
uint32_t blockSize; /* total frame size */
uint64_t timestamp; /* hardware counter timestamp for this frame (relative to recording start) */
uint16_t focalLength; /* in mm */
uint16_t focalDist; /* in mm (65535 = infinite) */
uint16_t aperture; /* f-number * 100 */
uint8_t stabilizerMode; /* 0=off, 1=on, (is the new L mode relevant) */
uint8_t autofocusMode; /* 0=off, 1=on */
uint32_t flags; /* 1=CA avail, 2=Vign avail, ... */
uint32_t lensID; /* hexadecimal lens ID (delivered by properties?) */
uint8_t lensName[32]; /* full lens string */
uint8_t lensSerial[32]; /* full lens serial number */
} mlv_lens_hdr_t;
Quote(yeah DARK isn't possible as of now so we would omit it, but maybe somewhen we can do that)
So we can uncomment some abort stops in mlv_dump to try this?
With all due respect flat frames needs some more testing and going throughs. As opposed to darkframes it's more tinkering time getting it to work to the better. Removing vignetting often causes grain and other side effects such as unexpected pinks in highlights. Not being able to upload examples isn't ideal here but I see if I find time here. Darkframes works very reliably imo.
Quote from: Danne on July 01, 2017, 01:31:57 PM
Removing vignetting often causes grain
Because of exposure pushing in the dark areas? Yes, that's expected.
Quote
unexpected pinks in highlights
White level may need to be adjusted; unfortunately, this will result in clipping useful highlights (that were not clipped in the vignetted area of the frame).
Possible workarounds:
- include some highlight recovery code in mlv_dump (hard, there are an infinite number of methods to do this, as it must guess what's in the clipped channels; that's the job of the raw processing software)
- variable white level support in raw processing programs (does any app support something like a white frame, or other sort of variable white level?)
- apply the vignetting correction after developing the raw image (after the highlight recovery is applied, but while still in some linear space) - depends on the processing app.
- decompose the flat frame into vignette correction (radial averaging, easy) and defects correction (dust spots, vertical stripes, whatever), and apply just the second part.
From all those, the last item appears to be the most practical to me; mlv_dump already does vertical averaging, so it's easy to add a similar option. A very large blurring filter might work as well, if the correction frame is not uniform for some reason.
i would keep mlv_dump as focused as possible.
if some correction is necessary in "mlv-space", i.e. where we act camera specific, where we are tightly coupled to mlv_lite/mlv_rec, i absolutely agree implementing that in mlv_dump.
also some "primitive" corrections like chroma smoothing, bad pixel maps etc is well placed there.
also the DARK, BIAS, FLAT correction is quite camera - or mlv-workflow - specific and its good to place it in mlv_dump.
lens vignetting correction, well, thats something that we get "for free" using FLAT frames.
more complex and generic corrections might be best somewere else. (hightlight recovery for example)
Quote from: g3gg0 on July 01, 2017, 02:27:12 PM
i would keep mlv_dump as focused as possible.
I've got a vague feeling that I'm gonna remove a lot of stuff from my PR :P
Quote from: g3gg0 on July 01, 2017, 02:27:12 PM
also the DARK, BIAS, FLAT correction is quite camera - or mlv-workflow - specific and its good to place it in mlv_dump.
DARK, FLAT frame types!
Fantastic, simple, effective proposal for "correction.mlv".
Quote from: g3gg0 on July 01, 2017, 02:27:12 PM
more complex and generic corrections might be best somewere else. (hightlight recovery for example)
Would be so nice to not rely on any raw processor and have several advanced hightlight recovery methods in the arsenal. Could be implemented outside of "mlv_dump.c" just during exporting DNGs. Or maybe introduce plugin framework? ;)
these frames were proposed from the beginning on ;)
well, mlv_dump already has this framework also from the very beginning. lua callouts.
if we integrate highlight recovery - then why not noise reduction, rotation, cropping, lens correction, deflickering, red eye reduction etc.
for me this is definitely raw manipulation and nothing that is a) required to make mlv usable or b) very camera specific.
This test mainly verifies this
QuoteThat means, not a single pixel would have been clipped in this frame by increasing the exposure by 1 stop (which means, no information would have been lost). You should be able to grade this file identically to the original (please try).
Mainly this suggests pushing f-stop more "to the right" to achieve cleaner shadows prone to fpn and stripes.
Opened up the files in ACR, altered white level acording to a1ex test above http://www.magiclantern.fm/forum/index.php?topic=18975.msg186641#msg186641.
Matched exposure accordingly(+2, +1.5, +1, 0)
Note white level 5585 and white clip warning. Check 4th pic.
From 5th pic and onwards some heavy highlight recovery and acr behaviour.
Final pics good old dcraw -H 2 recovery.
16200(https://s2.postimg.org/pmu8wyuvd/image.jpg)
9123(https://s2.postimg.org/kcpa5oamh/image.jpg)
7050(https://s2.postimg.org/5brl25hah/image.jpg)
5585 Note that highlight clip warning is shown in ACR, upper right corner
(https://s2.postimg.org/x0e89o4ax/image.jpg)
Now some heavy highlight recovery. Slider all the way to the left(https://s2.postimg.org/8h7lemaix/image.jpg)
The result by eye with highlight recovery applied. Pic one and two without any clipping. One stop push seems ok as stated by a1ex before.
16200(https://s2.postimg.org/uf428epjd/image.jpg)
9123(1 stop push)
(https://s2.postimg.org/e1k2p99e1/image.jpg)
Next two are showing some funky rebuilding artefacts. Do have in mind we are pulling slider to the maximum.7050(https://s2.postimg.org/jogfmqbwp/image.jpg)
5585 (https://s2.postimg.org/vg4axj6ix/image.jpg)
dcraw output verifies this as well. I used following commanddcraw -T -w -H 2 INPUT.dng
(https://s2.postimg.org/rert5m3m1/091_16200.jpg)
16200(https://s2.postimg.org/y3yclmoy1/091.jpg)
9123(https://s2.postimg.org/h5zbzsfk9/092.jpg)
7050(https://s2.postimg.org/5jf7yns95/093.jpg)
5585 (https://s2.postimg.org/wfz76zb2h/094.jpg)
Quote from: g3gg0 on July 01, 2017, 11:36:23 PM
well, mlv_dump already has this framework also from the very beginning. lua callouts.
btw, never used it myself. Should be tested if I broke something after my changes. And no one can do this better than you ;)
Quote from: g3gg0 on July 01, 2017, 11:36:23 PM
if we integrate highlight recovery - then why not noise reduction, rotation, cropping, lens correction, deflickering, red eye reduction etc.
That's right agree, but would be cool, all in one Swiss Army knife, well perhaps red eye is too much :P
Quote from: bouncyball on July 02, 2017, 09:24:19 AM
That's right agree, but would be cool, all in one Swiss Army knife, well perhaps red eye is too much :P
ehm. this was ironic... :)
@Danne
Thanks for thorough investigation!
So I did a DFA on the example you all are looking at, it did not help with the FPN. I think it actually made it worse. Mentioned this before on 'mlv_dump on steroids' before Bouncyball fixed a bug that only made VS corrections to the first frame; it's as if Darkframe Averaging makes more contrast for the stripes, but this might be bollocks from me. Again, just a hunch and you guys know a lot better than me what the correction/averaging and so on are actually doing.
I took the decompressed file in to MLVProducer where I made a FPN profile to try and remove it. In parts on his cheek and throat, the results were better, but it also introduced vertical FPN stripes in a few other places instead, like in brighter splotches in the background and so on. Comparing MLVProducer's output, the MLV_dump DFA output and just normal Vertical Stripe fix direct from MLV_dump (danne). It looks to me as normal Output from MLV_dump with VS Fix had the best result of the three.
Maybe someone else could have a go with the DFA, in case I made a mistake. I don't remember what ISO I shot this at, after decompression, the MLV says ISO 100 and shutter 1/120, the shutter is accurate for sure. So I assume the ISO is true aswell?
All three outputs vary in Colour and Luminance. Which might have skewed my perception of the Striped results (less contrast, more contrast, green darks, magenta darks etc..) Bottom line is, the FPN is still there. Unfortunately I don't have a lot of time to fiddle around matching these frames and outputs, I got a music video coming up which requires a lot of my attention on planning these days. But I still wanna help out as much as I possibly can, I really wanna get rid of this FPN, I mean I think the FPN in those shadows is surrounded with totally usable shadow details (after NR techniques), its just the Stripes making it unusable (to me).
MLVProducer made the MLV 2 Frames longer than the other converters, so I don't know what's going on there, I outputted 59.976 from MLVProducer, as it read the file, maybe it doesn't know how to count properly or interpolates somewhere along the line. The software is a bit funky sometimes, but I also never really used it, I was more of a RAW2CDNG guy.
Another question. When a MLV has been decompressed, it shows Metadata like ISO and Shutter speed, but after DNG's have been made, that metadata is gone? Wouldn't it be an idea to include that aswell in the DNG output?
Well, I'm actually looking for some sample footage with dark frame correction, and another one also with flat frame, as I've just started a test suite for mlv_dump (http://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_4k_test/). Currently I've used the MLV from this thread (hope you don't mind), and a buggy MLV from another thread.
So, if you have a small dark MLV to match this clip, I can look into it.
Not sure if these MLV frames and included flatframes could help any. Shot with eosm 14bit. Compressed with lj92(for upload size reasons, bad connection) in post so obviously will need decompressing back.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/eosm_flatframes.zip
332MB
Thanks - ideally it should be a real-world example with visible defects that would be corrected by these frames (maybe extract a short clip from one of your dark/flat experiments, where this correction makes a difference).
If possible, it should have some clipped highlights as well.
Currently I'm trying to set up a visual diff, to highlight image changes between different versions. These things are already used in web development - some examples (https://sqa.stackexchange.com/questions/2459/compare-screenshots-of-rendered-web-pages). I've also posted a proof of concept for ML menus (http://magiclantern.fm/forum/index.php?topic=12396.0).
I see what I can do.
@a1ex
I am uploading a unprocessed dark mlv, ISO 100 1/120.
I cut the MLV down to 30 Frames, I hope it is enough, the connection makes this the very max size for me 41 some MB. If you need more, you gotta wait 1-2 days and I'll have 3G or possibly real internet ;)
EDIT:
ISO 100 mlv
https://mega.nz/#!8NgVAAaK!9-870FPyEXgTXA9FGaV2za5NxkHRM8WFmJke661sqDo
Thanks - here's my attempt, but unfortunately the dark frame introduced additional artifacts...
Maybe these dark frames really have to be recorded for each clip (ideally at the beginning and at the end - or just at the end, to avoid missing the action).
https://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_darktest/ (0 = plain, 1 = with dark frame)
Maybe this isn't relevant to this thread, but could there be a darkframe feature added to mlv/lite where it closes the shutter for a short time at the end of recording and records a few frames in "DARK" blocks? Mlv specific apps could take good advantage of this ;)
That's a nice idea but it would prolong every clip with around two seconds. Also note that recording a darkframe for every clip isn't needed so then maybe it would be useful as a menu option.
Another idea I have is to build a lua script which would record a short darkframe sample of each iso on any specific cam and then the user could use these files in a dark frame storage. Let's say one loads the script and then it ends when all files are recorded instead of manually having to do this.
Yeah, it would be nice as a menu option. A script is a good idea, but I think noise pattern might be different at different shutter speeds, probably a good approximation though. Does anyone else think it's worth adding this as an option?
I don't think that electronic rolling shutter can be turned off and sensor still be scanned and data read out still be functional. Otherwise we'd be able to use whole sensor as optical black area. Am I wrong?
If mechanical shutter could be closed in live view that would be another story.
alex and i already discussed that stuff while ago.
main problem is, we maybe could make a 0-exposue readout. this is a BIAS frame iirc.
a DARK frame involves the correct exposure time and ISO (or at least correct exposure and a fitting BIAS frame for the ISO etc)
we could somewhen also move the mirror/shutter down and then expose for a DARK frame,
but currently we do not know how to achieve this.
this is most likely controlled in the MPU which we didnt reverse engineer yet.
Hi guys!
Some more updates on the 1st post, download link is the same.
regards
bb
The power of "c". Great work B.
Running time command comparing creation time building the first croprec focus pixel file for eosm:
fmputil
real 0m0.021s
user 0m0.012s
sys 0m0.005s
fpm.sh
real 0m4.717s
user 0m10.848s
sys 0m2.304s
Nice, it's about 225 times faster :)
@bouncyball @Danne @a1ex @ g3gg0 @anyone else interested.
Bug Report:
I talked to A1ex about this one, as I feared the Lossless compression in 5D3 had messed up badly on two shots I made, it looks like data corruption or failed compression. To my great relief and anyone else using Lossless MLV, it is not the camera. A1ex thinks it is some kind of filtering or some bug of some sort in the 'mlv_dump on Steroids' causing this.
Quote from: a1ex on August 15, 2017, 07:13:36 PM
It's not underflow (that was my first guess, but then I've compared the DNGs in octave and it's more likely to be some sort of filtering operation). If the mlv_dump on steroids has some option to output unprocessed raw data, you could try that.
JPEG screenshot:
https://mega.nz/#!hAo32YwD!iSr0T0XTJdKZ3FJupV6tApUhk7rcixlr5xde8PW7PJI
DNG:
https://mega.nz/#!IIBSDbBT!r4cwb58zCnbuZwcxcPoXY9rixJ2jr3uok5iFQktgtzQ
MLV:
https://mega.nz/#!lZ4A0SyD!Fl1SxrgbM0XtP7RL8vPwK43q9ePhBkSR2iK1bowCsiw some 500 MB.
The "corruption" is stuck in specific spots, its actually in many different places if you look closely its just much more visible in the "high probable aliasing" of the building. Perhaps Vertical Stripe fix messing up?
Both shots where this messed up are of this same building, but it might just aswell be in other shots I have, I just really noticed it on this one.
Converted with Danne's Batch Script combined with "his" mlv_dump on steroids.
5D3 - 123
June 19th build
Let me know if you want any more info.
I tried processing the file over here on windows and I couldn´t reproduce the artefacts in dng 245. Are you using any additional settings in the menu when developing? I just ran it without any adding.
No, no additional processing.
And it is reproducable, again and again.
With your Batch Script Menu, I tried outputting Non-Processed Raw (3). But the program crashes.
Just tried -p option with and without spaces, in the same folder to another drive etc, without darkframes folder and with. It all worked outputting raw dng files with the setting -p. Strange.
You could try outside the script by:
mlv_dump -p --dng INPUT.MLV
You don´t have any darkframes in your folder? Could you try disbling vertical stripes fix. Also disabling cold pixel fix.
´´It is not the Darkframes causing it, it is the combination of Cold Pixel Fix with Darkframes causing this corruption. First tried disabling Vertical Stripe fix with DF's and then with --no-fixcp and now it works with Darkframes.
Thank you Danne and A1ex!
Nice catch Kharak.
Can you upload a dark frame for this one, so I can add it to the test suite?
Yes, will update this post when done.
Edit:
DF MLV straight out of camera:
https://mega.nz/#!dAg0jQjZ!bs2kX3TwL5UD-kXL7_fCgRQI_nC7-eFwLYD0l6gYKFY ISO 100 216 MB
Averaged DF via Mlv dump on Steroids:
https://mega.nz/#!1AgkwAwR!J0-1xq7emM9-2UFIL4Kk0NMtKMxcrK0Gemow6vmR7F8 2.6 MB
@kharak
Switch uses both mlv_dump versions, for simplicity me and Danne are calling them: ml-dng and steroid version :).
Can you be more specific about version you used? Steroid supports two interpolation methods for bad pixel removal - from mlvfs (more like cr2hdr) and from raw2dng which is used by standard mlv_dump.
Try both in conjunction with dark frame:
mlv_dump -bpi 0
mlv_dump -bpi 1
I'm really curious if there will be difference.
thanx
bb
Actually, he is using batch_mlv which uses your steroid version only named mlv_dump :P
I should rename it. Anyway. Interpolation method tests are important. I havn't inserted this to batch_mlv but as soon as Kharak uploads his darkframe shot we could check this.
Ah well, never mind. I tried it myself with your posted darkframe sample. Either bad pixel removal method causes artifacts, different ones thought. Can't say which one is better :P both are ugly in it's own way.
I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.
bb
@Danne: ah I forgot that Kharak uses windows :)
I just ran the provided darkframe and matching MLV file from Kharak through my version of mlv_dump in Switch and it was not affected by any corruption in file nr 245. My ml-dng version should be closer to unified mlv_dump. I did not disable cold pixel fix.
https://bitbucket.org/Dannephoto/magic-lantern/branch/ml-dng-unified_11b
I could reproduce the corruption with mlv_dump_on_steroids in Switch.
Not affected version of mlv_dump:
terminal output:(beginning)
MLV Dumper
-----------------
Mode of operation:
- Input MLV file: 'M31-1527.MLV'
- Decompressing before writing DNG
- Enforcing 14bpp for DNG output
- Convert to DNG frames
- Subtract reference frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'
- Output into 'M31-1527_1_2017-07-31_0001_C0000_'
File M31-1527.MLV opened
File M31-1527.M00 not existing.
Loading subtract (dark) frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'
terminal output(ending):
Processing...
Vertical stripes correction:
1.000 1.000 1.004 1.003 0.998 1.001 1.000 1.003
Frame0 : cold pixels found: 0
Reached end of chunk 1/1 after 367 blocks
Processed 348 video frames at 59.98 FPS (5.80 s)
Done
Hmm... It says: "Frame0 : cold pixels found: 0" this equals to not touching data and not fixing anything hence no artifacts.
Now I understand better what's happening. After dark frame subtraction, MLVFS/cr2hdr bad pixels scaning code tries to find all flavors of bad pixels eg: cold, hot, dead...
raw2dng code is for cold pixel revealing only. In steroid for revealing bad pixels I've used only advanced scaning code from MLVFS but for interpolation both methods are available trough the switches.
bb
For reliability sake, all pixel fix methods should be default set to off. When pixel peepers see dead, cold, hot things in their dng files they should reprocess with correct pixel method set.
Otherwise we are gonna keep on narrowing down these little never ending mysteries for a long time :)
I think I have only ever seen the Cold Pixel Counter say 1, once. Are Cold Pixels usually an issue?
How does a Cold Pixel look like? Is it a dead pixel e.g. Black?
I guess it's more like dead pixel and is nearly black. Someone correct me please :)
Think they can come in different colors.
http://www.magiclantern.fm/forum/index.php?topic=17795.msg171191#msg171191
Okey, luckily not an isuue I come across on the 5d3. Perhaps the 3x3 bayer helps get rid of it, as it has 8 other pixels to draw on in the case of a cold pixel?
On the 5D mark III there is seldom any use of cold pixel fix. If you use darkframes they take care of them anyway. As well as the hot pixels if any.
Cold pixels used to be a problem before this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/778); see also this analysis (http://www.magiclantern.fm/forum/index.php?topic=18393.0). I think most of them were actually pixels marked as bad by Canon firmware (they mark them as 0, so they are very easy to detect by our "cold pixel" heuristic). Some of them were not 0; they were only below the black level, although I don't remember the specific conditions one would get such artifact.
Need to double-check whether there are still cold pixels on other models (60D and 600D were troublesome, possibly also 550D and 500D, not sure about others). One has to compare a MLV (or a DNG developed with all processing turned off) from any recent build, vs any 2016 build.
Some of the "cold pixels" used to appear near very strong highlights (there were some bug reports about this for 5D3, with green artifacts: this one (http://www.magiclantern.fm/forum/index.php?topic=7122.msg109315#msg109315) or this one (https://bitbucket.org/hudson/magic-lantern/issues/2593/green-noise-in-fire) or this one (http://www.magiclantern.fm/forum/index.php?topic=16401)). These are dynamic and won't be fixed with default processing settings (they require scanning every frame for bad pixels, which is very slow: option --fixcp2).
Those examples all seem to derive from bad post-processing software of the mlv/raw. No?
The only thing i see as a consistent issue with the 5D3 is with red channel in high contast to other colours, be it black or blue or green etc. there is an example on the forum from maybe 2 years ago or more where a guy showed an example from some red flowers in the foreground with a blue sky behind (red and blue channel interacting) and he had grey pixels howering over the red colours (in the blue sky) some few pixels away. I cant find it, but remember it as later I myself have seen this a lot. It comes mostly from "really" red sources.
I spend so much time on fishing vessels shooting for clients and for personal use. Here the issue comes up a lot, the bright red/orange Oil Clothes that is the typical colour of any fishermans outfit, in contrast to basically any other darker colour will exerborate this behaviour. Gray pixels hovering over the edge of the Red/orange.
I also recall back in the day shooting on boats, pre dslr, shooting with Panasonic and sony, crappy bit depths, they could not reproduce the true colours of the Outfits, they would turn in to a red blob/mess or orange mess, very difficult to work with.
Dont have my computer, so no examples to show from.
The colour is very intense, its made to be as visible as possible in case of MOB or Blackouts.
Quote from: Danne on August 18, 2017, 01:47:58 PM
On the 5D mark III there is seldom any use of cold pixel fix. If you use darkframes they take care of them anyway. As well as the hot pixels if any.
+1 8)
Quote from: Kharak on August 18, 2017, 10:06:41 PM
I also recall back in the day shooting on boats, pre dslr, shooting with Panasonic and sony, crappy bit depths, they could not reproduce the true colours of the Outfits, they would turn in to a red blob/mess or orange mess, very difficult to work with.
Yeah I remember that mess of deep saturated colors with my SONY and JVC analog and early DV cameras. :) That JVC had 25p progressive scan mode and it hooked me, funny the DR was about 5-6 stops I guess :P
If you'll find the time, post gray pixel artifact examples please.
bb
I will find you some examples.
Quote from: bouncyball on August 17, 2017, 10:46:25 AM
I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.
bb
If only that was true, but not :(
I've processed file with --no-stripes --no-fixcp twice - one time with black frame, one - without and than pixel peep the difference.
Hot pixels(bright ones) are really gone with black frame, cold/bad - no sure, i dont see any on both shots,
BUT, vertical stripes are the same on both!!! keep investigating, though...
Here is an example with the Red Channel having difficulties with the high contrast edges.
Like in this example, look at the Boots of the guy closest to camera, the Red Oil Pants are outlining the boots around the edges. The edges get This is pretty much consistent from 5D3.
It makes the Boot look "cut out" of the picture or placed there. The Gray pixels appear all over the edges.
When Sharpening the image which I always do, this effect becomes very obvious.
DNG:
https://mega.nz/#!1MoRjCKZ!yU7zbQFKMEpEi4VxCqYIqOdkfVLzDvfnKgOf3JKt-aQ
@Kharak
Sorry about very late answer :)
Quote from: Kharak on September 03, 2017, 06:46:10 PM
It makes the Boot look "cut out" of the picture or placed there. The Gray pixels appear all over the edges.
When Sharpening the image which I always do, this effect becomes very obvious.
Yeah you are right, now I understand what did you mean.
bb
I was working with the latest version of the crop_rec_4k branch on my 700D and the current linux (and windows) version of mlv_dump on steroids doesn't handle focus pixels any more. If you can update it, would be much appreciated. (I've compiled my own version for now, but I'm sure there are others with the same issue.)
Focus pixel fixing has been breaking on various experimental builds. Seen it happen on the EOSM too. Not sure what is going on but in one instance everything was shifted 8 pixels to the right.
Could you document which build, what settings, etc. you're seeing the focus pixels so we can try to reproduce the issue? Also posting a short MLV file would help. There's a way of trimming down an MLV file to only a few frames.
mlv_dump -o output.mlv -f 5 input.mlv
Quote from: ErwinH on October 24, 2017, 10:33:30 AM
I've compiled my own version for now, but I'm sure there are others with the same issue
What do you mean by this? Did you patched code of mlv_dump (+shift/-shift pixels) or managed to forge focus map for your particular case?
[/quote]
Quote from: dfort on October 24, 2017, 03:32:51 PM
Could you document which build, what settings, etc. you're seeing the focus pixels so we can try to reproduce the issue? Also posting a short MLV file would help.
+1
If you meant that only MacOS binary is updated and not Win/Linux ones - then yes, that's right ;)
Both now uploaded (https://bitbucket.org/bouncyball/magic-lantern/downloads/).
This.
Quote from: bouncyball on October 24, 2017, 07:12:59 PM
If you meant that only MacOS binary is updated and not Win/Linux ones - then yes, that's right ;)
Both now uploaded (https://bitbucket.org/bouncyball/magic-lantern/downloads/).
When I have a little more time I'll look at the different versions and where it goes wrong. I'll come back on this.
I have a recording where the old mlv_dump-on-steroids fails and the new one works:
Old version:
MLV Dumper
-----------------
Last update: 33ac267 on 2017-07-25 14:35:07 UTC by bouncyball:
pixel_proc: now supported both: newer .fpm with header or origin...
Build date: 2017-07-25 17:09:30 UTC
New version:
MLV Dumper
-----------------
Last update: f44ffe0 on 2017-10-15 13:02:25 UTC by bouncyball:
pixel_proc: changes to focus pixel map generator
Build date: 2017-10-24 08:10:55 UTC
The command I use is:
~/mlv_dump.linux --fpi 0 -f 5 --dng
I'll send the mlv file per PM.
MLV Dumper
-----------------
Mode of operation:
- Input MLV file: 'M17-1504.MLV'
- Verbose messages
- Verify file structure
- Dump all block information
File M17-1504.MLV opened
File M17-1504.M00 not existing.
Processing...
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 5477059852982235116
FPS : 25.000000
File : 0 / 0
Frames Video: 74094
Frames Audio: 0
Class Video : 0x00000001
Class Audio : 0x00000000
Block: RAWI
Offset: 0x00000034
Number: 1
Size: 180
Time: 0.782000 ms
Res: 1280x720
raw_info:
api_version 0x00000001
height 1189
width 1808
pitch 2260
frame_size 0x002900A4
bits_per_pixel 10
black_level 128
white_level 1013
active_area.y1 28
active_area.x1 72
active_area.y2 1185
active_area.x2 1808
exposure_bias 0, 0
cfa_pattern 0x02010100
calibration_ill 1
Block: RAWC
Offset: 0x000000e8
Number: 2
Size: 32
Time: 0.797000 ms
raw_capture_info:
sensor res 5184x3456
sensor crop 1.62 (APS-C)
sampling 3x3 (read 1 line, skip 2, bin 3 columns)
Block: IDNT
Offset: 0x00000108
Number: 3
Size: 84
Time: 0.811000 ms
Camera Name: 'Canon EOS REBEL T5i'
Camera Serial: '9C93E2F79'
Camera Model: 0x80000326
Block: EXPO
Offset: 0x0000015c
Number: 4
Size: 40
Time: 0.823000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0/1024 EV
Shutter: 20000 microseconds (1/50.00)
Block: LENS
Offset: 0x00000184
Number: 5
Size: 96
Time: 0.847000 ms
Name: 'EF-S18-135mm f/3.5-5.6 IS STM'
Serial: ''
Focal Len: 24 mm
Focus Dist: 132 mm
Aperture: f/4.00
IS Mode: 0
AF Mode: 3
Lens ID: 0x0000102E
Flags: 0x00000000
Block: RTCI
Offset: 0x000001e4
Number: 6
Size: 44
Time: 0.863000 ms
Date: 17.10.2017
Time: 15:04:43 (GMT+0)
Zone: ''
Day of week: 2
Day of year: 289
Daylight s.: 0
Block: WBAL
Offset: 0x00000210
Number: 7
Size: 44
Time: 1.495000 ms
Mode: 9
Kelvin: 5500
Gain R: 483
Gain G: 1024
Gain B: 660
Shift GM: 0
Shift BA: 0
Block: VERS
Offset: 0x0000023c
Number: 8
Size: 172
Time: 58.597000 ms
String: 'mlv_lite built 2017-10-16 20:22:53 UTC; commit 8ee7858 on 2017-10-16 20:21:52 UTC by alex: mlv_lite: fix 8..12-bit lossless in 720p on small cameras '
As you named it - "Old" mlv_dump supports focus pixel correction only by .fpm map files, while "New" one has built in map generator, hence does not require map file in the binary directory. For overriding the built in generator you should put the map file beside mlv_dump and if the name of the file matches the cameraID and raw buffer resolution it will be used instead.
Your sample MLV is recorded for mv1080 video mode and should be processed w/o issue, but... as you can see raw buffer resolution is 1808x1189, not 1190 and there is no .fpm file with that name by default. That is why "Old" version did not work for you out of the box (you had to rename the file to 80000326_1808x1189.fpm, this is the established standard used after MLVFS). On contrary "New" mlv_dump does not care about vertical resolution mismatches and will work for any Y resolution.
There is also one thing. If you're gonna process MLV recorded in crop_rec mode, then to activate correct built in map generator mlv_dump needs the switch '--is-croprec' on the command line. It is temporary solution until I implement parsing of RAWC header information to make sure this mode is indeed crop_rec.
regards
bb
Thank you for the explanation. Good that it wasn't anything serious.
I'll be shooting a new session tomorrow and will test it again.
Small bug report. If you set the output to a folder -o DNGFOLDER/ the wav file isn't exported.
If you set it to -o DNGFOLDER/prefix the wav file is exported.
Thanks, I'll check it out.
Edit: is that the same in official/vanilla mlv_dump?
Official mlv_dump has the same issue.
OK thank you.
Hello, I've been following these forums for over a year now, finally decided to make an account. I'm really stumped right now, and have a few questions. :D
Will (mlv dump on steroids) process my raw, dual iso files? and is there a guide on how to use this mlv dump?
the (batch mlv in out) program works great but it does not have any option for dual iso, does it? maybe i missed somthing.
I can process dual iso raw video on my macbook already through the mlvfs, but it takes WAY to long to transfer back over to my pc. around 20mins when copying a single 1g file off of the fuse mount ???.
Thank you for reading. any help will be greatly appreciated.
@SharpieUser
mlv_dump does not support dual iso processing, just fixing bad/focus pixels for dual iso with '--is-dualiso' switch. I wanted to implement this feature but then gave up, not worth it.
Instead I implemented it to the MLV App (http://www.magiclantern.fm/forum/index.php?topic=20025.0). Try it and and tell what do you think :)
regards
bb
Possible bug: http://www.magiclantern.fm/forum/index.php?topic=15088.msg194917#msg194917
@a1ex
Hmm... LINK (https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff). Could this be corrected/done in camera in the future? To not mess with channel data offsetting by 128,256,512,1024 in post? This (http://www.magiclantern.fm/forum/index.php?topic=15088.msg194950#msg194950) will be temporary solution for post tools or am I missing the basic idea?
bb
Well....
- the link is for new-style DryOS task switching hooks on recent D5 models; it should not impact any functionality (makes the way we are overriding tasks more "unified", adds CPU usage, some error checks, stuff like that)
- these 128,256,512,1024 black level offsets are a LJ92 encoding bug on 6D, that we have no idea how to track down, so I thought it may be a better idea to work around this in post
- the sample DNG files (coming from mlv_dump on steroids) could not be adjusted with exiftool -BlackLevel (but others found out they can be adjusted with -IFD0:BlackLevel)
- the same sample DNG files (coming from mlv_dump on steroids) could not pass dng_validate.exe tests (and that's the bug report for you).
- edit: OP found out he was using an older version (https://www.magiclantern.fm/forum/index.php?topic=15088.msg194960#msg194960), so please disregard.
"Unfortunately, MLRawViewer just locks up my current machine (bringing down the entire operating system), so it's not straightforward for me to play with this code. Otherwise I'd probably maintain this program, to some extent."
@a1ex Then we should avoid it. Is there another player for just viewing MLV files?
Thx a1ex, got it. I'll try to fix 6D case in software, but... I really do not like this ;)
btw, I compiled dng_validate for linux. If someone needs it, it's here (http://nic.caucasus.net/ml/?dir=&download=dng_validate). Note: It might require some libs to install.
Quote from: JADURCA on December 22, 2017, 02:22:06 PM
@a1ex Then we should avoid it. Is there another player for just viewing MLV files?
a1ex said this before @hwdbk made his changes to support losless MLVs. Link is Here (https://bitbucket.org/hwdbk/mlrawviewer/downloads/). It is not finished thought and needs further development.
Meanwhile you can play MLVs with MLV App (https://ilia3101.github.io/MLV-App/) for all 3 OSes (Mac, Linux, Win), or use mlvproducer (https://ilia3101.github.io/MLV-App/) (windows only).
bb
Hey guys some updates to the 'mlv_dump os'. Added feature to auto generate focus pixel maps according to latest break through development.
Downloads are on the first post as usual.
bb
Thanks, @bouncyball!
Quote from: bouncyball on December 22, 2017, 04:55:56 PM
a1ex said this before @hwdbk made his changes to support losless MLVs. Link is Here (https://bitbucket.org/hwdbk/mlrawviewer/downloads/). It is not finished thought and needs further development.
Meanwhile you can play MLVs with MLV App (https://ilia3101.github.io/MLV-App/) for all 3 OSes (Mac, Linux, Win), or use mlvproducer (https://ilia3101.github.io/MLV-App/) (windows only).
bb
Thanks! After I wrote this, found MLV App and mlv_dump on steroids. I'm using both actively, mlv_dump os for darkframes high ISO footages. MLV App dont have this darkframe feature right?
Quote from: JADURCA on January 15, 2018, 06:42:21 PM
MLV App dont have this darkframe feature right?
Not at this moment, but... ;)
Quote from: bouncyball on January 12, 2018, 05:10:49 PM
Hey guys some updates to the 'mlv_dump os'. Added feature to auto generate focus pixel maps according to latest break through development.
Downloads are on the first post as usual.
bb
Great! I just download it and overwrite mlv_dump file. Right?
Thanks!
Having problems with both apps now. MLV app and mlv_dump os :-\
Quote from: JADURCA on January 17, 2018, 12:04:42 AM
Having problems with both apps now. MLV app and mlv_dump os :-\
Could you be more precise please? Maybe you can provide a small MLV for debugging as well?
I also continue having problems with MLVApp on both: my laptop Lenovo 3000 N200 with 4 GB of RAM and my desktop PC - QuadCore Q6600, MoBo DP35DP, 8 GB DDR2 RAM, GTX750 Ti GPU. On both of them I have Win7x64 Ultimate running. MLV_dump on steroids works on both PCs fine.
I tried the 64-bit version 013 of MLVApp on both of them. When I click on the software icon it starts normally and opens the main window. When I try to import an MLV file, regardless of its size, resolution, bitrate and/or compression, the software crashes and the message "MLVApp.exe has stopped working" appears. I also tried the 32-bit version and observed the same behavior.
Interestingly, the software works flawlessly on a different laptop of a friend of mine who has exactly the same version of Win7x64 installed. His laptop is a Dell Precision M6600 with i7 2720 and 16 GB of RAM. Could it be that MLVApp requires at least 16 GB of RAM to work properly?
I'm running it w/o issues in Win10x64 on my 5 year old Asus Zenbook laptop with 4gb mem.
Edit: And in virtual machine with Windows7 x64 (4GB) under Vmware ESXi @ my work.
Quote from: JADURCA on January 17, 2018, 12:04:42 AM
Having problems with both apps now. MLV app and mlv_dump os :-\
Please tell us what kind of problem is it.
Upgraded from Windows 7 64-bit to Windows 10 64-bit. Maybe is that the problem? Where I can download mlv__dump on steroids again? To try again right away and send you photos in case it don't work.
Hm... strange. I have MLVApp running on Win7 64bit Core i3 4GB RAM, Win8.1Pro 64bit Core i7 8GB RAM, OSX 10.9 16GB RAM, 2x OSX 10.11 8 & 24GB RAM and Ubuntu 12.04LTS VM without that problem. It is bad, that Windows never tells the reason for a crash...
Do you activate "AMaZE Cached"? This crashes Windows 32bit versions on my systems.
@masc True is that MLV app is not the one crashing, is mlv_dump on steroids. I'm having other problems that can't be reproduced with MLV app, so I can't use it anymore for extracting, just for playing mlv files. Can you help where to locate mlv_dump on steroids? I can't find it to download again...
Ah okay. Then I missunderstood. Is it here?
https://bitbucket.org/bouncyball/magic-lantern/downloads/
Here are the latest versions of mlv_dump on steroids both mac and windows
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_os.zip
If on mac make sure you give it priviligies:
chmod u=rwx mlv_dump_os
enter
Thanx Danne. When I'll update it with latest stuff compile the mac binary for me. My macos VM does not boot anymore ... IDK why.
I think I'm confused here, I'm looking for an archive called "batch_mlv_in-out" to download it so I can retest it using mlv batch process and darkframe folders, all of that. Im not making myself clear.
Ok, you can replace it with the one provided in the link. Just rename mlv_dump_os.exe to mlv_dump.exe and replace the one inside. If you still have issues please post about them in the other forum post.
Quote from: Danne on January 17, 2018, 03:51:34 PM
Ok, you can replace it with the one provided in the link. Just rename mlv_dump_os.exe to mlv_dump.exe and replace the one inside. If you still have issues please post about them in the other forum post.
Thanks! Then I'm renaming and replacing to test. Can you please tell me the title name of the other forum post?
The one where you got batch_mlv_in-out from. Try pasting in search.
Quote from: JADURCA on January 15, 2018, 06:42:21 PM
MLV App dont have this darkframe feature right?
Quote from: bouncyball on January 15, 2018, 07:32:13 PM
Not at this moment, but... ;)
Dude... don't be such a tease!!! :P
Quote from: DeafEyeJedi on January 17, 2018, 05:41:08 PM
Dude... don't be such a tease!!! :P
All right, all right! I won't be... until... :P
C'mon don't play with our feelings ;D
Is there any way I could save only odd or even frames out from an input MLV file and output that into a new MLV file?
I need to do this to save some space of redundant duplicated frames of some MLV written by mlv_rec.mo that wrote duplicated frames in 1fps frame override mode in 1x crop on 5D2.
I tried -f switch but that only allows input of frame or frame range, not comma delimited frame numbers, -f 0,2,4,6,8 for example.
Any ideas would be appreciated. Thanks!
Yeah... unfortunately this is not supported by mlv_dump directly (MLV -> MLV).
What's wrong with exporting DNGs and just picking the needed ones?
Quote from: bouncyball on April 27, 2018, 01:30:21 PM
Yeah... unfortunately this is not supported by mlv_dump directly (MLV -> MLV).
What's wrong with exporting DNGs and just picking the needed ones?
Nothing wrong with it, that's what I did. It is just that I would usually keep the source MLV, just in case in I need to re-dump the MLV with a newer/better mlv_dump with improved vertical banding removal, etc. maybe who knows.
I understand your point.
But IMHO the better way to fix this with mlv_rec.mo to not duplicate frames when fps = 1. Implementing something like even or odd only frame saving is not that hard but barely widely usable, despite duplicates can turn out be like: 01010111010100 (given that 0 and 1 are frames) this requires picking not only odd/even frames and will be even more work for developer :).
regards
bb
How about -f "frameslist.csv" ?
make the -f switch when referencing a user generated .csv to only include those frames listed in it.
This should be done in the mlv_dump code not in the code I added, so maybe you will try to ask g3gg0 b/c this proggie is his exclusive child ;)
Quote from: bouncyball on April 29, 2018, 08:19:06 PMthis proggie is his exclusive child ;)
not correct ;)
accepting pull requests if they fit in cleanly.
and such addition sounds if it can be done in a clean way.
Quote from: g3gg0 on April 29, 2018, 09:00:54 PM
not correct ;)
Well... right ;). Sorry about being too lazy.
@g3gg0: BTW could this be done with mlv_dump's LUA scripting feature?
yeah good point. could be done, have to check this.
maybe we have to add a callback but no big issue.
I like this :)
Thanks bouncyball & g3gg0 :D looking forward to the feature addition.
Not sure if this is the right place , getting a
Win32 Expceptions
debugging it in Microsoft Visual Basic 2015
Exception thrown at 0x7548A04C (msvcrt.dll) in mlv_dump.exe: 0xC0000005: Access violation reading location 0x02D31000.
Unhandled exception at 0x7548A04C (msvcrt.dll) in mlv_dump.exe: 0xC0000005: Access violation reading location 0x02D31000.
mlv_dump.exe (downloaded today) from Branch: crop_rec_4k (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/72/artifact/modules/mlv_rec/mlv_dump.exe) on
the https://builds.magiclantern.fm/experiments.html page -- running win7 pro ,
in call stacks this
> msvcrt.dll!___ascii_strnicmp ()
ntdll.dll!77acfaea()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
seems this line in the disassembly
7548A04C 66 0F 6F 4E 10 movdqa xmm1,xmmword ptr [esi+10h]
most of this is over my head but i thought it my be useful for others.
No sure if it's just my test .mlv that's the problem , wondering if any one else has had a "Win32 Expceptions"
Is that exception happens when just running the binary or if you specify some switched to do some job?
Please post exact CLI command and exact MLV you used.