Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - smasry

Pages: [1] 2
Reverse Engineering / Re: UHS-I / SD cards investigation
« on: October 08, 2020, 05:20:53 PM »
Ok. Work only with sd card in your camera.

You are gonna need this build:

Replace the to the one below:

Enable The rest of the modules will be enabled automatically.

1 - Enable the sd uhs patch called 240Mb/s and restart camera.
2 - Enter photo mode and run the 1 minute benchmark test. Are you getting high readings? Should be bitmap file on your sd card recorded.
3 - Now take a photo, then rerun benchmark test. Are you still having high numbers? If not your card will not respond well with 240Mb/s. If results still are high move on to step 4
4 - Enter movie movie mode. Start recording in anamorphic mode. Do about three to 5 takes. Everything orking as expected? If so, this card should be working with the patch.

Ok, Danne, done 1, results of step 2 are:

- Following step 2 from scratch:

- Tried to take a photo, got a warning about card 2 being unavailable, followed by an Err 2 crash. Upon restart, deleted the photo (which *did* save), and reran step 2 with higher benchmark results:
- Just in case, I restarted the camera, deleted the new images, ran the benchmark, took two more photos, and reran the test with equally high results:

- Entered movie mode, tried to record in anamorphic mode, got 'CARD FULL' error, over and over again. Tried restarting the camera, trying to record video failed again with the same message:
- Trying to start a video after the card full failure message results in:

- Eventually, after a few restarts, I was able to shoot a video for around 10 seconds, which then auto-stopped. Trying to take another video after resulted in the same 'CARD FULL' message.

Reverse Engineering / Re: UHS-I / SD cards investigation
« on: October 08, 2020, 02:25:50 PM »
I need to prepare a build later tonight. However. You need firmware 1.1.3 or 1.2.3 to even be able and install magic lantern. Ever installed before?

Sorry, Danne, forgot my firmware version... it is 1.2.3.

And yes, I’ve been using ML for years now, am presently on your Nightly.2020Jul05.5D3123, as reported by the Magic Lantern launch menu.

Reverse Engineering / Re: UHS-I / SD cards investigation
« on: October 07, 2020, 02:40:05 PM »
Beautiful numbers Zeek.
Could we have a 5D3 tester with this card? Let me know if so.

Hi Danne, I have the 512Gb x170 Sandisk SD and a 5d3 running 1.2.6, and am happy to test.

Just point me to the right build and tell me what to do, and I’ll post the results up here.

While I have had as good an experience with my 256Gb 1066x Komputerbay card as most others on this forum, once it failed, Komputerbay support *never* responded to my emails, even once. After sending them 3 emails over several months, I gave up.

Sandisk works superbly well, and they have always responded and replaced failed cards, impeccably, that’s the difference in price you pay.

I have a 256Gb, several 128Gb and 64Gb Sandisk 1066x cards, and can say that they all work consistently well, and that the speed differences between them do exist but are fairly minor.

Just my experience.

Hardware and Accessories / Re: External monitor
« on: June 02, 2018, 11:39:47 PM »
@allemyr I have an Atomos Ninja 2, from the cheaper end of their range, and it mirrors the ML display, treating it as a focal plane. The result? All ML UI elements are in focus, and are highlighted by focus peaking. In other words, it doesn't work (for me, with 5D3 1.2.3) for focus peaking.

I know that the waveform, histogram and false colours work, but it may be reacting in the same (false) way; I haven't tested them. For focus, articulation and a backup recording, it's fantastic!

Thanks, @a1ex, can't wait to test this! And thanks, @danne for already building the mac cr2hdr binary!

Yes, aliasing is bad; I should have invested in a wider lens and shot this in crop mode. It wouldn't have been too terrible then.

As for the aliasing reduction using your script, hugin, enfuse and all the other options listed in that thread, it looks fantastic, just one problem: we lose the RAW image, leaving less data to work on, in the colour-correcting stage. I hope I'm wrong on this issue, would a 16-bit TIFF or EXR may give us the same leeway as a 16-bit DNG?

MlRawViewer (1 month old is too old, really?) doesn't show any obvious flicker.

I stand corrected; I was sure development of MlRawViewer was discontinued, but am happy to be wrong in this case.

If you have some other footage that shows flicker (in particular, extreme cases), I'd like to see it as well (small cuts like this are OK).

I have plenty; I was worried that something's wrong with my 5D3 or my workflow since I seem to be getting flickering in most if not all dual-iso footage, while no-one else seems to be reporting anything. Perhaps it's because I push it beyond your recommendations of 400/1600 or 200/1600 and go all the way to 100/1600, leaving fewer midtones?

Unfortunately, I've deleted a lot of test clips, taken while I was learning, and experimenting with, this feature. Here are more clips from the same shoot, though:

The last two are more extreme, as they are looking towards the sun. I think the setting are still the same (100/1600), but can't remember for sure.

You're right, the upload failed, sorry. The link works now, though.

Hi @a1ex,

Sorry, unable to recover the input file from output images...

Quite right, here's a cut-down, thus lighter, MLV file:

I'll have to read through your shell script and instructions carefully and test. That and the zip file you sent.

Was the input MLV flicker-free before processing? (if so, that must be a bug in cr2hdr)

Hard to say, since I apparently have no idea how to monitor it properly; MlRawViewer is old, Footage is, well, dead in the water. In MLV App, my present favourite (thanks for the great software Illia, Masc and company!) it obviously plays back with flickering when I bring up the exposure, which is what 75% of the image needs. If I leave it with no exposure change, the image is quite dark, and it's hard to tell.

Is there a recommended way to preview/monitor the MLV file prior to processing? Sorry if it's an obvious question...

Hi @Danne, @bouncyball. I've processed the MLV with Switch again, using no extra flags, and can still 'see' the problem.

On the other hand, what you and @a1ex have said is only now making sense; it's the developing software that I haven't questioned. You're all right that that's where I see the problem manifest itself. In my case, I'm developing these [c]DNG files in Resolve, and that looks like where this (mis)interpretation is happening.

How are the rest of you managing, with dual-iso in Resolve, to hide flickering?

So, I’ve been doing it wrong all this time, by going into the (hdr?) submenu.

I’ try again, exactly as you suggest, by re-enabling dual-iso automation and running it , with no other options.

Thanks @Danne!

Hi @Danne, @DeafEyeJedi,

I hope neither of you think  I was criticising Switch? It's great, it works well, I can set it and walk away, to come back to processed files. Thank you for this software.

My trouble is the constant flickering - on almost every shot I've taken so far - and this is my search to find ways to fix it.

Using Exiftool (and exiv2), I can't see the Baseline Exposure Offset tag as having been set, on DNG (dual iso) files, processed by either Switch, or the mlv_dump + cr2hdr combination.

This is what I do see happening across 2 frames which exhibit flicker - the first of the two is subtly darker than the second one:

AppFrame 513Frame 514
mlv_dump (unprocessed)BL: 2047, WL: 16200BL: 2047, WL: 16200
cr2hdr (with --same-levels)BL: 8192, WL: 65128BL: 8192, WL: 65128
Switch (with (s) same-levels)BL: 8192, WL: 63768BL: 8192, WL: 63768
MLV App (can't remember deflicker target value)BL: 8188, WL: 64800BL: 8188, WL: 64800

As you can see from the above, all three approaches result in changed Black Level and White Level values, relative to the 'original', unprocessed .dng file (out of mlv_dump), and they all differ from each other. However, they are also all equal across frames, resulting in very visible flickering.

I second that @Wayne H, I just came over from the cr2hdr thread.

I tried a few random numbers in the 'Deflicker target' box, and can clearly see that the footage overall is darkened for raised values, however the flicker is still there. I can't find any documentation in your source code for this function, and would like to know how to determine the 'best' values to try, or what values have shown themselves as most effective?


Thanks @a1ex, I'll try mlv_dump on steroids. MLVFS seems random to me; sometimes it works, sometimes it crashes under the load, and it's hard to say which it's going to be until you've waited a long time.

I had no idea darktable had this feature, I must investigate. MLV App definitely has it, and it's an elegant and powerful app, but I can't find any documentation about how the flicker suppression works, though their 'Deflicker target' setting clearly makes a difference, making the footage darker overall, but I haven't been able to fix it altogether yet. I assumed it built off cr2hdr.

@OlRivrRat it's possible; without bumping into it again (and again), and working out the common factor triggering it, I just can't say.

As you can see, the damage to the frame is huge in the full-res LiveView, and extensive on the 3.5k one. I'll leave this be until I have more time to play with combinations of settings, with 'exotic' fps choices! If something similar comes up again, I'll report back.

Thanks all.

Uncheck dualiso automation? I thought I read that was there to disable dualiso processing, on a folder you know doesn't contain any dual iso MLVs?

Either way, as I wrote, I subsequently processed one MLV manually, using mlv_dump, then cr2hdr manually, as in the original post in this thread, using the --same-levels switch (no puns intended Danne).

With this procedure, I can confirm that white and black levels are stable, across a number of frames, including those where there is an obvious darkening and lightening in exposure - showing up as flicker. That's why I didn't report this as a Switch issue.

So, if the recent (last two years' worth) check-ins to cr2hdr.c have improved the flicker-suppressing algorithm, I'd like to compile and test the latest cr2hdr, but am running into compiling errors (3 posts ago).

Otherwise, it's something I'm doing that's throwing the algorithm off, as no-one else is reporting flickering in all their dual-iso videos.

Can someone help?

Thanks for the reply @Danne, I didn't know this.

I do all the post in Resolve, but have not ever changed the black level; I didn't even know about it. Do I set it to the value reported by exiftool?

Do you know where to set it in Resolve?

Sorry for the silly questions...

I only recently discovered dual iso, and as you've all been saying for years, it's nothing short of amazing.

My trouble is that in every video I shoot, there's loads of flickering - the exposure seems to change noticeably from frame to frame, making the video seem to 'flicker' when played at normal speed. I've read the entire thread, and haven't been able to find a working solution, apart from faulty black and white levels, which should be fixed by exiftool. I tested this, reading a few of the (cr2hdr) processed DNGs using exiftool, and can confirm that the white and black levels remain constant across frames, including those exhibiting the apparent change in exposure, at 8192 and 65128, respectively.

I suppose this can be changed by constantly altering the exposure in post, but it will be very laborious to have to do it throughout the length of each video, so I'm hoping the latest cr2hdr has an improved algorithm to detect these variations.

All footage is shot on a 5D3, on the October 27 4k_lossless branch build. At first, I processed it with Switch, then to confirm with the mlv_dump + cr2hdr combination, as documented on the original post in this thread, using the --same-levels argument.

I'm on macOS Sierra (10.12.6), and have tried the cr2hdr in the OP, which is quite old, if I remember, as well as the one from Danne's Switch, which is:

Code: [Select]
Last update: ab1e90c on 2015-11-24 09:50:19 UTC by a1ex:
cr2hdr: moved safeguard from median_int_wirth to kth_smallest_i..

As far as I can tell, there have been a number of changes to cr2hdr.c in the repo, which I've tried building, but end up with build errors:

Code: [Select]
make cr2hdr
/bin/sh: /Volumes/Data HD/Users/sacha/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3: No such file or directory
[ gcc      ]   cr2hdr
cr2hdr.c:59:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
cr2hdr.c:59:17: error: expected ';' after top level declarator
cr2hdr.c:573:51: error: use of undeclared identifier 'dual_iso_strings'
    printf("Last update: %s\n", module_get_string(dual_iso_strings, "Last update"));
cr2hdr.c:1334:46: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist rggb.m"));
cr2hdr.c:1334:46: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1472:46: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist bddb.m"));
cr2hdr.c:1472:46: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1748:51: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist iso-curve.m"));
cr2hdr.c:1748:51: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:2184:55: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist fullres-curve.m"));
cr2hdr.c:2184:55: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:2518:56: warning: if statement has empty body [-Wempty-body]
            if(system("dcraw -d -r 1 1 1 1 edges.dng"));
cr2hdr.c:2518:56: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:2531:59: warning: if statement has empty body [-Wempty-body]
            if(system("dcraw -d -r 1 1 1 1 edge-map.dng"));
cr2hdr.c:2531:59: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:2557:21: error: function definition is not allowed here
cr2hdr.c:2571:31: warning: implicit declaration of function 'edge_interp' is invalid in C99 [-Wimplicit-function-declaration]
                    int pi0 = edge_interp(dir);
cr2hdr.c:2824:51: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist mix-curve.m"));
cr2hdr.c:2824:51: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:3308:55: warning: if statement has empty body [-Wempty-body]
            if(system("octave --persist soft-film.m"));
cr2hdr.c:3308:55: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:3652:34: warning: if statement has empty body [-Wempty-body]
        if(system("octave wb.m"));
cr2hdr.c:3652:34: note: put the semicolon on a separate line to silence this warning
11 warnings and 3 errors generated.
dcraw-bridge.c:101:3: warning: unused variable 'unique' [-Wunused-const-variable]
} unique[] = {
1 warning generated.
amaze_demosaic_RT.c:983:10: warning: absolute value function 'fabsf' given an argument of type 'double' but has parameter of type 'float' which may cause truncation of value
                                        if (fabsf(0.5-hvwt[indx>>1])<fabsf(0.5-hvwtalt)) {hvwt[indx>>1]=hvwtalt;}//a better result was obtained from the neighbors
amaze_demosaic_RT.c:983:10: note: use function 'fabs' instead
                                        if (fabsf(0.5-hvwt[indx>>1])<fabsf(0.5-hvwtalt)) {hvwt[indx>>1]=hvwtalt;}//a better result was obtained from the neighbors
amaze_demosaic_RT.c:983:35: warning: absolute value function 'fabsf' given an argument of type 'double' but has parameter of type 'float' which may cause truncation of value
                                        if (fabsf(0.5-hvwt[indx>>1])<fabsf(0.5-hvwtalt)) {hvwt[indx>>1]=hvwtalt;}//a better result was obtained from the neighbors
amaze_demosaic_RT.c:983:35: note: use function 'fabs' instead
                                        if (fabsf(0.5-hvwt[indx>>1])<fabsf(0.5-hvwtalt)) {hvwt[indx>>1]=hvwtalt;}//a better result was obtained from the neighbors
amaze_demosaic_RT.c:1215:10: warning: absolute value function 'fabsf' given an argument of type 'double' but has parameter of type 'float' which may cause truncation of value
                                        if (fabsf(0.5-pmwt[indx>>1])<fabsf(0.5-hvwt[indx>>1]) )
amaze_demosaic_RT.c:1215:10: note: use function 'fabs' instead
                                        if (fabsf(0.5-pmwt[indx>>1])<fabsf(0.5-hvwt[indx>>1]) )
amaze_demosaic_RT.c:1215:35: warning: absolute value function 'fabsf' given an argument of type 'double' but has parameter of type 'float' which may cause truncation of value
                                        if (fabsf(0.5-pmwt[indx>>1])<fabsf(0.5-hvwt[indx>>1]) )
amaze_demosaic_RT.c:1215:35: note: use function 'fabs' instead
                                        if (fabsf(0.5-pmwt[indx>>1])<fabsf(0.5-hvwt[indx>>1]) )
4 warnings generated.
make: *** [cr2hdr] Error 1

Can anyone who's been able to build on the mac help with either the errors, or to share their build environment settings/dependencies? I have installed dcraw and exiftool, and Python's docutils (someone's build failed because of this dependency).

Hi @a1ex,

Yeah, reproducibility is the problem here. If it helps to 'read' the MLV file, all you need to do is ask.

But, that's not the end of troubles, sorry. I've just had time to process another MLV from the same batch, this time a full-resolution crop_rec mode MLV, at 5784x3248, 1 fps. The problem here is quite a bit worse; the usable image is only in the lower half of each DNG, with travelling colourful aberrations and horizontal lines in the top half (maybe top 60% of the image).

Another MLV from the entire batch fails to process completely, in Switch, mlv_dump and MLV App, but let's not even go there.

The only thing that I did between these shots was to change resolution and/or bit depth including from 14-bit lossless to 14-bit. Nothing 'seemed' out of the ordinary, apart from the [usual and expected] slow-down in Magic Lantern responsiveness, which is correlated to the selected fps.

One thing was annoying and unexpected, but not insurmountable: trying to get back into the Magic Lantern menu often failed, and pressing the 'bin' button would (two thirds of the time) instead of drawing the ML menu, bring up Canon's own LiveView HUD picture style selector. Waiting longer and trying again would sometimes work properly, other times it would leave the LiveView on, drawing only the bottom two-three menu items on top, without any of the rest of the ML menu.

Again, I've only experienced this now, and only through playing with different fps values. It never happens in non crop_rec mode or at the expected 23.976 or higher, (or lower, down to 4) fps.

So, this shoot - while only a test - went wrong in three different ways.

If you'd like to see any MLVs, just ask.


Hi a1ex,

I already had a conversation with you about vertical stripes - through the full width - in the liveview, and you explained that you know about it, that it's harmless, and that you'd need to work hard at removing it.

This report isn't about that, it's actually in the MLV file recorded in the 3.5k mode (with 5x magnification); the vertical stripes are recorded in the first 700 frames, only appear in the right half of the image's width, each of different colour and with random spacing. And, the only other similarity to the live view is that they fade towards the end of the 700 frames, visually disappearing from all further frames.

The reason I reported it is that: I've never experienced it before; I haven't read about it anywhere in the forums since you introduced the 4k_compressed branch; and as you're still treating the 4k_compressed branch as experimental and not stable, I thought you may want to know about edge pathological cases. Though, to be fair, since I don't know what would reproduce the behaviour, that does limit the usefulness of the report.  :-[


I've just bumped into a problem I don't remember anyone reporting on the crop_rec or lossless branch - I don't know exactly where this issue belongs, so could be posting to the wrong place.

Since it came out in April, I've been using the 4k losless branch, and have never come across this before: coloured vertical pinstripes across the right-hand side of the frame.

jpg to img

I'm using the crop_rec_4k.2017Oct28.5D3123 release, and was shoting in 3.5k crop_rec mode, with a resolution of 3584x1320 at 1 fps. This is shot on a pinhole lens, with a raised exposure of 1/30, ISO 800. It's shot in 14bpp lossless.

The MLV processing was done in the latest version of Switch, using its mlv_dump option, which I imagine is the latest.

The vertical stripes persist for nearly the first 700 frames, towards the end of this range getting more sparse and faint. Around frame 700, they are no longer visible, and the footage appears normal and artefact-free.

The MLV file is 8.4 Gb, and even if I cut it down to only contain the first 800 frames, it will still be very large to upload, though probably small enough to send by WeTransfer, if looking at it helps?



Yes it must be (an issue with zsh). I can't see why without testing, since all of the commands on those lines work independently. Perhaps the problem is in the piping, routing or in character escaping. In either case, I'm not wedded to zsh, so it's a non-issue.

I've run a complete test on a Dual-ISO file (interlaced), and Switch works just fine, creating the expected .dng files, then going over them all, applying a1ex's algorithm and renaming them to .DNG files.

The only trouble I've run into is where the process fails, and it's whenever the camera is pointed at a very bright light source. Then the files are left untouched. I suspect this is no different to the camera going into panic when in 14-bit lossless, at ISO 100, and it's pointed at a very bright scene which it can't adequately compress.

Thanks again for the quick help, Danne.

Hi Danne,

It's the exact same error if I drag into Switch.

Commenting out the line brings me to another error further down (another tee command):

Code: [Select]
/Applications/ ; exit;
ls: /tmp/DUALISO/crop_rec?: No such file or directory
ls: /tmp/DUALISO/RAW_demolish: No such file or directory
ls: /tmp/DUALISO/what_cam_lock: No such file or directory
/Applications/ line 2751: syntax error near unexpected token `>'
/Applications/ line 2751: `    exec &> >(tee -a "$(cat /tmp/DUALISO/samples | head -1 | cut -d "." -f1)"_samples/"$(cat /tmp/DUALISO/samples | head -1 | cut -d "." -f1)"_LOG.txt >&2 )'

[Process completed]

Commenting out line 2751 does actually take me to the menu, where everything appears to work - I'm testing to see if there are any obvious problems.

And, yes, very good question, I've changed my shell to zsh, I haven't tried it in Bash and expect that it will work since you've tested exclusively in Bash, right?


I have a problem running Switch. I'm running it on macOS Sierra, installed by just moving it to /Applications as directed. As far as I can tell from the original post on the thread, there are no othe

Running it, I get the pop-up window asking for the source folder, as expected. then this:

Code: [Select]
ls: /tmp/DUALISO/crop_rec?: No such file or directory
ls: /tmp/DUALISO/RAW_demolish: No such file or directory
ls: /tmp/DUALISO/what_cam_lock: No such file or directory
/Applications/ line 1976: syntax error near unexpected token `>'
/Applications/ line 1976: `    exec &> >(tee -a "$(cat /tmp/DUALISO/samples | head -1 | cut -d "." -f1)"_samples/"$(cat /tmp/DUALISO/samples | head -1 | cut -d "." -f1)"_LOG.txt >&2 )'

[Process completed] well as a forever-spinning gear icon in the menubar, displaying as 80% completed.

Needless to say, there is no crop_rec?, RAW_demolish, what_cam_lock or samples in /tmp/DUALISO/.

What there is, is:

Code: [Select]
ls -l /tmp/DUALISO
total 32
-rw-r--r--  1 sacha  wheel  26 11 Nov 12:21 DBG_path
-rw-r--r--  1 sacha  wheel   1 11 Nov 12:21 DUALISO
-rw-r--r--  1 sacha  wheel   0 11 Nov 12:21 MOD
-rw-r--r--  1 sacha  wheel   0 11 Nov 12:21 fpss
-rw-r--r--  1 sacha  wheel   0 11 Nov 12:21 image_size
-rw-r--r--  1 sacha  wheel  32 11 Nov 12:21 path_1
-rw-r--r--  1 sacha  wheel  35 11 Nov 12:21 path_2
-rw-r--r--  1 sacha  wheel   0 11 Nov 12:21 what_cam.txt

To be honest, I haven't tried to debug the Menu.command file, but I ask if anyone has had this problem, before I try to read through the file looking for where it fails.


Pages: [1] 2