Messages

Use raw zebras. Solid color appear where both ISOs are clipped, weak zebras appear where only one ISO is clipped (half-res) and no zebras appear where the scene is exposed properly.

I assume that by 'solid color' you mean black zebras?

By this I would expect black zebras when clipping at 100/100, since both ISOs must be clipped. But that's not the case, as the initial zebras are green.

In either case, I can't corroborate the statement through experimentation. Here is my repro:

1) Set Dual ISO to 100/100
2) Open iris until first zebras appear
3) Close iris 1/3 stop (zebras disappear)
4) Set Dual ISO to 100/800
5) Expect: zebras to appear, since the ISO 800 lines must now be clipping
6) OBSERVE: no zebras appear.

I used ACR to debayer.

Here's the original: http://popspring.com/drop/M21-1222_00038_interlace.dng

Do you have a DNG frame that I could diagnose?

http://popspring.com/drop/M21-1222_00038.dng

If the high-iso data is not clipped, it's called "midtone" and rendered in full resolution.

So to accomplish full-res highlights, do I, say, run DualISO at 100/800 and expose to the right minus 1.2EV?

Or if I run 800/100 and expose to the right, are high-ISO highlights automatically preserved?

As an example, I'd think the aliasing in #1 could look more like #2, if highlights were not half-res. And #3 could be clean.

I'll need to re-read your paper to understand why resolution must be sacrificed at both highlights and shadows. I assumed that if the high-ISO highlights are not clipped, then they could be kept, rather than replaced with line-doubled low-ISO highlights.

Forgive me if this has already been discussed.

Dual ISO is not useful to me for video due to the crawling aliasing I see at boundaries between 1/2 res highlights and mid/shadows, or 1/2 res shadows and mids/highlights  - when the camera or scene is in motion.

The question is whether it's possible to perform dual ISO in shadows only, leaving highlights at full resolution? I figure that this would sacrifice 1/2 the DR benefit, but I hypothesize that my main objection for dual ISO for video would be improved in that there would be no aliasing between highlights and mids, but only between 1/2 res shadows and highlights/mids.

This is based on my assumptions that the aliasing between 1/2 res highlights and 1/2 res shadows is the most noticeable and disagreeable, and that aliasing between 1/2 res highlights and mids is more noticeable that aliasing between 1/2 res shadows and mids.

[EDIT] Or do I accomplish this myself by setting Dual ISO to, say, 2.4EV DR gain (100/800) and underexposing by -1.2EV?

New XCode doesn't include GCC anymore and uses Clang in its place, which doesn't recognize this GCC option, but it's not needed either, so you can safely remove it from Makefile.cr2hdr.

It builds with that change. Thanks.

Any idea the issue here?

falconmacbook::steve [modules]$ make

Building module raw_rec...

Building module mlv_rec...

Building module mlv_play...

Building module mlv_snd...

Building module file_man...

Building module dual_iso...
REBUILDING
[ README   ]   module_strings.h
[ CC       ]   dual_iso.o
[ MODULE   ]   dual_iso.mo
[ STRIP    ]   dual_iso.mo
[ EXPORTS  ]   dual_iso.sym
00001368 dual_iso_calc_dr_improvement
000013c0 dual_iso_is_enabled
000013e0 dual_iso_get_dr_improvement
00001560 dual_iso_get_recovery_iso
00001580 dual_iso_set_recovery_iso
[ DEPENDS  ]   dual_iso.dep
Will load on:
    5D3
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
[ gcc      ]   cr2hdr
clang: error: unknown argument: '-mno-ms-bitfields' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
make[3]: *** [cr2hdr] Error 1

********************************************************
WARNING: module dual_iso failed to build, deleting
********************************************************

[ RM ]  dual_iso.o dual_iso.mo dual_iso.sym dual_iso.dep module_strings.h *.o *.d *.dep *.sym hgstamp
[ RM ]  cr2hdr cr2hdr.exe

After my initial attempt, I used the latest Makefile.user.default (with my two changes) and still got the location.map error. It's only after I re-cloned could I build. But yes, I am now using a thin makefile.user.

It's a little dissonant because Makefile.user.default isn't really what the default would be according to this - empty.

I re-cloned the project and it works now. I don't know what was wrong.

Yep, did that...

I did have that line, but removing it results in other errors:

falconmacbook::steve [magic-lantern]$ make
make -C  /Users/steve/magic-lantern/platform/5D3.113
[ VERSION  ]   ../../platform/5D3.113/version.bin
[ VERSION  ]   ../../platform/5D3.113/version.c
[ CC       ]   version.o
make -C ../../tcc
make[2]: Nothing to be done for `all'.
[ LD       ]   magiclantern
../../tcc/libtccx.o: In function `start':
crtstuff.c:(.text+0x180): undefined reference to `main'
crtstuff.c:(.text+0x198): undefined reference to `__bss_start__'
crtstuff.c:(.text+0x19c): undefined reference to `__bss_end__'
../../tcc/libtccx.o: In function `exit':
crtstuff.c:(.text.exit+0x2c): undefined reference to `_exit'
../../tcc/libtccx.o: In function `_fstat_r':
crtstuff.c:(.text._fstat_r+0x1c): undefined reference to `_fstat'
../../tcc/libtccx.o: In function `_open_r':
crtstuff.c:(.text._open_r+0x20): undefined reference to `_open'
../../tcc/libtccx.o: In function `_sbrk_r':
crtstuff.c:(.text._sbrk_r+0x18): undefined reference to `_sbrk'
../../tcc/libtccx.o: In function `_write_r':
crtstuff.c:(.text._write_r+0x20): undefined reference to `_write'
../../tcc/libtccx.o: In function `_close_r':
crtstuff.c:(.text._close_r+0x18): undefined reference to `_close'
../../tcc/libtccx.o: In function `_isatty_r':
crtstuff.c:(.text._isatty_r+0x18): undefined reference to `_isatty'
../../tcc/libtccx.o: In function `_lseek_r':
crtstuff.c:(.text._lseek_r+0x20): undefined reference to `_lseek'
../../tcc/libtccx.o: In function `_read_r':
crtstuff.c:(.text._read_r+0x20): undefined reference to `_read'
make[1]: *** [magiclantern] Error 1
make: *** [5D3] Error 2

I'm now running 4.8.3 and I get the same error:

falconmacbook::steve [magic-lantern]$ make
make -C  /Users/steve/magic-lantern/platform/5D3.113
[ VERSION  ]   ../../platform/5D3.113/version.bin
[ VERSION  ]   ../../platform/5D3.113/version.c
[ CC       ]   version.o
make -C ../../tcc
make[2]: Nothing to be done for `all'.
[ LD       ]   magiclantern
arm-none-eabi-gcc-4.8.3: error: unrecognized command line option '-Map=location.map'
make[1]: *** [magiclantern] Error 1
make: *** [5D3] Error 2

Thanks.

Updating gcc? Which version and how do I get it? Sorry, this build environment stuff is an utter mystery to me.

falconmacbook::steve [magic-lantern]$ make V=1
make -C  /Users/steve/magic-lantern/platform/5D3.113
echo -e bbbbMagic Lantern v2.3.NEXT.2014Mar14.5D3113
echo "const char build_version[] = \"v2.3.NEXT.2014Mar14.5D3113\";"
~/mltools/gcc/bin/arm-none-eabi-gcc-4.7.3  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./version.d -Wp,-MT,version.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fsingle-precision-constant -fshort-double -fno-builtin-printf -std=gnu99 -Winline -I../../platform/5D3.113 -I../../platform/5D3.113/include -I../../src -I../../picoc  -DCONFIG_5D3_MINIMAL=1 -DCONFIG_NO_SNAP_SIM=1 -DCONFIG_MAGICLANTERN=1 -DCONFIG_5D3=1 -DRESTARTSTART=0x00069500 -DROMBASEADDR=0xFF0C0000 -DVERSION=\"v2.3.NEXT.2014Mar14.5D3113\" -DCAMERA_MODEL=\"5D3\" -DCONFIG_DEBUGMSG=0 -DCONFIG_FW_VERSION=113 -DCONFIG_CONSOLE -DCONFIG_TCC -DCONFIG_MODULES -DCONFIG_MODULES_MODEL_SYM=\"5D3_113.sym\" -c -o version.o ../../platform/5D3.113/version.c
make -C ../../tcc
make[2]: Nothing to be done for `all'.
~/mltools/gcc/bin/arm-none-eabi-gcc-4.7.3 -o magiclantern -N -u strcasecmp  -T magiclantern.lds entry.o ../../platform/5D3.113/stubs.o version.o boot-hack.o fio-ml.o mem.o ico.o edmac.o menu.o debug.o rand.o posix.o util.o electronic_level.o cfn.o gui.o picstyle.o exmem.o bmp.o rbf_font.o config.o stdio.o bitrate-5d3.o lcdsensor.o tweaks.o tweaks-eyefi.o lens.o property.o propvalues.o gui-common.o chdk-gui_draw.o movtweaks.o menuhelp.o menuindex.o focus.o notify_box.o dialog_test.o vram.o aj_port.o fps-engio.o shoot.o hdr.o lv-img-engio.o state-object.o tasks.o vsync-lite.o tskmon.o battery.o imgconv.o histogram.o falsecolor.o audio-ak.o zebra.o vectorscope.o beep.o crop-mode-hack.o ph_info_disp.o flexinfo.o screenshot.o fileprefix.o lvinfo.o ml-cbr.o console.o tcc-glue.o ../../tcc/libtccx.o module.o video_hacks.o misc.o afma.o raw.o chdk-dng.o edmac-memcpy.o asm.o dietlibc.a newlib-libc.a newlib-libm.a gcc-libgcc.a -Map=location.map
arm-none-eabi-gcc-4.7.3: error: unrecognized command line option '-Map=location.map'
make[1]: *** [magiclantern] Error 1
make: *** [5D3] Error 2

#User-defined values for ML build system
#Copy me to Makefile.user, keeping ONLY the options that you want to customize

UNAME:=$(shell uname)

ifeq ($(UNAME), Darwin)
   # Variable declaration for Mac OS X
   UMOUNT=hdiutil unmount
   CF_CARD="/Volumes/EOS_DIGITAL"
   STAT_CMD=stat -f "%N: %z bytes"
else
   # Default settings for remaining operating systems
   UMOUNT=umount
   CF_CARD="/media/EOS_DIGITAL/"
   STAT_CMD=stat -c "%n: %s bytes"
endif

# Supported camera models
SUPPORTED_MODELS:=5D3

# GCC-related stuff
# for yagarto or the official ARM toolchain use ARM_ABI=none-eabi otherwise use ARM_ABI=elf
ARM_ABI=none-eabi
GCC_VERSION=-4.7.3
ARM_PATH=~/mltools/gcc
ARM_BINPATH=$(ARM_PATH)/bin

CROSS_COMPILE=$(ARM_BINPATH)/arm-$(ARM_ABI)-

ifeq ($(OS),Windows_NT)
CC=$(CROSS_COMPILE)gcc$(GCC_VERSION).exe
else
CC=$(CROSS_COMPILE)gcc$(GCC_VERSION)
endif

LD=$(CC)
OBJCOPY=$(CROSS_COMPILE)objcopy
OBJDUMP=$(CROSS_COMPILE)objdump
AR=$(CROSS_COMPILE)ar
RANLIB=$(CROSS_COMPILE)ranlib
READELF=$(CROSS_COMPILE)readelf
LD_REAL=$(CROSS_COMPILE)ld
STRIP=$(CROSS_COMPILE)strip

# You can customize CFLAG here
HOST_CFLAGS=-g -O3 -W -Wall
#CFLAG_USER = -march=armv5te -mcpu=arm946e-s -O2 -DCONFIG_RELEASE_BUILD \

# Additional binaries
HOST_CC=which gcc
CCACHE=ccache
GREP=grep
AWK=awk
PYTHON=python2
RST2LATEX=rst2latex.py

NIGHTLY_ROOT:=$(HOME)/public_html/nightly
NIGHTLY_ARCHIVE:=$(NIGHTLY_ROOT)/archive
NIGHTLY_DIR:=$(NIGHTLY_ARCHIVE)/$(shell date +'%Y-%m-%d')-$(shell hg id -i -r .)

# Naming convention for Magic Lantern builds:
# General rules:
# - Always specify the camera and its firmware version number in the build name (e.g. 550d.fw109)
# - For non-release builds, specify the build date and author's (nick)name.
# - For experimental builds, add a short keyword indicating the particular feature tested.

# Examples for experimental builds:
# magiclantern-2010dec07.550d.fw108.cropmarks.a1ex.zip
# magiclantern-2010nov23.550d.fw108.selectable-audio.piers.zip

# Example for pre-release builds:
# magiclantern-2010dec17.550d.fw109.PRERELEASE.alex.zip

# Release builds:
# magiclantern-0.2.0.rc1.550d.fw109.zip
#~ VERSION=0.2.0.rc1.550d.fw109
BUILDVER=$(shell whoami).$(shell hg id -i -r .)
VERSION:=v2.3.NEXT.$(shell LC_TIME=EN date +'%Y%b%d').$(MODEL)$(FW_VERSION)

# Build configuration - untested and debug stuff handy for devs
CONFIG_PYMITE       = n
ifndef CONFIG_RELOC
CONFIG_RELOC        = n
endif
CONFIG_TIMECODE     = n
CONFIG_PTP          = n
CONFIG_PTP_CHDK     = n
CONFIG_PTP_ML       = n
ifndef CONFIG_GDB
CONFIG_GDB          = n
endif
CONFIG_GDBSTUB      = n
ifndef CONFIG_CONSOLE
CONFIG_CONSOLE      = y
endif
CONFIG_DEBUGMSG     = 0
CONFIG_CCACHE       = n
ifndef CONFIG_PICOC
CONFIG_PICOC        = n
endif
CONFIG_TCC          = y
CONFIG_MEMCHECK     = n
CONFIG_MODULES      = y
CONFIG_TINYPY       = n

# Work in progress - would be nice to run the unmodified autoexec.bin in QEMU, but we are not there yet
# so, for now, we'll just hack the source code to work around the limitations of the emulator
CONFIG_QEMU         = n

Any idea?

falconmacbook::steve [magic-lantern]$ make
make -C  /Users/steve/magic-lantern/platform/5D3.113
[ VERSION  ]   ../../platform/5D3.113/version.bin
[ VERSION  ]   ../../platform/5D3.113/version.c
[ CC       ]   version.o
make -C ../../tcc
make[2]: Nothing to be done for `all'.
[ LD       ]   magiclantern
arm-none-eabi-gcc-4.7.3: error: unrecognized command line option '-Map=location.map'
make[1]: *** [magiclantern] Error 1
make: *** [5D3] Error 2

With RAW zebras ON I get a kind of zebra "snow" in seemingly random places, and nothing in some places where
I think zebras should show.  Also I use an external monitor and HDMI has issues with coordinates being off.

I didn't know about UniWB. I'll try it.

Since RAW zebras don't work (for me), does anyone know a rule of thumb for using the regular zebras while recording under rawrec? For example, if I knew that raw video clipped 1 stop over where normal zebras kicked in, I could simply open 1 more stop from there.

You might "see" a worse aliasing while working within resolve and looking at its color tab window and enlarging it. However when you export the file and open that file in after effects and put it side by side, they are the same.

Good to know. Of course it's nice to see what you're going to get before exporting, especially when working with blur/sharpening.

To my eye, Adobe Camera Raw debayering looks better than Resolve's.

Ah, you should have linked it from the beginning. I thought it's about today's findings.

It can run in camera, but expect around 10 seconds (maybe more) for one picture.

Yeah sorry, I just read the whole thread for the first time today. I thought 'FPN' captured it.

So I assume your idea was for an post tool addition. I'm pitching the case for video. Banding is bad, but it's really bad when the bands crawl - the motion catches your eye, and you really see them.

Maybe I've described it wrong. I'm talking about these mentions:

On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.

I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.

The banding pattern accounts for only a small part of these histogram distortions.

Proof:

ISO 100 dark frame (5D3), original and corrected


Histograms of horizontal and vertical banding correction (per-line and per-column)


Image histogram before and after correction:


How I've estimated the correction:

Code Select Expand


# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im);  fv = fv - mean(fv);

# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png

# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;

# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png


I took that as an idea for new software to de-pattern shadow noise.

Is the FPN filter mentioned earlier something that would run in-camera, or post? If in-camera, is it possible that it could be used in video mode, even if the ISO tweaks couldn't?

Making that deep shadow noise prettier would be big plus for video, especially given that the bands crawl, in my experience.