Messages

I install the last firmware (magiclantern-Nightly.2015Aug18.6D116) and didn't find the "silent picture" function. It isn't present now?

You did venture to enable the "silent" module :-> ?

90MBit/s not 90MB/s...
Different things, 8 times different
90MBit/s = 11MB/s
Easy to handle for a 20MB/s card interface...and the 6d has even a 40MB/s card interface

Right, thanks for correcting me, mean of you to use strange denominations lime MBit/s for the simple people over here :-p ... and sorry everyone for spreading wrong information.

It was because my card did block when using all-i frames, but in hindsight that was when it was nearly full and probably badly fragmented - so in these cases it probably pays to have a card spec'ed far beyond the interface limit so it'll keep the speed until the very end.

Made a pull request for that

I approved it on bitbucket just for the enthusiasm of yours :-) ... but my advice from experience is to open a thread in the general dev forum and discuss promoting cygwin to a "supported" compilation environment. Imho there are good reasons to use cygwin on windows instead of a Linux vm, and it lowers the barrier for quick compilation and patching.

Do note: if the main devs don't really want this, your pull requests will have little luck and you're in for much frustration and a lifetime of cygwin patching as every new code addition will break cygwin again. Better get everyone to agree to take care of the little cygwin needs in the first place (other like for example throwing the arm gcc compiler in the home dir as the only option).

I didn't work on cross Now comes the waiting game.

Well, as you see this could take months atm so if you've got time on your hands make sure this pull requests covers full cywin/win-mingw support for everything (including make zip & make docs). I'd advise to use the cygwin version of the tool required (tex, mercurial, ...) to prevent script bugs because of different unix/win paths.

I didn't work on cross compiling dng2raw because I'm not really sure what it does and all my attempts at cross compiling it resulted in errors so I left it pretty much alone.

I didn't look at that, but that's the very reason for the existence of Cygwin (as msys doesn't have a recent compiler for the emulation environment): some sources need *nix POSIX functionality (like gettext) that Windows simply lacks. Some things can be done with little patches like an additional include line or a more precise var type, but some code would need to be completely rewritten so it's simpler just to use the emulation and be done with it.

The 6d has the ALL-i option in canon menu, 90mbit/s all-i h264 compression.

To add another random comments: You have to give it to Canon, adding a 90mb/s video mode to a cam equipped with a 20mb/s card interface and a very small buffer nominates them for the "most useless specs" award :->

I also don't shoot subjects that would require encryption in case I get captured and tortured to show my photos. (Usually it is the other way around.)

:-p ... The more probable real-life situation is that you shoot an event, and the organizer pressures you to "just give him your shots" which a pro photog would never do. If your files are encrypted, you can just say "be my guest" and state that you have no idea why his whatever-he's-using-raw-converter doesn't work :->

EC in M mode? Had to look look it up in my ML to English dictionary. The autoexposure module is something else I haven't played around with so I turned it on in my camera and whoa!

My (now private) auto_iso modules also contains ec in m, it's very useful if shoot in changing lighting conditions and still want to pre-set the exposure (esp. for flash or capturing motion) and aperture (depth of field). Nikon has it, and Canon graced their 1dx with this feature in a recent fw update, but other Canon cams are too dumb.

Anyway, it's just an example when something nice doesn't quite work due to the limited Canon fw - the EC indicator in M mode is just made for rough visual confirmation of over/underexposure, not an exact or fast meter you can use reliably by ML code ... we'd need direct hardware access to the metering system for that.

You mean something like this?

Yeah, something like this ... here's the end of my current "just works" compile script. I see no sense to use make zip locally as I instantly want to use the files, thus I copy the stuff over a (recent) official nightly version in case I missed some files and then copy everything to the card (if it's available). You'd need to add that part to yours and use a variable for my hardcoded 6d.116. The "copy all modules" line should be of special interest, took me a while to get that shell line right :-p

Code Select Expand


mkdir -p ../dist/ML/modules
rm -f ../dist/ML/modules/*.mo
mv platform/6D.116/autoexec.bin ../dist/
mv platform/6D.116/6D_116.sym ../dist/ML/modules/
cp modules/*/*.mo ../dist/ML/modules
cp modules/dual_iso/*.exe ../dist/ML/
cp modules/raw_rec/*.exe ../dist/ML/
cp modules/mlv_rec/*.exe ../dist/ML/
cp modules/io_crypt/*.exe ../dist/ML/
mkdir -p ../dist/ML/fonts
cp data/fonts/* ../dist/ML/fonts


By the way, you can brick a camera with a lua script. Take a closer look at dmilligan's code

Ugh, I didn't realize and thought there were some additional safeguards with lua. So note: When in doubt, use new ML code in a C mode :-> ... beats having a heart attack every time your camera not turning itself on (mostly b/c the battery is empty) :-o

If I ever run into g3gg0 or a1ex I may just fall to my knees and scream, "I'm not worthy!"

These two basically carry the ml development, and finding devs with this ability and time on their hands is the reason why there's always the danger of the trunk getting stalled. Lucky us ml is working ok right now and essential features (like dual_iso & raw video) are production ready.

As for frustration with development: You might be lucky and and everything works as you want to, but there's always the possibility the Canon fw doesn't play ball (for example b/c a button doesn't send an event code) and you end up with endless sessions of trial & error, swapping the card between computer and camera. And sometimes, you have to realize it won't work at all properly - like reading the exposure from EC in M mode, w/o direct hardware access it'll be always dodgy.

Remember that if you do shoot raw, your post work will take longer as there are a few extra steps you have to take before you can start editing raw the footage.

The upside is that the other parts of the editing workflow are finished much quicker - because you can just record a few seconds of consecutive raw video on the 6d :-> (well, in a somehow reasonable resolution that is).

One final thought is that if someone is using this approach: would they be able to video a demo for us non-coders who are trying to get into all this 'coding stuff'.

Imho the simplest way is to provide a bash shell script that people just would need to run after installing (or unzipping) cygwin - this would automatically download the latest ml code, copy a working Makefile user, compile everything and then throw the autoexec.bin & modules .mo into a folder. People would just have to look at this shell script to understand how it works.

You don't really need much in the way of coding skills to contribute.

Personally, I'd advise caution to prevent frustration. If you can code C and know the basics of distributed software development (i.e. Mercurial) then it's easy to get started.

But if you want to change non-trivial things a lot of learning about how the Canon fw and the ML code works is involved, and there are always things that simply don't work as you expect because ML depends on what Canon provides. And for trivial changes, in the future you can just use user scripting with lua and use ML's lua api which has the sandbox advantage of not being able to brick the camera.

Last not least, real embedded hardware hacking is another matter altogether and far beyond my skills - an example would be the "new sound system" branch.

Hopefully that will be changed when the 20-bit version of cr2hdr is merged. Then maybe we can have it use the 64-bit compiler if possible?

The 20bit version doesn't compile with x64 either, and alex has stated that this is known and most likely "wontfix" because the code changes are not worth the possible benefits.

There is no end to this is there?

You probably spend a lot of time figuring out msys and mingw :-) ... but actually cygwin works quite nicely recently, it used to be much worse. Basically it's just the compiler paths for the module exe helpers and some other smallish stuff. Given the nearly stalled ml development, I don't expect any major ml build system changes soon that would break cygwin again.

And in any case, very few people will hack the source themselves and the need for "compile it yourself" is basically gone with the nightly builds. So the one real use for compiling ml is the integration of some branch or pull request you cannot wait for to be merged (like nss audio on 6d), and again I doubt many people will do this. The download counter for my former cygwin zip package was extremely low, go figure.

When you get to cross compiling the command line tools some tools look for the MinGW gcc compiler in the home directory so we need to create a directory structure and make a symbolic link to that compiler in there:

The much better option would be to modify all exe helper Makefiles to use one central compiler path that is found in the Makefile.user ... symlinking stuff works, but I don't like the approach.

Here's a hack to try out--trick the system to use the 64-bit gcc compiler instead of the 32-bit one. Please leave feedback so we know whether or not this works and on which tools.

The issue is not setting up the mingw windows compiler to x64 (simply use x86_64-w64-mingw32-gcc.exe instead of i686-w64-mingw32-gcc.exe) but at least cr2hdr definitely doesn't work with 64bit integers ... a x64 c compiler isn't a drop-in replacement for x86 but often needs adapted code. As there's really little reason to have the exe helpers as x64 (well, cr2hdr might be a bit faster (or not)) imho just stick with x86 to prevent platform-specific bugs to confuse debugging ml even further.

Wow, look what I found

I saw these command line options, too, but didn't have much luck with them - for example I was unable to find a way to update all packages from the "exp" repo w/o going through the gui, essentially what you can do in any Linux distro with rpm or apt-get. It seems the cygwin installer isn't optimized for command line use, but feel free to try the options and post suggestions to the cygwin mailing list - as cygwin is actively maintained there's a good chance any reasonable requests might be incorporated quickly.

Seriously, I was thinking the same thing, deleting all the extra packages until it was down to the bare essentials to compile ML but then isn't it a problem updating the installation? It would also take lots of time and testing to make sure nothing breaks.

No problem there, if you de-select all packages on installation and then carefully select only those required for ML (some trial and error is involved) the cygwin installer takes care of the (minimal) dependencies and thus no problem on updating exists. You might want to de-select all manual/docs packes and even delete the /share/doc folder to save space - on installation the docs just get installed again.

If you found my old posts, there is even a simple script included that pulls the ml repo, copies Makefile.user and compiles the whole thing - I don't quite remember how elaborate it was.

Last not least, the docs do work - I got it to work using the cygwin tex, but you have to get all dependencies right and it needs a lot of disk space so including it into a "bare bones" zip package might not be feasible.

Glad to be of service :-) and I hope cygwin will be recognized and maintained as a viable way to quickly compile a full ML (including docs and all) under windows with minimal disc space requirements, i.e. w/o running a full Linux vm. The basic compiler (arm gcc from launchpad) should give the same results in both Linux and Windows (though the newest 4.9 version generates buggy ML autoexec.bin for many cams for unknown reasons).

Note that most module exe helpers are targeted for x86 code, i.e. even if you run cygwin x64 (which you should on a x64 Windows) you need to use the i686 mingw toolchain.

* Btw you should select the "exp" radio button when running the cygwin setup using the cygports packages, this way you'll get more recent versions of the packages. You can always select older individual packages by switching through the versions, but the official cygwin policy is rather conservative so the non-"exp" repo often contains rather old versions.

* Fyi the one disadvantage of the Windows kernel is the lack of fork(), for example this results in a crawling automake ./configure so in these cases running a Linux vm is actually way quicker than the windows cygwin emulation layer - but this doesn't concern ML.

* Suggestion: Some time ago I posted a 7zip of a cygwin setup dir with all the required packages for ML already added and a working Makefile.user (cygwin is portable by default, just drop the unpacked dir anywhere). Amazingly, the required size was only a few megs. It just needed the launchpad gcc dropped in the respective dir. Maybe someone could refresh this, it would spare people going through the motions of selecting the packages by themselves as I don't know of any method to feed a list of packages to the cygwin installer.

Many have tried and failed -- and so have I but this is soooo close.

Yeah, getting mingw to work reminds me of being a donkey chasing after a carrot hanging from a stick in front of his nose :-)

Just one comment: There's a big distinction between msys/cygwin-"mingw" and "native" mingw without any emulation layer. If you're downloading "mingw" from somewhere, you're probably also running bash which automatically means you're using msys - which in turn is an outdated cygwin fork. If you do so, imho it doesn't make a lot of sense to use a windows-native make and so on, but you can just stick with using the unix toolchain with the emulation layer. In this case, only dedicated programs which don't benefit from the emulation layer will be "native" windows - for example like the arm gcc compiler from launchpad.

As already commented in one of your pull requests, feel free to get a "native" mingw build system running, but there's little (newsgroup) support for it, and you'll certainly end in "line-ending"-hell and "path separator / \"-hell sharing scripts in both native windows and cygwin/msys-emulation. For the average user, getting cygwin compilation working (it nearly does except for some modules) will be the quickest way to windows-based success as cygwin has a nice installer, pulling/updating all dependencies and preventing problems from sharing win tools (mercurial, latex, ...) with the unix-ish ml build system. In any case the autoexec.bin and .exe tools will be the same as the compiler is native win32.

The Windows version is way behind the Mac/Linux version and does not handle WB correctly.

Doh - any chance to get the win version updated as the webdav solution isn't available anymore (and less straight forward than mounting) ... or are the code bases so different the win version will lag behind?

I believe cr2hdr recomputes the WB, b/c the processing it does has an effect on WB (I could be wrong on this point).

I dunno if it's supposed to recompute wb, but it definitely has an effect on wb at least with my cams (60d, 6d) so with the adobe raw converter I usually have to do some manual tint/temperature adjustments.

@Marsu42: You trying to lossy compress with Adobe unprocessed Dual ISO? Why? I've read that lossy DNG compressed files no more Raw, it's demosaiced data. Demosaicing algorithm didn't expect to meet Dual ISO raw - this why artifacts, no?

My understanding that a dual_iso file should be the same as a vanilla raw after it's reconstructed by cr2hdr. Adobe supports these 16bit raw files in ACR (Lightroom, Photoshop, DNG Converter), and I see no reason why lossy compression shouldn't work as well - though of course you can never be sure with closed source software.

It's an important feature for Adobe apps - smart previews enable you to do faster and offline editing, and lossy dng is great for archival of less important shots w/o resorting to jpeg. Btw the "standard" preview in Lightroom is broken as well with the same artifacts, seems to use the same compression algorithm.

On the bitbucket bug tracker, dmilligan guesses that this might have to do with the non-standard black level dual_iso files. I cannot say and I'm unable to change cr2hdr to test this, but lossy dng *does* work fine with mini_mo files which also have a custom black level (but break dxo's raw converter).

I'm finding the dual ISO files a bit greener. 5D mark III 1.2.3 but build number doesn't matter because it's been all the way since I started using Dual ISO. Anyone notice this in the DNGs? WB is odd.

I'm on 6d/60d, and I consider wb "more than odd". I cannot verify the theory that dual_iso and vanilla should just look alike, just with dual_iso having cleaner shadows. My current method is to let Lightroom figure it out with the "auto" setting on import which works better than the current cr2hdr detection attempts for guessing auto wb. The "auto" setting also gives you a hint in which way to correct files with a fixed wb setting from the camera.

Having said that, hdr pictures somewhat do look a bit odd in the shadows anyway (esp. concerning green/magenta tint), so that might be one cause of confusion and bogus bug reports.

Fyi all in spite of cr2hdr offering lossy dng compression this doesn't quite work (for me). I've created a bug ticket: https://bitbucket.org/hudson/magic-lantern/issues/2333/cr2hdr-breaks-lossy-dng-aka-smart-previews

It's known that dual_iso dng files cause artifacts when converted to lossy dng in the regions of deep shadows that have been interpolated as half the the scanlines were clipped in the original raw file.

Apart from just being a "missing dng feature" in comparison to vanilla raw files, this problem has become more severe in recent Adobe apps since it's the base for the offline "smart preview" files. The free dng converter can be used to convert dng to lossy w/o Photoshop or Lightroom, it's basically 8bit jpeg-derived compression with the added benefit of keeping lossless wb adjustment.

I hope there's a way to do a minor cr2hdr adjustment to circumvent this problem. Alas, as Adobe's compression algorithm is closed source it might be tricky to identify the exact cause of the problem, but imho it's well worth a try. The dng converter seems to erroneously identify and compress these specific regions, maybe adding some noise or similar could be added to prevent this from happening?

Here's a sample how the lossy dng artifacts usually look: https://bitbucket.org/hudson/magic-lantern/issues/attachments/2333/hudson/magic-lantern/1438442876.44/2333/buggy_lossy-dng.png

Unfortunately so far "no can do" anything advanced flash-related because the flash props in the Canon fw are volatile and only accessible/existent when the flash is active. It seems the flash part of the fw is a "module" probably programed by another dev team or copy/pasted between cameras, so it's not as integrated as the rest.

I'd love to do lotsa other flash automatization features, but so far the only thing that reliably works is what is already in the stock ML (internal flash fec)... any dev feel free to re-investigate and prove me wrong though.

4.9 also miscompiled 5D3 and M builds last time I checked, same symptoms, ie. the led stays on and nothing else happens and neither of them has Digic 4.

Doh - so someone has to sit down, look at the 4.8 -> 4.9 changes and try to track down the culprit by disabling new optimizations.

Btw I faintly remember compiling 60d with 4.9 some longer time ago, so the bug might have been introduced in a very recent 4.9 minor release - if I'm not mistaken going back some revisions on launchpad might be worth a try if anyone is desperate for a 4.9 compile (resulting in a bit smaller binary).

One thing I've never quite understood about the inner workings of the camera: How is it that we need these awful keypress simulations to get some things done? Isn't there some way to *directly* access whatever-that-key does, there has to be some entry point in the Canon fw we can simply call like for "metering and af start" - or am I getting this entirely wrong?

The current annoyance is that I'd like to engage *H (expo lock hold) with ML, but no button it is available has a gui event on the 6d - thus I cannot simulate it. Harrrrrrgnnn ...

... (background): this would be the solution to the braindead 6d af-pt-biased eval metering, the idea is to auto-expo-lock whenever engaging the af with halfshutter. The "dual button method" with simple hold on halfshutter and backbutton af is too slow and cumbersome for quick shooting.

This works, but annoyingly you have to add a complete fork for every single pull request, even minor changes. The method to prevent this is to create a branch from the trunk of your fork, and then do a pull request from there.

1. create a 'fork' of the ML repository (you can do this from bitbucket)
2. 'clone' your fork to your local computer
3. make some changes
4. 'commit' the changes you made
5. 'push' the changes back to your fork
6. create a 'pull request' on bitbucket from your fork to the main repo

Ok, here you go if you want/need audio: UNSUPPORTED 6D NSS TEST RELEASE

1. Download official 6d nightly, unpack & install: https://builds.magiclantern.fm/
2. Delete /ML/modules/ directory to be on the safe side
3. Download my 6d nss build, unpack & overwrite: link removed
4. Smell the sensor burning :-p

This has been built with the latest gcc 4.9 and launchpad's libc_nano library, so I would advise against mixing these binaries with others with the stock toolchain. Don't expect me to update this regularly, if at all. I didn't test video, and it contains an older mlv_play module from the nss branch since I was unable to merge trunk to it. But stills shooting is just fine for me, and you can even listen to mp3 audio with your 6d now :-)

Any other options, besides the "wait and hope"?

The nss branch currently doesn't even merge, there's a conflict in mlv_sound and the collisions are such that I personally cannot figure out what is vanilla-updated or nss. If you look at the commits, it's safe to say w/o new dev resources you're in for a longer wait: https://bitbucket.org/hudson/magic-lantern/branch/new-sound-system

Any other options, besides the "wait and hope"?

Other than that specific mlv module, I can offer to share my personal 6d build with nss on a 100% absolutely strictly "no warranty at all" and "no questions asked ever" basis if forum policy allows for it.

The build works fine for me and Maqs somewhat tested a merged build, too - that's why my idea is to merge the nss branch at least as a 6d compile-time feature so it goes straight to nightly as the lack of audio is such a severe drawback.

Anyone faced these problems b4?

I don't do the video thing and thus haven't encountered these before. One question though: Is this with vanilla Canon h264 video, or with h264 and ML tweaks, or with a custom ML raw video module?

In any case, imho good thing 6d.116 is merged and there's more widespread testing - with the multitude of configs and use scenarios out there it's virtually impossible to finalize some beta build only through a handful of daring testers compiling it themselves or grabbing it of  a buried forum thread.

On Ubuntu I have no probs with using the delivered libc.a from 4.9.3 launchpad embedded gcc. libc_nano.a is said to have same or better performance compared to dietlibc. That's what I seen on forum discussions on the inet.

My 6d works fine with both launchpad's 4.9 libc.a and libc_nano.a, and from what I understand I should prefer the latter? Btw with 4.9 I really had to delete the memcpy-stub line while with 4.8 it was working 'as is'.

Most likely the differences are minor anyway, but makes you feel special to have a custom ML running :-p

You should first try
to use the libc.a from 4.9.3 rather than libc_nano.a and see if that solves your 60D issue.

Nope, still crashes on start - but works fine with 4.8.4 launchpad libc, so the compiler seems to take the blame or some digic4 garbled ml boot code. Probably there's a reason why the default makefile uses 4.8 :-p

Can't speak for cygwin.

I solved that: as the gcc binary is mingw/windows, it wants to have the libc path in windows notation - unlike the rest of the cygwin makefile which wants the unix-ish paths. Easy to get confused with mixing these both systems, but cygwin is just darn convenient for a quick ml compile.

Once the following PR's are merged

Yeah, right - but it's rather *if* than once :-p though I do like the dng module approach. Really not picking on alex here, but the time doesn't seem very convenient for big ml refactoring... real life tends to get in the way and code beautification might not make it to the top of the "most wanted" list.

Modify the newlib path to your downloaded 4.9.3 gcc arm path which contains libc.a or libc_nano.a.

Thanks, two questions though even if I appear rather incompetent (again :-p)...

1. What's the difference between vanilla libc and _nano, what's the upside of manually specifiying the _nano flavor in the Makefile.user?

2. Just replacing the path doesn't work for me, the compiler error is (cygwin with launchpad gcc) ... any hints?

Code Select Expand


make -C  /cygdrive/H/SHARED/ml/ml-modm42/platform/60D.111
make[1]: Entering directory '/cygdrive/H/SHARED/ml/ml-modm42/platform/60D.111'
[ VERSION  ]   ../../platform/60D.111/version.bin
[ VERSION  ]   ../../platform/60D.111/version.c
[ CC       ]   version.o
make -C ../../tcc
make[2]: Entering directory '/cygdrive/H/SHARED/ml/ml-modm42/tcc'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/cygdrive/H/SHARED/ml/ml-modm42/tcc'
make[1]: *** No rule to make target 'C:\cygwin\opt\gcc-arm-484\arm-none-eabi\lib\libc.a', needed by 'lib_a-setjmp.o'.  Stop.
make[1]: Leaving directory '/cygdrive/H/SHARED/ml/ml-modm42/platform/60D.111'
Makefile:18: recipe for target '60D' failed
make: *** [60D] Error 2


I can report better performance on raw zebras with O2 / O3 also read32/64 but no positive effects on write speed. Therefore I tried to see if I can replace dietlibc with the newlib-nano and compare the end results

Well, write speed is a train wreck on the 6d anyway, so never mind that :-p ... but somewhat faster overlays are definitely a plus as it's immediate better usability.