Tragic Lantern for EOS M

[maxotics]

I haven't changed my MLs on my cameras in weeks, fearing problems.  We need to agree on some way of separating what are stable versions from the problematic.  I don't want to change anything until I have a robust workflow, from filming to NLE.

Making progress on ffmpeg conversion of TIFFs to high quality clips.

Rewind, as you know, I don't think the PDR problem is fully solved.  Your new crop mode version seems to be working.  But now my non-crop doesn't.  I'm working on a .NET app that I will open-source.  Love your input when you have time, or interest!

I might try to learn Java and work on the existing PDR, but it'll be faster for me with .NET and my strategy of using static maps will mean that I would probably use less of the existing PDR code.  Anyway, probably good to start from scratch, just to learn.

My current strategy is.

1. Focus camera on white wall and take some RAW.
2. Alter in image editor to turn background black.
3. Run through .NET code to mark each pixel that is a focus pixel.  Most are in blocks of 9.  There are also some single pixels used for focus or something.
4. Build static map of ALL sensor's focus pixels, which can, with the right x/y offset, map onto any image from camera.

Phase 1

1. Go through image, by row, col saving last good pixel, if reach bad pixel, mark replacement (previous good pixel).  Yes, this will mean 3 focus pixels in a row will get same left-hand good pixel.  My guess is this will, despite sounding rough (replacing 4.4% of pixels in crop mode), work visually.

Phase 2

1. While going through frame, check to see if center pixel is redder, bluer, or greener than pixel outside of focus block.  As you know, focus pixels don't always fire.  They only seem to do so on edges, clipped readings, etc.
2. If focus block seems active, then do a more robust interpolation around those 9 pixels (or 1 pixel).

Lots of questions!  Anyway, that's where I am. 

[manly]

Well I bought a Sandisk extreme 45MB/s UHS-1 16GB on ebay for 19$, shipped. Seriously ebay is amazing for camera gear prices. It came new in sealed pack.

[Lou]

I'm a new user and need some advice. I have tried to browse the tens of pages about ML on the EOS-M but may have missed the obvious - so apologies up front.
I have a the latest Canon FW loaded and also have 22mm and the 11-22mm. I installed the ML version referred to by feureu (reply 1801). It works fine except in one particular instance.
a) in the Canon menu I set the Focus Method to AF.
b) in the ML menu I loaded the raw module and set raw in the video menu.
c) pressing the red record button shows that recording takes place. Pressing it again stops recording.
d) during RAW recording, pressing the shutter button half-way freezes the camera with the picture frozen on the screen and no record light flashing.
e) the on.off switch is also dead, so the only solution is to pull the battery.
f) this happens on the 22mm and 11-22mm lenses.
g) if the Focus Method in the Canon menu is set to Manual then the problem does not occur.
If already a known problem then I would appreciate a pointer where to look.

[andyroo]

Hi! I am new to ML too (Long term user of CHDK).

The EOS M is my first larger sensor camera, and I am trying to make it into a stable intervalometer that just turns on, so I can bolt it onto a plane and collect aerial imagery. I read through other forum posts and saw that PicoC was/is implemented(ish) in ML, but is that the case for the EOS M alpha builds too? When I tried to put an autorun.c script in the /scripts directory on my EOS M install, (1) it didn't run and (2) I couldn't figure out how to even find it from the ML menus.

If the Picoc code isn't implemented, then is there any way to autostart the built-in intervalometer on boot up? If not, and I want to try to add it in, would I be able to access the source code for the ongoing alpha on the bitbucket link, or is it elsewhere?

I almost hate to admit it, but I am a scientist, not a photographer, so my only goal is to make it so when the camera is powered on, it starts taking pictures. Ideally it would just start taking pics at whatever the intervalometer is set at, but since I have the "shutter-bug" too, it's already more complicated than I hoped. And I have to make this simple enough that a pilot can get the camera up and running without me (I already have a Canon D10 doing this with CHDK, and got the EOS M because I am excited about the HUGE sensor, and the camera is small enough to fit in the wing of a Cessna 172).

Incidentally, my first post was over on the shutter-bug thread - I found a (pain-in-the-ass) workaround, and a repeatable way to both exhibit and bypass the shutter-bug, and it seems like that was something lacking so far.

I have the 18-55 mm lens, camera fw 2.02, lens fw 2.00 (never upgraded, my camera came with 2.02). ML on a Transcend SDHC Class 10 32 GB card

I installed magiclantern from these two files:

http://ml.bot-fly.com/EOSM1202.FIR
http://ml.bot-fly.com/magiclantern-v2.3.NEXT.2013Sep17.EOSM202.zip

Anyway here's the baby script I am trying to run. Power saving and "yes I am working" indicators are in the wings but I want this to work first:

Code Select Expand

/*
@title PlaneCam intervalometer
*/
//adapted from Francis' demo code on ML website
//http://www.magiclantern.fm/forum/index.php?topic=5155.msg31568#msg31568

printf("Hello from PlaneCam!\n");
sleep(10); //Wait 10s before starting to take pics

for (int i = 0; i < 5000; i++) //Repeat the stuff below 5000 times or until there is no more power
{

takepic(); //Take a picture
sleep(3); //Wait 3 seconds
}

[jerrykil]

Andyroo,
i've been meaning to play with the scripting capability too. i get errors, however, if i try to run the tcc module. i'm not sure how to build the picoc module either. I read on some forum posts that picoc is being abandoned for tcc, anyway.

here are the errors i get with the tcc module:



EDIT: so i searched forums for tinypy, and there isn't much posted about it. i can't get it to compile but i think its something up with module_strings.h. i don't know of any examples for using tinypy, either

[DanyoO]

Video raw crop 1280X720 200ISO PDR Rewind for EOS-M crop 25p
Canon Eos M + 22MM F2

Youtube
http://www.youtube.com/watch?v=ULLDo-i0m6I

Vimeo
https://vimeo.com/74902807

[Rewind]

Video raw crop 1280X720 200ISO PDR Rewind for EOS-M crop 25p

Nice video! What this guy performs is not an easy shit at all. He's a man.
But be careful, doubleposting is not much appreciated here

[tetsu]

I´ve found out that the older EOS M firmware had the video zoom option (3x to 10x) while retaining full hd quality, this was disabled in the latest firmware, is there a way for ML to re-enable this? Not in raw just plain h264. I know that raw uses this method when filming, but it would be nice to have also the h264 option when you don´t need that kind of quality.

Thanks 1% and a1ex and the others for all the great and hard work!

[maxotics]

@DanyoO, very very nice!  Did you used the original PDR or Rewind's fix?  Also, can you can tell us more about your workflow?  Thanks for sharing!

[DanyoO]

SORRY MY ENGLISH IS BAD 



I USE FOR THE POINTS ROSE PDR Rewind for EOS-M crop
VIDEO 25FPS PAL
MY MEMORY (SANDISCK EXTREM PRO 95MB) 32GB
AND VIDEO  1280 X 720 CROP MODE
200ISO    22MM F2



DROBOX VIDEO RAW:        https://www.dropbox.com/s/qcsz91y9hn2amoe/AIR%20FLARE%20DanyzoO.avi

[maxotics]

Danyo0 No problem!  Your Video Skills are great   Thanks for the info!

[maxotics]

I sent Mixer2 a private message and about PDR issues and his response was so long and interesting I'm sure others will want to read it:

> Also, selfishly, I'd like to create a windows version so I can process my RAW files all in one batch.  RAW -> DNG --> DotRemoval --> TIFF --> .MOV/Cineform, etc.

it's still a bad idea to directly convert to mov or cineform. some editors like adobe (after effects, premiere, speedgrade) or davinci resolve allow to open dng directly. in any case it's a good idea to use image sequences instead of normal video codecs. just because of compatibility reasons. for example if you want to do some vfx with programs like voodoo camera tracker, blender, slowmoVideo etc you'll get a lot of problems with video files.
my current workflow is:

raw -> dot removal -> raw2dng -> color correction and grading in darktable (like lightroom, but free) -> jpg -> noise removal -> (vfx if needed) -> cutting with lightworks -> encoding

of course this workflow isn't perfect, since color grading should be the last step. that would make vfx more simple to match the scene and noise removal would also work better. but the editors i use don't support dng and as soon as the dng is converted a lot of information needed for color grading is lost.
more simple is the adobe workflow that i use less, since i don't use much windows. there you can load the dng directly into premiere, do the noise removal and vfx in after effects and grade in speedgrade. then you don't have a forced fix order because of compatible formats and can do it just as it's best for the specific project.

a workflow with cinemadng format in davinci resolve maybe in combination with lightworks as nle should also work quite nice, but never tried. it may be the most professional one and is completely free if you use davinci resolve lite.

> 1. My analysis of the DNG files is inconclusive.  Single pixels and blocks of 9 pixels may be used for focusing?  Do the RAW files have complete bayer information? Is it copied over to the DNG.  I'm a bit lost on this.  What is your opinion?

just single pixels are used for focusing. you can get the dot pattern from the dot data text files of the pdr. the pattern is fixed and can't change with software versions, but the position may change dependent on the crop ml uses. for crop mode and non crop mode you'll get different patterns since non crop mode is downscaled by skipping lines and rows. that's why non crop mode has diagonal neighboring focus pixels. the pattern does also change for different modes like silent dng etc, because of the different scaling ratios.
the raw does contain the really raw bayer data from the sensor plus some extra header and footer. the dng format does also allow raw sensor data, so for dng conversion it's just copied over frame by frame and each dng gets also a correct dng header. you shouldn't care about dng conversion, there are already good tools to convert to dng or cinemadng that can be used in scripts via cli.

> Do you think it a strategy worth pursuing, that I write a small Windows .NET utility that writes a FixBadPixellist to each of the DNGs, based on the crop mode of the video?  (I'd also have a utility that would calculate the section of the map that would apply to that clips frame).  Can I get to those single pixels. 
It seems the DNGs show focus effects in many blocks of 9.

most video software ignores the bad pixel list of dngs. that's why we recommend using interpolation in pdr, instead of marking them. i don't understand what exactly the advantage of a new utility would be. the problems are exactly the same as with pdr.
investigating in more advanced interpolation algorithms (for bayer data) would be very helpful.

> If there are blocks of 9 pixels in the DNGs, and there is no way of avoiding it, does this make sense to you?

there aren't blocks of 9 pixels in the DNGs. at least i never saw anything like that. a block of 9 pixels would lead to a dot of at least 21 pixels after demosaicing.

What about?
1. Focus camera on white wall and take some RAW.
2. Alter in image editor to turn background black.
3. Run through .NET code to mark each pixel that is a focus pixel.  Most are in blocks of 9.  There are also some single pixels used for focus or something.
4. Build static map of ALL sensor's focus pixels, which can, with the right x/y offset, map onto any image from camera.

foorgool also started with a similar method to create first dot maps. check the first pages of the pdr thread. since the dot pattern is very consistent any detection isn't needed. the positions of the dots can simply described as we did in the dot data files of pdr.
maybe an option to automatically find and fix bad pixels if no pattern does match would be worth to investigate in. the new interpolation algorithm that is used in the pdr test version also describes a method to detect bad pixels in the paper. it should be simple to implement that.

>While going through frame, check to see if center pixel is redder, bluer, or greener than pixel outside of focus block.  As you know, focus pixels don't always fire.  They only seem to do so on edges, clipped readings, etc.

i don't think it's worth careing if the focus pixel fires (is visible) or not. if the pixels around are nearly of the same color you won't lose much information by interpolation anyways, since it should get nearly the same color. if they are not, the information of this pixel is lost with both methods.

i think it shouldn't be to hard to get pdr work with current versions. one of the most problematic stuff that has to be done with pdr is compatibility for dual iso, since the interpolation would be much more complex with different exposures.

hope this information clears it up a bit.

[mixer2]

@DanyoO:
really nice video! the shaky stuff form 0:15-0:20 is bit strange, imho. but it's definitely the best eos m raw video i've seen so far. the uncontrasty yellowish look is fantastic.

[DanyoO]

Thank you Mixer2

[gary2013]

1%,

When I record raw movies, it disables the audio, which I expect. But, when I disable raw recording of movies to go back to recording normal H264, the audio is still disabled. Can you make it so when we disable the raw recording of movies it will switch back to the "previous" settings of audio manual or auto recording?

Gary

[gary2013]

The loading of modules is all messed up. I am trying both Sept 19th and Sept 17th versions. I Liked the old way where we select either on or off for each module and the select "load modules". There is no load now and I always get these err labels on each module listed. I managed to get some to load with using the debug modules on in the next tab to the right of modules. But, it all seems very unintuitive and I am just guessing with maybe four attempts of cycling on and off hoping they load.

Gary

[jerrykil]

gary, there are more modules built into the last couple nightlies. you should not enable scripting and tcc. they don't work, yet. see if that helps. i will disable them for tomorrows build

[gary2013]

I sent Mixer2 ...

There will be an update on Oct 15th for Adobe Premiere Pro CC that will then allow DNG  file sequences to load. This will be good, if anyone uses PPro CC. Then we can shoot raw, pdr the raw file and extract to dng. Then just import the dng seqq to PPro CC and the ACR (Adobe Camera Raw filter) pops up that allows a first grade which is always needed to see something usable to edit. Make sure to use Select All in the top left of the window and then tweak away. The filter has many settings including noise reduction, but I always use exposure and color temp. Then select Done and edit in PPro. I then apply Neat Video as a noise reduction filter. You can always recall ACR and make new adjustments as needed which is the benefit of DNG as Mixer2 stated in the previous post.

This will be much better then using AE or other workflows, IMO. Plus, you can round trip with Dynamic Linking that is simple using Speedgrade for advanced color grading and correction. Or maybe Davinci or Assimilate Scratch. I have been using Photoshop to import DNG seq and then it's the same with ACR, but I now have to use Save As Tif to all the DNG's and then edit with Tif seq and that is not as good since I lose the benefits of raw DNG grading. Staying in DNG seq for edit, FX, grading is the easiest and best quality and you can even save the final project as DNG and then make conversion of any type needed for delivery. Like Mov, AVI, etc. or even back to H264. etc.

Gary

[gary2013]

gary, there are more modules built into the last couple nightlies. you should not enable scripting and tcc. they don't work, yet. see if that helps. i will disable them for tomorrows build

Ok, I will try it now.
Gary
Edit-just tried sept 19th build and shut off tcc and script modules. cycled on and off the cam and it shows all err for all modules except the two I shut off.

[gary2013]

dot tune, mem spy and bolt rec are also causing problems for the other modules to load. I had to shut all of them off before the rest of the modules would load properly.

Gary

[a1ex]

Tip: "make zip" in platform directory will bundle only the modules that can load on your camera without errors.

[gary2013]

i deleted the offending modules from the memory card and after three reboots they all loaded ok. Except bolt rec says oldapi.
this is sept 19th build

gary

[maxotics]

Andy posted about a new stable release for the 50D which I downloaded and tried.  Works great.  Is there someone creating stable builds for the EOS-M?  (I understand, strictly speaking, that all builds are alpha.)

[jerrykil]

Tip: "make zip" in platform directory will bundle only the modules that can load on your camera without errors.

Thx for the tip, a1ex. I noticed my make zip will build whatever is in tragic-lantern-6d/modules/Makefile.modules.default and then package whatever modules it notices were built before. mlv_rec will not build and will cause make zip to abort:

Code Select Expand

make[2]: *** No rule to make target `lzma/7zAlloc.host.o', needed by `lzma/lib7z.a'.  Stop.
make[1]: *** [mlv_rec] Error 2
make: *** [CONFIG_MODULES_compile] Error 2

1% said he'd include the lzma directory at some point already. the module builds fine, but the converter doesn't, he told the 99%

mlv_eec is included in the Makefile.modules.default. Anyway, this is minor housekeeping stuff and I don't think its any problem, I'll just tinker the makefile to not build all that stuff.

Gary, i've updated the sept19 build with these modules only:

raw_rec file_man pic_view ettr autoexpo dual_iso arkanoid silent dot_tune

i had to make this change to the Makefile.platform.default for 'make zip' to finish

Code Select Expand


ML_MODULES_DYNAMIC ?= raw_rec file_man pic_view ettr autoexpo dual_iso arkanoid silent dot_tune #mlv_rec

ps. new fonts look sweet

[gary2013]

I also see EOSM_202.sym   in the modules folder. What is that for and does it belong there?

gary