Messages

ah hadn't noticed, that full time crop moved from "don't click me" to an extra menu item. thx!

update worked fine. there is no full time crop mode so far, is it?

ah, ok nice! thx

1%, can you upload a precompiled version. don't want to install more than a gig of dependencies just for a single compilation.

hurm... that is interessting. are the dot positions in crop mode for 650d and eos m identical? they should be, since they share the same sensor, but we handled them seperatly, since the downscaling without crop mode of 650d and eos m are completely different (totally useless for eos m) and that caused different dot positions on those models. we should rename the eos m crop mode to crop mode then.
the 007 version is btw still buggy with crop mode. all resolutions that are not divisible by 4 shouldn't work. there will be an update to 008 soon.

The silent DNG contains the correct model, yes. But the raw video files doesn't contain the model information. Where does rawanizer get the model from? the raw footer contains:
magic, xRes, yRes, frameSize,  frameCount, frameSkip, sourceFpsx1000, reserved3, reserved4, raw_info
(https://bitbucket.org/hudson/magic-lantern/src/aa2ba85342faf956e8c85f514ca967ffb6f4a880/modules/lv_rec/lv_rec.h?at=unified)

char magic[4] is RAWM for all models. and raw_info contains:
api_version, buffer, height, width, pitch, frame_size, bits_per_pixel, black_level, white_level, crop (x, y, width, height), active_area (y1, x1, y2, x2), exposure_bias, cfa_pattern, calibration_illuminant1, color_matrix1, dynamic_range.
(https://bitbucket.org/hudson/magic-lantern/src/tip/src/raw.h)

really don't know where i should get the info about model out of that meta data. and  both silent dng and raw doesn't contain any information if it's recorded in crop mode or not.

the most recent version of the dot remover:
https://dl.dropbox.com/s/77bkw4u9nh7dt4d/PinkDotRemover_2013_07_15.zip

@1%:
it would be great if camera model and if it's recorded in crop mode could be stored in the raw footer. maybe you can use some bits from reserved3 or reserved4? what are the reserved bits for? maybe this info should be generally added to the raw footer. at least the camera model would totally make sense to be stored in the meta information.

@foorgol: still don't think that the method causes loss in detail at all. not all af dots are visible in all conditions, but they are still there with an incorrect value. the difference of the af-pixel value and the value that the pixel should have is just to small to notice it. however even in that case the interpolated value should be more exact than the wrong af-pixel value.

@spider: some new features that enhance the usability will come soon. we discuss features like optional overwriting the dngs, extra output folders and removing dots directly in the raw instead of the dngs. you can add feature requests on github: https://github.com/Foorgol/PinkDotRemover/issues
that way you can be sure, that the feature idea doesn't get lost between the other posts in the forum.

ah, ok, then i guess you round 1447 up to 1448, is that correct? i'll try that, thx!

EDIT:
it works... just finished a dot removal test for 73 different resolutions and they all do work perfect now.

@1%:
can you tell me the exact vertical pixel offsets of the crop frame for the resolutions 1536x562 and 1600x576?
Sensor resolution is 5184×3456, so i guess they should be (3456-562)/2 = 1447 and (3456-576)/2 = 1440.
if i calculate the center for each resolution 1536/2 = 768, 562/2 = 281 => 768,281 and 1600/2 = 800, 576/2 = 288 => 800,288 and add the offsets, i should get the exact same sensor pixel, and that should be the one left top of the 4 center pixels (5184/2 = 2592, 3456/2 = 1728 => 2592,1728). if i use the calculated values it works: 281+1447 = 1728, 288+1440 = 1728.
but if i read the pixels from the dng it's 1 pixel vertically shifted. it seems that the vertical crop offset of 1600x576 is 1439 (instead of 1440) or the vertical crop offset of 1536x562 is 1448 (instead of 1447). can you check that and tell me which exact offsets are calculated in ml.

oh, ok... thought taking a photo of a white surface is good, because the dots are very clear on that.
there are the new ones:
http://dl.dropbox.com/s/xmq2n463udces23/with_dots_2.dng
http://dl.dropbox.com/s/attqx24gkkvrg88/without_dots_2.dng

hope they are correct now..

this is a mask of the dot grid the dot remover uses. the black pixels get interpolated. maybe this will help too.
http://dl.dropboxusercontent.com/s/o3qe0i1swnp47i1/dot_mask_1536.png

with dots:
http://dl.dropbox.com/s/738xtis2y8b621j/with_dots.dng
and without:
http://dl.dropbox.com/s/pzbrwho0vasche7/without_dots.dng

hope they help...

on bayer data each pixel contains only color information of one color channel.
what data do you exactly need? a dng file without removed dots?

it's always just the center pixel. the 4 pixels around get distorted by demosaicing, if the wrong pixel isn't corrected before. and i don't think the center pixel does contain any useful data, does it?

no it doesn't. the only effect you may get is underexposed footage. that's what htp does, underexposing and brighten shadows.

what's wrong with the method of the dot removal tool for removing dots? if you need a dot map, you can directly use the dot grid data from the dot remover, if that helps:
https://github.com/Foorgol/PinkDotRemover/blob/master/dotData/EOSM_Crop.txt

first value is horizontal offset (center based), secound is vertical offset (center based), third is vertical limit (center based), fourth is horizontal step size and fifth is vertical step size. this grid data should work for video recordings in full time crop.

@handbanana: i wouldn't have to much hope, that eos m will ever record clean raw video without crop. but i don't know... maybe 1% can say anything about that. maybe anything has changed with the new firmware?

@1%: i noticed, that af works with crop mode and even while raw recording. i really don't need it while recording, but it's quite handy before i start recording, since zoom doesn't work and that's the fastest way to get good initial focus. btw, you can only focus once while recording, then it won't work, until you stop the record, if anyone needs af while recording. my problem is, what i hated at the eos m af anyways, that the mixed auto+m focus mode doesn't allow manual focus, if you didn't let it do af first. so if i use af for initial focus i can't focus manually while recording. is there any way to change the behavior of that mode, that it always uses manual focus if you turn the focus ring. that would be great also for other situations... like the mz enables while turning the af ring it should also switch to manual focus.. and if you don't turn the focus ring for 2-3 secounds switch back to af+m. don't know if you can switch af mode and if it's possible to detect focus ring changes while not in manual focus mode.
another thing i noticed is that it is possible to record h.264 digitally zoomed in full time crop. that's really nice.

the 18mp are 5184x3456 and the magification is bit more than factor 3 (18mm on crop is bit closer than 55mm without, as you can see in the videos), so i guess it is just a 100% crop, yes. then the factor should theoretically be 3,24 with 1600x576. and exactly 3 with 1728.
the more often i watch the video the more i like the 1728x496 the aspect ratio is extreme... but it really looks clean and sharp... compared to h.264 the 1600x576 is also much sharper, but it's just not as clean as the 1728... hurm have to do more videos with both resolutions and check which one i like more.

if you record video with eos m without crop mode you'll get very unclean edges. there is an example video h.264 first, then raw.
http://dl.dropbox.com/s/lincex5op3relr7/dotremoval.mp4
there are imho 2 reasons that cause this problem:
1. you get less vertical resolution than horizontal and has to strech it vertically
2. the way the 18megapixels get scaled down to the ~2 megapixels used for video isn't high quality. i think they just skip the pixels they don't need, instead of any interpolation

today i did some new shots... they really suck, because i focused on doing shots with the different settings, instead of what i shoot. but... it's good enough to judge quality, imho.
here's the description why i added that setting to the demo video, for each setting:
1. h.264 normal 18mm, to compare the angle of view to 18mm crop
2. there it is... 18mm crop that's how it's magnified at crop mode
3. raw 1280x662. that's the closest you can get to 720p for continuous recording
4. raw 1600x576. this is imho one of the best resolutions for raw recording. it doesn't has to be as much upscaled as the 1280x662 and has an acceptable aspect ratio
5. 1728x496. the highest horizontal resolution you can get. to get continuous recording the aspect ratio is very extreme... but it shows what the maximum quality is.
6. again h.264 18mm for angle of view comparison
7. h.264 55mm to compare the angle of view, sharpness and dof to 18mm crop
8. h.264 18mm crop
9. h.264 55mm and crop mode

http://dl.dropbox.com/s/06dgg2nfkqa0jgf/raw_h264_comparison.mp4

hope this helps you to judge the quality you can get and if raw video on eos m makes sense for you. if you want any raw files directly from the camera, just tell me which ones, then i'll upload it.

@Canon eos m:
i found some time to create the screencast for you. most is simply the same as in all the other ml raw workflow videos on youtube, but there are 2 steps that may help you especially.
1. it shows how to remove dots
2. it shows how to scale it to 1080p in after effects
one problem is, that i accidentally didn't record the mouse cursor. i usually don't do screencasts on windows, so i didn't know the screencast software. i hope it's understandable anyways. the other thing is, that i recorded it without sound. but i'll write you a description here:
https://dl.dropboxusercontent.com/s/9tetcfem6qb43nz/post_processing_dotremoval_ae_neatvideo.webm

1. droped the raw video onto raw2dng.exe to extract the dngs (i canceled it after 76 frames to keep the video short)
2. moved the raw out of the dng directory
3. opened the first frame to show that there are dots
4. opened the dot removal tool and added the files to the list
5. changed the dropdown in dot remover to EOSM_Crop
6. got sure, that mode is "interpolate"
7. converted files (it shows an error for the last frame, this is because i interrupted the dng extraction at frame 76 before... it wasn't completly extracted)
8. removed the old dngs (with dots), all files without "_" prefix in the directory
9. showed that the new first frame doesn't has dots anymore
10. opened after effects
11. imported multiple files (selected the first dng)
12. whitebalance in camera raw (camera raw opens automatically while importing dng to after effects)
13. clicked onto done to close the import dialog
14. created new composition
15. changed composition settings to my target resolution. (lock aspect ratio has to be disabled)
16. dragged the edge of the video to match the size of the new composition (hold shift while dragging to keep aspect ratio)
17. added neat video
18. opened neat video options
19. selected an area where neat video should create a profile from. that way wouldn't lead to the best results, you should do noise profiles for different iso values with extra footage and then just use them instead of creating a new noise profile. but if no noise profile is available this should work to.
20. adjusted noise filter settings... i do a lot of sharpening with the noise filter... just try which way of sharpening works best for you
21. added unsharpen mask to do even more sharpening
22. added levels and curves to increase contrast, midtones and light in dark areas
23. added a photo filter to bringt back a bit more of the pinkish tone of the down going sun
24. checked the video and noticed the wrong frame rate.
25. changed composition options again to match the correct frame rate.

hope this helps you to get dotless and noiseless videos in post

@handbanana:
i tried to create some demo videos for you yesterday, but after work it was to late and to dark.. the videos just sucked
i'll do some demo stuff now and upload it for you in few hours.
the main problem with raw video on eosm is the extreme crop... so i doubt, that it may be more usable than 60d. the faster sd controller let you record more pixels, but you're limited to the tele range. and the quality of 100% crops is just not as good as properly downscaled data.

@1%:
did the new dot removal data work for you?

@handbanana:
i can upload you some sample data later if you like?

silent dngs do use the active area, which the video frames don't. they should be easy to support (they do work for 650d), but the active area just confused me. i got even more confused by the term "frame skip" since i thought about skipped video frames.
since the calculation of the dots is center based and the active area crop is not, it has to be corrected with the correct offset. is not really complicated at all, was just my fault -.-

EDIT:
try this one for silent dngs (select EOSM not EOSM_Crop from the dropdown):
http://dl.dropboxusercontent.com/s/j727o0iaho4i3vd/EOSM.txt

i'll add it to the repository tomorrow.

it is 8bit and there is data lost already... but if you increase the contrast before you get it out of the camera, you'll lose even more information.

the dng is a silent dng, isn't it? still had no time to fix the silent dngs. there is something wrong with the calculation because of the frame skip. all video dngs should work, i planned to do the silent dng at the weekend. but... i'll check if i can do it now

ok... nice. i'm not often in 720p anyways... rarely need more than 25fps...
any more detailed information, why you don't think that flat profiles increase dynamic range?

intervalometer does work on eos m... if that is what you need, it should work for you.

@1%: i still never got any black screen with the full time crop mode btw. even after accidentally activating in photo mode or switching to photo mode while it's activated. is there any reason to leave the full time crop mode, except for h.264 video recordings?