Messages

Baldand, as requested I PMed you links to some sample RAW footage with pink dots from my 650D. Please let me know once you've dl/ed them so I can clear up some space on my Dropbox acct.

Thanks,

Dave

Yes, it supports .RAW files also. They would also be useful for me.

I've PMed you some RAW files

A small request to people using camera with Pink Dots...

MlRawViewer 1.1.6 introduced a new GPU-based bayer pre-processing option that tries to identify any "bad pixels". This is mainly for dead or hot pixels, but I have also been testing with a little material I have from PinkDot cameras, and I believe it can also detect and suppress many of those with a minimum of impact to the rest of the scene.

The feature is still in development, and would need some more tuning to be totally effective in the pink dot case. In particular I probably need to change the work flow so you can "train it" on a certain resolution, and then re-use that training data in other shots.

If anyone is working with raw on those cameras, and would be interested to help me develop this further, please try out the current code and let me know how it goes. I would also appreciate being sent more such MLVs to test it against.

Thanks!

If your code works on RAW, not just MLV, then I'd be happy to help.

I haven't moved on from RAW to MLV yet since, last I checked, you can't achieve the same resolution (continuous record) with MLV.

In any case, I can provide you some .RAW files if they're of any use in your testing. Let me know.

[PinkDotRemover]

Pink dots in RAW is a well known bug, you can fix that by using PinkDotRemover. You're welcome.

Just to be clear, my post was about artifacts in live view on an external monitor during recording. I'm well aware of the methods available to remove focus pixels in post-processing.

I'm investigating purchasing an external monitor, so I plugged my 650d into one of my desktop 1080p monitors to see what it was like.

(I apologize if any of this is covered already - I've read this thread in dribs and drabs over the last long while and don't remember any issues with HDMI, and the forum offers no way to search specific threads so I keep finding HDMI issues relating to the 5dIII)

Here's what I found:

1) Canon preview mode gives incorrect framing. Greyscale ML and HaCkED don't appear to work.
2) Focus peaking - a bar of peaking dots appears across the bottom third of the screen independent of what's in focus
3) covering the light sensor to activate arrow keys stops working
4) RAW playback isn't full screen and gives random pink/green/blue/red noise in the bars above&below.

Please take this as a bug report not a complaint.

I'm thinking I'll still spring for the monitor since focus peaking isn't really critical once your screen is larger, and I might be able to make a custom framing cropmark to frame the image properly (addressing #1). #3 is just a mild annoyance.

Cheers,

Dave

Thanks alex! Are all the versions which are posted in the first post updated with this? (specifically, is the cs2x2 version updated?)

Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar

Hi Edgar,

Here's a short RAW:

https://dl.dropboxusercontent.com/u/30586759/M25-2141.RAW

In this example there is only 1 pixel that is bad and it happens on the 12th frame (I have some footage that has many more of the artifacts if you need it for testing but the RAW files are huge). Look for the bad pixel towards the bottom left of the flame. It is there if you convert to DNG with raw2dng cs2x2 but not if you use rawanizer.

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

Glad I could help!  If you have a mac nearby, the best tool i've found so far is PDR interp by rewind.  It's written in java so i'd assume it would run on a PC as well?  It's very robust and produces superior results (imo).  It certianly can be difficult to find though, as its buried in a PDR topic somewhere on the forum.  I can re-up it if its beneficial, or someone can reply to this with a forum link if they find it.

Sidenote, if you're affiliated (even remotely) with an educational institution, you can get Adobe CC for $19/mo if cost is an issue with adobe's suite of stuff.  I'm not particularly a fan of leasing software... but its an option.

The MAC version of PDR continues to work with the latest nightlies? The windows version used to work but stopped after a certain version of ML (for reasons unbeknownst to me) so I switched to raw2dng...

escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:

Code Select Expand

Cold pixels : 0

I looked quickly through the source code and thought I saw something about not targeting pixels that are in over-exposed regions. The dark pixels I have are ONLY in over-exposed regions
If you have time maybe you can investigate? If not I will try, but I am just starting to learn C so it may be a long while!

EDIT: Also, here's is a more thorough description of my problem:

https://www.dropbox.com/s/ijgu5z62cia2zzi/Screenshot%202014-02-21%2022.47.33.png

http://www.magiclantern.fm/forum/index.php?topic=7473.msg103033#msg103033

I'm running Ubuntu 10.04 LTS 32bit, and having trouble compiling raw2dng as an .exe

DISCLAIMER: I'm a bit of a newb at this

I've cloned the source, installed mingw32 and pointed the Makefile to use /usr/bin/i586-mingw32msvc-gcc for the compiler

I'm getting the following error:

Code Select Expand

root@dcerouter:~/raw2dng/magic-lantern/modules/lv_rec# make raw2dng.exe
[ MINGW    ]   raw2dng.exe
../../src/chdk-dng.c: In function âfind_tag_indexâ:
../../src/chdk-dng.c:263: warning: implicit declaration of function â__builtin_unreachableâ
[ MINGW    ]   raw2dng.exe
raw2dng.c: In function âmainâ:
raw2dng.c:72: warning: implicit declaration of function âfseekoâ
[ MINGW    ]   raw2dng.exe
raw2dng.o:raw2dng.c:(.text+0x1468): undefined reference to `_fseeko'
collect2: ld returned 1 exit status
make: *** [raw2dng.exe] Error 1


In searching I found one forum thread which indicated gcc-multilib needed to be installed, but it hasn't resolved things.

Does this mean that the compiler isn't recognizing fseeko() as an already defined function from stdio.h? Shouldn't it since stdio.h is included in raw2dng.c?

Dave,  I was able to replicate the issue after poking through frame by frame.  After some testing and mucking around, i'm leaning toward a cs2x2 issue.  AE doesn't show the dots because whenever you're working with DNGs it will automaticially run dot remover behind the scenes, thus a clean image.  Actually, all adobe tools do this when working with raw/dng.  I baked in the black dots using Davinci to export into a prores sequence and took that into AE and that made them visable.

My best suggestion would be to take the source footage and without using cs2x2, or any PDR for that matter, check the frames.  Sure it'll have pink dots but it will identify the issue: either Davinci's raw engine is wonky (not that likely) or its cs2x2 (more likely).

Thank you! Seems you are right, it IS raw2dng. I wasn't aware that adobe was running a dot-remover - makes a lot more sense now.

I guess it's time for me to finally learn C++ and dive in to trying to fix raw2dng. I like the workflow I have with raw2dng and Resolve and I really shouldn't be using the copy of AE that I found on the side of the road (nor do I want to buy it)

Hey Dave,
I tried it on my mac using davinci 10.1, didnt' see any noticable difference between DR and AE or even lightroom (for raw/dng viewing).  Can you post a screenshot?

Thanks jonsium,

Here's a screenshot:

https://www.dropbox.com/s/ijgu5z62cia2zzi/Screenshot%202014-02-21%2022.47.33.png

Wonder if it's an issue specific to Resolve and my specific video card. I'm on the latest nvidia driver and resolve version. What video card do you have? Mine is a NVIDIA GTX 560

[Can you update PinkDotRemover to remove dots from (2.20:1) 1344x610 .RAW file (recorded with mlv_rec)?Or is there more programs made for remove pink dots?]

Can you update PinkDotRemover to remove dots from (2.20:1) 1344x610 .RAW file (recorded with mlv_rec)?
Or is there more programs made for remove pink dots?

DNG: http://www.upload.ee/image/3899487/000000.dng

Have you tried raw2dng cs2x2?

Can I ask the help of someone who has Resolve 10 lite on their system? I need someone to load my project on their setup to see if they are getting the same artifacts that I am.

This was recorded with the latest ML nightly on my 650d, converted to DNGs using raw2dng cs2x2, and I'm getting dark pixellation in the overexposed areas that DOES NOT seem to appear in AE or Windows Photo Viewer.

Here's the .DNG stills (50MB):
https://dl.dropboxusercontent.com/u/30586759/bmd_help/m12-1831.zip

Here's the .DRP (project settings) file:
https://dl.dropboxusercontent.com/u/30586759/bmd_help/bmd_help.drp

Thanks!

EDIT: As jonsium pointed out, the artifacts don't appear in adobe products because they run a bad pixel algo when you import DNG

Why would you use such an outdated version? There's no reason not to update

There may be one exception... I just moved to the nightly from the stable August version.

As far as I can tell the Stable August version has the advantage that PinkDotRemover (PDR) v0.8 works successfully on the footage, while the nightlies do not. For nightlies one must use raw2dng cs2x2 to remove focus dots, which Rewind points out is not as good an approach as PDR (my limited understanding:  targeted pixel interpolation in PDR versus blanket chroma smoothing in raw2dng). To my less discerning eye, raw2dng is fine except when there's heavily overexposed areas of the image.

Improvements in the nightlies I've found are:

* slightly increased realized SD card write speeds, meaning higher resolution you can record at continuously.
* crop mode
* dual ISO mode (haven't really used it yet though)

I'm sure there's more but I've only been using ML for about a week and had an interest in serious video recording for about the same amount time.

Hi Alex,

I'm was having an issue with green dots in overexposed areas from my 650d... i used your raw2dng whitefix version and it got rid of the green dots except left some black/grey dots in the overexposed regions during playback in Resolve 10 lite. I went to inspect the dngs with windows picture viewer and these dots couldn't be found! What could your program be doing to cause Resolve to interpret these pixels as dark grey?

Thanks!

Nevermind, looks like a bug related to Resolve and possibly my graphics card...

(http://forum.blackmagicdesign.com/viewtopic.php?f=3&t=12941)

EDIT: Or not... the solution in the above link doesn't work for me, and I also noticed that the "black" pixels are actually a very dark green. If you have any ideas Alex I'd love to hear them!

Hi Alex,

I'm was having an issue with green dots in overexposed areas from my 650d... i used your raw2dng whitefix version and it got rid of the green dots except left some black/grey dots in the overexposed regions during playback in Resolve 10 lite. I went to inspect the dngs with windows picture viewer and these dots couldn't be found! What could your program be doing to cause Resolve to interpret these pixels as dark grey?

Thanks!

I understand mlv results in a drop in max resolution one can continuously record at on the 650d, or a reduced number of frames one can record for a given resolution. (I'm not sure why, my best guess is that more data - metadata - is being saved with EACH frame).

In order to maintain the max performance but allow an algorithm that targets bad pixels instead of blanket chroma smoothing which Rewind points out is non-ideal, could the raw recording module be modified to spit out a separate .fpd (frame position data) file which can be passed to the PDR tool as a reference point to properly determine bad pixel coordinates?

There's probably a good reason why this hasn't been proposed, but I'm just too new to know!

Hi Dave, I asked the same question a while back and received no real solid answer. So I went ahead and downloaded the latest build and 1280x720 is 99% red dot free with PDR tool 7 or 8. 1472x626 however is loaded with dots. There is a cool tool a1ex made called " raw2dng-cs2x2...hdlike_noise.exe" which cleans up the 1472x626 pretty much. Just Google that tool name and you should find the threads with download links. Also, I have found that adobe camera raw (ACR) used with most adobe software like After Effects helps with properly displaying the tool converted DNG files. I'm not sure why but that's what I'm having to use after PDR is applied. Yes...it is a mind bender and a bummer at times but just imagine what the devs go through...upmost respect for those guys!

Thanks anewcomb!

I've been making my way through the forum threads and changelog, though not finished yet. Lots of it is technical beyond my understanding, but I gather they've squeezed a bit more performance out of the write speeds in the latest builds, and hence increased the maximum resolution you can record without dropouts. That's reason enough for me to update ML!

I'll give the raw2dng tool a shot. I'm partial to Resolve, so hopefully I can still fit that in my workflow. If it breaks things for me I suppose I can always go back to the August build

Cheers,

Dave

Hi all,

Getting started with Magic Lantern on my 650d, and loving it!

I'm using the magiclantern-v2.3.NEXT.2013Aug03.650D104.zip version from the OP so far for RAW video with good results but wondering about the latest nightlies. Is there a summary of what's changed/improved somewhere?

Right now the PDR tool works well for me, but I understand it's broken for the latest nightlies... I assume in exchange for some other advantages? Which ML version is generally recommended for someone who's primarily concerned with RAW video quality and features relating to RAW video (e.g. I have no need for audio or stills)?

Thanks!

Dave