Messages

Anybody know the status on the pink dot remover for RAW video?  I just got an EOS-M for b-cam/personal shots and was excited to try RAW with the faster card reads than my t3i.  I tried the PDR in maxotics package but it didn't seem to do anything.  Would it be worth trying to edit the PDR eosm file or I could also try to code a simple pink dot remover/raw2dng python batch runner.

The status of PDR is the first two devs are gone and it doesn't look like they're coming back. Rewind knows a bunch, but we (I) probably drove him away in aggravation (he doesn't use an EOS-M). 

If you're shooting in crop mode, which I strongly suggest with the EOS-M, then the "PDR Rewind for EOS-M crop" should work, but pick "EOSM" in the drop-down (not crop mode).  If you're not shooting crop mode, the "Original" build should work.

I developed my own pink dot remover.  It does crop mode only now, but could do others.  I open-sourced it and put it here.  It wore me out.  It could be vastly improved.

https://bitbucket.org/maxotics/focuspixelfixer

If you drag a RAW file onto it, it makes a copy, and then interpolate around the pixels using my own screwy logic

I had a bitch, bitch, bitch of a time figuring out how the RAW files were encoded.  So the source should save a you a lot of grief there if you want to use Python.  That was actually my first choice, but I couldn't get Pillow installed.   Unless you're super genius, you'll find little "simple" with this stuff.

Feel free to PM me with any questions, or ask them here.

Were could I get the new alpha versions for the EOS M now? I can´t reach the old sever any more.

I updated my starter set with a build from 11/09.  You can find a link to it on this page.  Jerykil warned the server might be down for a few weeks.  Hopefully it'll come back.

http://www.magiclantern.fm/forum/index.php?topic=8825.msg82945#msg82945

Or

http://maxotics.com/software/downloading.aspx?file=EOSM_MLStarterSet.zip

The 50D needs the card formatted Fat32, won't work with exFat, etc.

Yes, being able to use EF lenses is a big plus.  Does anyone know of a way to see the what the 50D's recording in crop mode.  It shoots in 3x but shows 5x on the screen.  Good for focusing, of course.  Am I missing something easy? 

Could always just buy a 5D3 with cash from a big retailer with a great return policy    "I just took it out of the box and put the battery in.. AND NOTHING!"

lol.. only kidding guys

What do you mean, when our wives go shopping they do it all the time  

[The bottom line is that there are few tools to work with RAW video data in the specialized way you're looking to do.]

If you figure out how to make a spot brush pattern in ACR and save it to use all over other footages - pls, share)
I have a concept, but dont know how to realise it - make a matte layer with all hot pixels, than use it in AE with WarpStabilizer + Synthesize edges to remake - http://blogs.adobe.com/aftereffects/2011/06/warp-stabilizre-redux.html. Maybe someone can help and say how to make such universal matte mask?

Hi Guys, sorry if you already know this.  For hot/dead pixel work you really want to, probably have to, worked with image data straight from the sensor.  On any software has de-bayered each pixel, you're basically now working with 4-pixel blocks, because each pixel channel, red, green or blue, has borrowed the information from the neighbor pixels and there are 2 greens to ever 1 red and 1 blue.

Unfortunately, only PhotoShop and Photivo (free) allow you to effectively work with images in this fashion.  And they are meant for still photography.  I don't know where to begin.  The bottom line is that there are few tools to work with RAW video data in the specialized way you're looking to do.

That's part of the reason why I created stuff to do this.  But even I have a bunch of things I wish I had time to do.  Like work on the moire issue in non-crop mode. 

Here is an article you both might read that is very informative and will help us work together if we continue on.

http://www.stark-labs.com/craig/resources/Articles-&-Reviews/Debayering_API.pdf

[the]

To be clear, I don't think there's anything wrong with offering money to fix problems. If it becomes a trend, then just make a pay-for-code forum section. Easy. But all of this nonsense about why ML needs to be polished and improved beyond whatever the people working on it are interested in doing is nothing but a collection of silly rationalizations from people who want more and can't do it themselves.

Film-making is the most team-oriented artistic endeavor.  Of the 100s of people you see listed on a film's credits, which ones are the ones with "silly rationalizations" of what they need, to better accomplish their goals?  Again, it's fine if the dev is only interested in saying doing an ETTR shot of a fire hydrant outside his door, but many people who come to ML have plans that require ML to fit into a team effort.  And, as anyone who has worked on a film-set, of any kind, will tell you, reliability is the most important quality you can have.  You can be the most genius writer, director, actor, grip, etc, but if someone forgets to charge the batteries because they were working on some new feature they thought would be useful that day you'll say, 'hey, no problem'? 

I feel, there is a bit too much dev "hero-worship" on these forums.  As someone who has done a lot of development I can tell you, at least for me, I was born with it.  There's a photo of me taking apart all the tea-bags in the house when I was barely 4.  I took apart everything.  And still do.  And for a while, like many of you, I thought other people were just lazy.  But after the experiences of showing up to a place where someone has forgotten the batteries say, I REALLY appreciate that person who can't program but can remember, and organize, 100, to me, excruciatingly boring details.  Though it has slowed me down working with such people, and is frustrating, the end work was 100x better.   

The OP who posted for the fix to the camera.  What do we know about him?  Perhaps all he thinks about is money and he wants to maximize the value of his camera.  Or maybe he is a technical dolt but has studied painting, and painted, for 12 hours a day and has a genius eye for composition.  He may one day shoot the next Citizen Kane.  Will it be on ML though?

To me, and just my opinion, everyone who posts or does anything helpful is on the SAME level as the devs.  Again, no matter how cool a feature one of the devs may get working, if I couldn't have done it without some person's tutorial who is the one who really got me there? 

What I have tried to do in these posts is explain why I am moving way from ML, and why I think others have/are doing the same.  Experiments are fun by yourself, but as soon as you start to work with others, certain things don't fly.  I wish they did.  I really do.  But experience teaches an unforgiving lesson.

Again and again again.  I can only describe my experience and thinking.  Everyone should do what they want with their free time.  If the devs and most users enjoy things the way they are, then by all means, rock on!  When I have free time and it's for fun, I'm here!

Simeon, I am happy you have been able to escape those Nikon fascists and speak freely here in the ML forum (JUST JOKING!).

Seriously, I was not arguing that developers should "pick the bugs off their fruit".  Only that, to continue that analogy, you may get sick of watching your co-workers start ignoring your fruit and never see people enjoy your garden as much as they could, if you put in a little bug-picking effort.  Yes, in the beginning of any hacking effort, you really can't worry about user happiness.  It should be enough that you enjoy it, and the few who share your interests.

But I believe, just an opinion, that ML has got to the point where its "coolness" is wearing off.  Again, no reason a developer should stop doing what they enjoy, even if only one person is following them.

I don't believe offering money for a feature is in any way disrespectful or wrong.  Devs don't have to take it up.  As Gibrain said, "Work is love made visible".  I believe the OP was expressing his love for ML in that way. 

Again, and again, and again,  It's a trade-off.  You may say you don't care if someone is following your Nikon work, let's say.  And you probably don't.  But you may care in the future, I am only pointing out that possibility--for whatever reason.

Finally, we rarely do something because we think of it out of thin air.  Most hacking is a reaction to some cool thing that CAN be done, but isn't.  Like RAW video from Canon cameras.  Where did we get that idea?  From people using Alexa and Red cameras, and such, which were borne of similar hackers with DSRL electronics.   

So hacking these cameras is partly social, though it doesn't seem like it.  Hackers may deny it, but I believe all hackers hack partly because they know it will please/impress others.  So where does the private challenge end and the social need begin?  Do we hack just enough to show it can be done, or do we hack enough so that some can use it on a professional shoot?  Or do we hack to get thanks on the forum?

No man is an island.

Hi hjfilmspeed, AFAIK, when a camera gets hot, and it does if the sensor is dumping images continuously, the heat causes sensels to get "hot" or act in strange ways, the same way heat debilitates everything.  The BMPCC camera, by the way, goes through batteries because it uses powered Peltier cooling on the sensor--or so I've heard.  In short, there's a good chance you're going to get hot pixels if you keep the camera on, shooting video.  The bigger question is can you be certain, even if you have a hot pixel list, that more pixels won't get hot as the camera continues shooting?

In normal video, the hot pixels are mostly smoothed out, but in astronomy, if that's what you're doing, it's not so simple.

You can just open a DNG in Photivo (for example), load the image at 1:1 in "unbayered" mode, get the position of each "dead/hot" pixel, and feed it into any DNG to TIFF conversion routines through a badpixellist.txt file where each pixel is in x,y, time (you can just put 0) coordinates.

Anyway, if there is a concrete need to create some sort of fix, let's talk, otherwise, let's just keep it on the list of things to do?

BTW, you can make your changes to your ML raw DNGs in ACR then export as DNGs then IF you need AMaZE/DCB etc you can import an ACR-tweaked DNG, set exposure settings to default and just play with the debayering algos then export to 16bit TIFF (wish there was video export in RT though)

Hope I don't sound dumb, but I'm confused on the ACR.  You mean Adobe Camera Raw, right?  And if so, why would you run DNGs through it again, isn't that what raw2dng is for?  What do you mean "ACR-tweaked." 

I generally used raw2dng, then if I want tiffs, UFRAW with options.

My problem was just opening a folder of DNGs.  I couldn't stop it from trying to create thumbnails of them all, which took forever.  Is that the problem you had with the new version?  Okay, I have 4.0.11.32...hmmm.  I have a decent i7 PC.  I have it set to open the smallest thumbnail, though I wish it wouldn't open any.

A cryptographic key used for code signing (no matter the underlying algorithm, in fact that's the whole point, it doesn't matter if the DRM is easily thwarted, it's still illegal to circumvent it) definitely falls in that category.

Absolutely, Canon is not obliged to use the strongest crytography available.  If you "break the seal" you're on your own.

Hi hjfilmspeed, so I can motivate myself on this.  WHAT are you trying to do?   I see the pixels.  Since they seem to be split among the 3 channels I assume it is an issue with those sensels at long exposure.  I don't believe RAW video outputs the dead pixel list from the camera.  So we'd need to make one up...and, urgh, where are we trying to go?  Thanks!

Thanks Andy.  I tried RAW Therapee but couldn't figure out a workflow where opening the folder of DNGs didn't slow it to a crawl.  Also, the batch processing I never got to work well.  Maybe I was just tired.  I looked, but couldn't find a good tutorial/info for processing ML dngs in Raw Therapee.  Do you know of one?  I agree, the image manipulation choices are awesome!  I'd actually rather write batch script to RAW Therapee than use Resolve manually! 

Thanks dmilligan, very interesting!  I'd just like to point out that it's probably not a good reason to say the keys are the intellectual property of Canon.  It's actually near impossible (if not impossible) to figure out a private key assuming it is of suitable strength (key length).  The whole idea of these keys is that they CANNOT be figured out so the public key can float freely.  So they either figured out the key because it was low strength OR someone at Canon slipped it to them.  I'd wager that if these are strong keys, the latter is what happened.  So the real risk of releasing these keys to any developer is that it might force Canon's hand into finding and prosecuting the person who released confidential information (which I don't think they want to do, presently). 

This is pertinent to the thread because the OP's request is essentially asking that any proof that a potentially stolen key was not used to unlock the camera.  I think that valid.  I hope I'm not going off-topic, but Alex's concerns that some development may lead to bricked cameras should not be dismissed lightly.  Should ML ever brick many of these cameras Canon would have no choice but to change the keys on all cameras leaving the factory.  I can't see them fixing any of the bricked cameras, or servicing ones that ran ML.   That they're letting cameras leave the factory with a broken, or hackable key probably already makes them very nervous, because, in a sense, it can be argued that the behavior implies that they approve of ML.

In short, the senior devs have made the right decision in limiting access.  But Alex's cautionary words should haunt everyone just the same.

True. Got some crop 50D footage with a touch of moire. It also helps to remember that different debayering methods can help a lot. I flit between ACR and Raw Therapee using AMaZE and DCB. You can never completely rid a shot of it but there are methods/tricks to concealing it. Aliasing is a bigger headache and you have to be very careful what you shoot.

Absolutely, on the de-bayering.  Have you tried Davinci yet Andy?  I suspect it has debayering that it tuned to video.  I think I'm going to run DNGs through Davinci in the future to DxHD and just edit from there.  I just wish I had more brain cells for that Resolve app.  I ran out a few years ago

Just want to point out that moire is a naturally occurring phenomenon and all cameras, in the right circumstances, will show it.

I'm going to continue working with the EOS-M which I accept will probably always be in Alpha.

I have lots of other stills cameras, so I no longer feel it makes sense to keep the 50D just for RAW video.  Without any time-line for a production version of ML on the 50D I'd rather put the money into a BMPCC.  I'm not saying this to discourage anyone from getting a 50D and using it for ML.  If I was young and this was my only camera I'd be quite happy.  Also, there are features that ML offers that the BM cameras do not.  However, I feel I should point out to anyone following this stuff what my final conclusion, for me, has been.  I just can't trust, when I take this camera out, that something won't happen as I expect it.  Last night I took it to shoot an event and though the RAW video recording worked fine, it didn't record a bunch of photos I took.  I'm sure there's a good reason.  But I'm tired of searching through threads trying to figure stuff out.

Has been a fantastic learning experience!  Definitely whet my appetite for a 5D3! running ML

Here's the camera if anyone is in the Boston area and wants it.

http://boston.craigslist.org/gbs/pho/4189491469.html

I'd get an account at box.com, or something like that, and I can download from there.

I hope I'm not the one that ends up doing this   If you can shoot a very short RAW file, just a few frames, at the same res as the corrupted file, I might be able to join the "good" footer from that to the end of the corrupted file (after removing the excess data of the last corrupted frame).  Again, hope someone else has a solution.  PM me if no one can help and you must have that footage.

I have a similar problem with the focus pixels in that I would like them interpolated around in the RAW file so that I can continue any post-processing path from there.  So that's what the app does, creates an exact duplicate of the RAW file and then loads and processes pixels in each frame.  So I could see this being used in that way, I'd just need for it to take a bad pixel list, like dcraw, and use that.  This is a Windows app by the way.  More important than the video file, is the DNG or RAW file.  If the RAW file is less then 100MBs and you can put it up, I could download that too. 

@hjfilmspeed, can you post a video of your clip and one of the DNG files?  I developed software that deals with focus pixels, so may be able to modify it for this problem.  Also, you can use ufRAW or dfRAW with hotpixellists, though it has a learning curve. 

here's a list of ML contributors sorted by most commits...

Thanks very much for taking the time to post the list.  Is A1ex and Alex the same person?  If so, then his taking a break is going to be felt!  I take it 1% isn't on list because he is running separate fork (another issue I know).  Am I totally off-base in thinking ML dev may slow down to a crawl without Alex?

try a newer build: http://tl.bot-fly.com
site moves on Nov 19th so there is going to be about a week of downtime there. i'll have to use dropbox or something gross

Do you want $ contributions for hosting fees?  What you're doing is fantastic and much needed!

... and b) to get to know why this is working on digic4-based cameras but not on 5d3/6d: If it's not for the lack of trying

How many devs have enough knowledge/experience to weigh in on this issue?  From my end, I can only see A1ex, 1% and g3gg0.  Alex seems to be taking a break (a very, very serious development IMHO).  g3gg0 seems dedicated to MLV.  Then there is 1%.  Who else has a reasonable chance of fixing this?  I don't follow the 5D3, so sorry if I don't know who the other devs are.