I'm not sure if this falls under the category of "real time video processing" or not... but hopefully not :)
I used the Sony A7S recently, and it has a "light trails" feature.
Basically what it does, is go into bulb mode and make it so that during each scan of the sensor, only pixels that are brighter than the previous scan are added to the photo, instead of globally adding brightness to each pixel.
So it starts as black (obviously), and the first scan gets the scene with your ISO/AP settings. Then each sequential scan will check for pixels that are brighter than the current compiled pixel set has, and brighten those pixels accordingly.
The photos I did with it are amazing and impossible without this sort of new-age tickery. Canon is always so far behind with the present day features in their prosumer lines. Hopefully you geniuses at ML can make it happen :)
Here is Sony's video that demonstrates the feature
https://www.youtube.com/watch?v=COAb3II1PSs
I would say this falls under the category of "things that can be done in post". Simply take individual exposures and then stack them in post with a maximum function. You can do this in Photoshop by loading all images into layers and then setting the blending mode to "Lighten". The results will be identical.
Features that are NOT possible - please don't request them (http://www.magiclantern.fm/forum/index.php?topic=950.0)
Quote
Things that can be done in post
Why spending development time on things like in-camera HDR? Magic Lantern is not a replacement for Photoshop ;)
dmilligan just stole my whole response away :)
I might add, you can do even better in post by doing something like this:
Stack the frames with an average() function (many astrophotography programs can do this) to reduce noise and get more DR, then blend that with a maxium() stack (as described above) using a mask based on the luminance. This will add back in the brighter star trails you get from using the maximum technique, while reducing noise in the rest of the frame. Now you get the best of both techniques: less noise from averaging and more prominent star trails from maximum
Sorry I guess I should have clarified... this feature is not related to star photos. It's more for similar things to car-light streaks that were demo'd in the Sony video.
As far as comparing this feature to post-compiling a series of photos...
Imagine you have a light trails (/painting) situation going on. Which is of uncontrollable fast moving items ( for instance cars, people, trains ). Let's assume time span needed to get the photo is 6 minutes ( as was the case in a photo I did with the Sony ).
So these fast moving items are racing across the screen. In the case of the normal series of photos, the camera would take a picture, delay a moment, then another picture and repeat.
So in the repeated photos style if you chose to take slow shutter photos for each shot.
You could take a 10 second photo, then the camera will shutter, buffer the picture, and start the next picture. In that moment you just missed movement of the scene. So there will be a gap of missed movement between each of those shots, even though they are long.
If you took fast shutter photos instead to try and make it so that there are even gaps.
You would have to shoot extremely fast in order to get any sort of steady (although still gaped) movement. let's say.. 30 frames per second, at least. And the last I checked... there are very few cameras able to get anywhere near that, let alone sustain that speed for 6 minutes.
The closest thing you could do to come even slightly close to this, is to film raw video ( extremely limited in resolution and a temperamental feature ), then adjust those frames, then compile.
As far as the noise reduction is concerned... that is already handled implicitly by the maximum(a,b) happening on every sensor scan cycle. So (without confirming) I'd guess most of the noise would already be taken care of, if not more, because the sensor would be getting far more data than any compilation of photos could ever get.
Trust me guys, I thought about it a lot :) The in-camera processing of each scan frame and controlling how it is added to the compiled final image is a huge feature that unlocks an unreal difference in abilities. Specifically this style :)
Sony does have a star trails mode too, but I haven't looked into it or used it yet.
So does anyone know if this feature is even possible with what Magic Lantern is able to access/accomplish?
thanks :)
remove the tag "invalid" please.
its probably something for FRSP - if we could reduce the lag. (yet this is the only problem atm)
keeping that in mind when working on FRSP
Plenty of people take images of car light streaks currently with current functionality. So I'm not really sure how this is useful.
sometimes you cannot go up with f/ and down with iso or cant afford a ND filter to expose 1 hour of bright stars.
I can certainly see the use of this, though if consecutive shots can be taken without any lag between the two then I don't think there's anything further for the camera to do that couldn't be done in post instead. Note that the 5D3 has some blending modes built in to the stock firmware already, though seamless light-trails currently can't be done because of the lag between each photo.
If blending was to be done in camera, presumably this would be the way to go: http://www.magiclantern.fm/forum/index.php?topic=13408.msg129483#msg129483 (http://www.magiclantern.fm/forum/index.php?topic=13408.msg129483#msg129483)
Audionut: It's useful because you don't overexpose the static parts of the image when blending in this way, compared to a single long exposure.
if you want to take 10 hours of 5-sec per frame footage in FRSP images into a MLV, that one picture will take a lot of space.....
so i can think of making it a good addition to "merge" frames
Interesting points guys... Definitely would be a nice addition to FRSP regarding "merge" frames!
I'll definitely follow this thread... Thanks @pursehouse!
Great, I'm glad you guys are interested in the feature :)
Yeah @chris_overseas , that EekoAddRawPath/TwoAdd using max(a,b) does seem to be exactly what I was talking about. I am assuming the image a/b they reference are single scans of the censor during the shutter being open?
And like @g3gg0 said, even if the frames were shot fast enough (which is impossible as far as I can tell), the space it would take to store them would be astronomical. Doing it in-camera before output to the file system opens up a whole new world of possibilities.
Perhaps if this feature was simply referenced as a "frame scan mode" (or something like that) with the user-available options listed on the page Chris linked ( additive, subtraction, max, min ). And if that existed, it could even open the door for having more complicated things like a time based mathematical curve option added to it to hide end points of motion... stuff like that. Endless possibilities :)
Also, this feature would make photos of lightning MUCH easier. I've used the motion trigger feature for lightning, and while being a great feature, it is still extremely difficult to get the right settings/setup for the changing and limited situation fast enough :)
Thanks everyone!
Oh, here is the train trails photo I took. This is an unedited export of the raw.
The shutter was open about 6 minutes, because that's how long of a time span there was between the two left/right trains passing me.
details: 15mm, f/4, ISO 100
http://garandy.com/t/japantrain1
As you can see, the area was pretty bright and extremely contrasted with the boarding platform lights on the right, the dark sky, and the building lights.. etc... And of course, the trains are moving pretty fast in the frame when they are passing near the camera.
I'm going to try a few light painting projects using this Sony feature as well. In semi-lit city areas, where hopefully the person-shadows will be a non-issue because of how the max() processing style
cool pic. did you use an ND filter?
If not, then how would you able to manage to keep it from getting overexposed during 6 minutes of open shutter?
especially @ f/4...
Thanks for sharing @pursehouse!
@DeafEyeJedi
no filters or anything. That's the beauty of the max(a,b) feature. It only brightens pixels that change between sensor scans, not brightening all pixels over time just because the shutter is still open. So, you could light paint in direct sunlight if you were able to darken your input enough ( probably need an ND for that ) and then had a light source bright enough. Now I have to try that out, light painting in the sun :)
Sounds marvelous!
Regarding the Sun painting -- You should use a ND or at least a UV filter otherwise...
[emoji6]
So what does it take to get this feature further into the likelihood of actually getting worked on by the people that know how to do it? :)
Nothing.
It's actually already been discussed before you even requested it:
http://www.magiclantern.fm/forum/index.php?topic=13408.msg129520#msg129520
and here:
http://www.magiclantern.fm/forum/index.php?topic=10176.msg98049#msg98049
Basically all this is going to do for you is save some storage space by not needing to save every frame. That's all we're really talking about here.
Right now if you want to do something like this, you need to capture every frame and do the blending in post.
Here's an example I did using raw video:
http://www.magiclantern.fm/forum/index.php?topic=12392.0
That was before FRSP (http://www.magiclantern.fm/forum/index.php?topic=12523.0), so I suppose you could do something similar at full resolution with FRSP, if you can make your shutter speed long enough, then the previous image should be done saving by the time the next image is ready, so there will be as small as possible time between frames (though it will still be significant, I think when I implemented a sort of FRSP 'live-view' for the luminance trigger (http://www.magiclantern.fm/forum/index.php?topic=12891.0), the best I could was about 90ms between frames IIRC).
I would love to play with the EekoAddRawPath stuff alas, I only have DIGIC 4 cameras:
Quote
Problem: these functions are only available on DIGIC 5 cameras.
@dmilligan
The delay between pics (any delay at all) makes normal merging photos style an unusable option for what I am talking about.
Things like waterfalls which repeat over again are simple to deal with, just like stars are easy because of their slow motion. This is for full motion scenarios. Even video (I assume 60fps) would be far to slow for this.
And although space is an issue, because a single raw file would be around 40mb, and 6 minutes of HD RAW 1920x1080x24fps video would be 29,880gb ( 74,700 at 60fps ). And that is a very low resolution for quality photos. Then you just spent your day setting up to take one photo per expensive card you have capable of handling that speed. And last I checked, 60fps RAW at 1080p was stretched up from a lower height and barely doable on my 5dmark3 + 1000xCF card.
And longer shutter speed without the maximum() processing goes against the effect of this feature.
But yes, the Eeko multiple features is exactly what I'm looking for, as we were talking about earlier in this thread :)
Quote from: pursehouse on January 28, 2015, 04:16:42 AM
The delay between pics (any delay at all) makes normal merging photos style an unusable option for what I am talking about.
Then what you are talking about is impossible. There's always going to be some finite delay between photos. However, I disagree that "any delay at all" makes what you are trying to do an "unusable option".
Quote from: pursehouse on January 28, 2015, 04:16:42 AM
But yes, the Eeko multiple features is exactly what I'm looking for
No it's not. I think you have a misunderstanding about what it does. All the EekoAddRawPath stuff does is merge two already captured frames (the main advantage is that it is hardware accelerated so it can do the merge faster than we could with the CPU).
Quote from: pursehouse on January 28, 2015, 04:16:42 AM
Even video (I assume 60fps) would be far to slow for this.
Why don't you stop making assumptions and actually try it.
It's weird that you are the only one arguing against this feature. There is a reason Sony has it in all their new cameras. And the feature concept is to add to sensor scans, not photos. And storage speed/size limitations make mimicking this feature in post impossible. I'm fairly certain I explained that multiple times.
@pursehouse: I totally agree with what dmilligan said here, and I bet the other devs do the same.
And he's not arguing against this feature, FYI.
ok so, you guys are the experts with this code base/exposed canon functionality.
Is it possible to override the way that the sensor adds to the image inbetween scans?
So that instead of the ( pixelResult += newScanPixel ) with I assume an internal maximum set during the process as opposed to when the buffer is output as a frame, instead it could be set to do ( pixelResult += max (pixelResult,newScanPixel) ) ?
If it's not possible to do that, then that answers my original question. Although I still consider the same ability in final frames to be worth it for many situations. Particularly the fact that the storage required for some projects makes them a nightmare to do out in the wild or while traveling. But, please, original question, can the sensor scan addition logic be altered?
thanks :)
There's no addition logic used when taking regular images (just hardware integration). So, there's nothing to alter.
take a bucket
hold it under a water-tap
let water flow into the bucket
now explain me how to do max(water, tap) of that constellation :)
haha @g3gg0
ok so... when a photo is taken, the shutter is opened ( or already open ).
The CMOS sensor is exposed to light.
The sensor software has it do a progressive scan across each pixel repeatedly, and continually add to the color value of each pixel.
So lets say during a 1s shutter time, it scans 1000 times ( depending on the sensor/resolution ).
Each of those scans add to each of the respective pixels over time to give the final resulting frame.
As far as I can tell, that scan and add code is driving the CMOS sensor. Is that code not accessible via the ML firmware? It seems that the dual pixel AF firmware upgrade from Canon a few years ago could have affected the CMOS' functionality.
Getting a final frame out of the CMOS is of course very speed limited in comparison to how fast the CMOS internally updates a scan of pixels in the future final frame.
I believe the some new CMOS chips are going to scan in groups of 4, but for now I believe everything on Canon is single pixel by pixel.
So I take it you are saying that there is no access to the CMOS scan processing code? That is a real shame :/
Well even if that's the case, it would still be huge to get the in-buffer merging done, it'd just be limited to whatever shots-per-second the sensor can handle outputting at the chosen resolution.
Just off the top of my head, the max()/avg()/min() ability in camera is huge for:
lightning photos ( leave in bulb mode in using max() until after lightning strikes )
waterfalls ( in avg or max mode, as dmilligan showed, but in full resolution raw file instead of low res flattened video )
light painting ( in max mode )
moving object omit/blur ( avg mode )
although I don't really do them, star trails would for sure be easier like this too ( like dmilligan brought up ), yes
as a traveling photographer, currently I have to do things like this with multiple dark ND filters, a large amount of time/testing/failure, and laptops in the wild are rarely an option, as well as storage space being fairly limited in comparison to dumping your 128gb card after every couple of tests, then going back after you compile and seeing things are totally wrong in post.
And photos like the one I shared of the trains, are flat out impossible without in camera processing ( assuming you want a high resolution raw ), due to card write speed and storage limitations for long length exposures.
Thoughts?
Quote from: pursehouse on January 29, 2015, 01:10:04 AM
The CMOS sensor is exposed to light.
The sensor software has it do a progressive scan across each pixel repeatedly, and continually add to the color value of each pixel.
That is not at all how CMOS sensors work. http://en.wikipedia.org/wiki/Active_pixel_sensor
Pixels (aka sensels) are basically just "photon buckets" (hence g3gg0's analogy). Photons land in them and are converted to electrical charge. The electrical charge accumulates during the exposure, then at the end of the exposure, the accumulated charge passes through an amplifier(s) to an ADC (analog to digital converter), which converts the analog voltage created by the charge to a digital number. We readout all the numbers for all the pixels and we get an image.
You can't control how photons accumulate on the sensels. They just accumulate, and this happens in the analog domain. You can only start doing custom processing once you readout the analog value of the sensel and convert it to a digital number. And doing that is actually capturing an image.
This is how all CMOS sensor work even Sony. Your 'light trails' mode on you Sony is just capturing normal images and then blending them on the fly. It is not some magical new sensor mode.
oooohkay... I had a very different understanding of how it worked. Guess I'm reading the wrong stuff. Lots to learn!
So, then the EekoAddRawPath is the way to go right? Each outgoing frame getting compiled into the resulting goal frame?
as alex somewhen pointed out, (iirc) bulb modes longer than 15 sec also do image addition with 15 sec exposures.
that's interesting... that's in the default Canon setup or is that something ML made happen?
the 15 second exposure thing isn't really the same as a high rate frame. Because then you are dealing with basically averaging all the pixels still ( in 15 second sets, which is a lot of motion ), instead of a max on something like... let's say 1/60s sets. So you'd end up with a faint blur of the moving items, instead of a prominent streak.
It would be even more interesting if you could have 2(or more?) goal frames being compiled during the process... with the a-b,a+b,min,max options listed on the other page... then a person could toy with those different compilations of light over time in post. Could be rather interesting for artsy style photos...
Oh and average(a,b) as an option too... having a combination of those photos to work with in post could open up entire new worlds of options in editing. Assuming it is possible to do 2 outbound images at the same time, which I have no idea if it is possible or not :)
update! haha
ok so, the Sony feature is doing frames at a minimum length of 1 second (max 15 sec ) at a time.
Which, I guess makes the overhead of processing the frames a lot easier on the hardware...
So what'ya say guys? A1ex and G3gg0's Eeko/TwoAdd setup to do this feature? Can it happen? Can I help? I code, I just haven't coded ML before. It looks like the demo code that A1ex setup is almost this feature, unless I'm misunderstanding?