recently i've been trying hard to use hardware engines to process raw video for playback.
unfortunately i did not get far enough to say we have realtime playback.
but using an hardware engine, i was able to speed up processing at least a bit.
a example video with 600 frames, 24fps, 1920x1080, of 25 seconds length
takes 96 seconds using "color" and "all", which means it will play all frames with nice colors and no skipping.
using a tweak module "raw_twk (http://www.magiclantern.fm/modules/modules/raw_twk.mo/raw_twk.mo)" (see source in unified), which adds new methods for the latest
mlv_play to play raw with improved speeds.
warning:
a) use it on your own risk
b) only compatible with mlv_play
c) THIS IS REALLY EXPERIMENTAL, DONT DARE TO POST HERE IF IT RUINED SOME SHOOT YOU MADE!
d) 5D3 only yet
maybe this is stable enough to make use of it in ML core?
it is making use of "ProcessPathForFurikake" DSUNPACK/DARK/PACK16/WDMAC16 engines which receive 14bpp raw stream and align it correctly into 16bpp.
this eases up the way we can read out the pixel data and basically the most CPU expensive thing is rgb->yuv.
for this reason ive also improved rgb2yuv:
(http://s1.postimg.org/h842ihp4d/rgb2yuv.png)
left: original code, right: handcrafted assembly (there are also 6 words of constans not shown)
source (http://www.magiclantern.fm/modules/modules/raw_twk.c/raw_twk.c)
I ran a few tests using the experimental raw_twk module on my 5d3 with both .raw and .mlv clips
Clip Info: 248 frames, 23.976fps, 1920x1080, 10.3 second clip (playback mode: Color & All)
Before After
Playback in-camera 38 sec 22 sec 73% speed increase
Playback via HDMI 174 sec 97 sec 80% speed increase
Observations
- HDMI playback benefited a bit more than in-camera playback (an additional 7% speed increase)
- Relative speed increases were the same for both .raw and .mlv files
- The popup mlv_play menu was noticeably more sluggish to appear and dissappear (even more so via hdmi)
- I did not notice any other odd behavior with recording, playback, deleting or navigating clips
- No smoke or explosions (at least for me)
Overall a really nice improvement in playback speed :)
Wow the posted results above are really a noteworthy performance boost. Congrats.
Out of curiosity tried this on 5D3.123 (ProcessPathForFurikake is at 0xFF3CCC14) and it seemed to work fine and visibly faster than without the module. Also looked up the function from EOSM.202 (0xFF425150) but had problem with task creation.
yeah the EOS M is afair at the limit of tasks to be run in parallel
Very useful but does not work on NB 18July 5D3 133. Old API
If playback is set to "fast", do you get realtime playback?
Anyways, really nice improvement, I will try this on 1.1.3 if thats not a problem? As above post states an issue?
on 5d3 1.1.3 works fine, no problems so far. Really nice improvement :)
Installed on 1.1.3. on latest nightly. See significant improvement
Played a clip that took 64 seconds to play before raw_twk and after it was 35 secs.
But latest nightly wouldn't play .mlv files that were recorded before I updated to latest nightly. raw files played fine though.
Nothing to do with raw_twk though, it was not loaded when I started playing the files and noticed that the mlv files would not play, just mentioning.
Thanks a lot for this geggo! Really nice improvement. Haven't really done any recordings with the raw_twk loaded, but also just updated to latest nightly so will be hard for me to tell if I encounter problems, if it is the raw_twk module or latest nightly that causes issues.
Quote from: Kharak on September 04, 2014, 10:09:05 PM
But latest nightly wouldn't play .mlv files that were recorded before I updated to latest nightly. raw files played fine though.
Nothing to do with raw_twk though, it was not loaded when I started playing the files and noticed that the mlv files would not play, just mentioning.
Can you try manually deleting the associated IDX file and letting
mlv_play re-create it? I adjusted the IDX format recently, but the change should only affect "exact" playback. And even then, playback should still work. But, perhaps some bug slipped past my testing.
Yes, makes sense that the idx is interferring as these mlv files have been played before on the earlier nightly that I was using.
Thank you, I will try this.
Actually, more useful would be if you save a copy of the old IDX file instead of deleting it. If re-creating the IDX file fixes your issue, send me both the old and new IDX files. Thanks!
Good speed-up! But magiclantern-Nightly.2014Sep01.5D3113 and raw_twk.mo has a problem with 5x zoom ML Grayscale. White lines on the screen and camera hangs sometimes.
Quote from: vstrglv on September 05, 2014, 09:27:51 PM
Good speed-up! But magiclantern-Nightly.2014Sep01.5D3113 and raw_twk.mo has a problem with 5x zoom ML Grayscale. White lines on the screen and camera hangs sometimes.
I got same build, I'll see if I get same results with Crop mode and ml grayscale.
Edit:
Confirmed
What is confirmed? Is there this issue or not?
Quote from: vstrglv on September 05, 2014, 10:21:49 PM
What is confirmed? Is there this issue or not?
I confirm that I also get stripes in Crop mode with ML Grayscale ;)
Thank you!
Got error when trying to load module on latest nightly for 5d3 123 FW. Only tried it with zacuto evf attached.
Quote from: Kharak on September 05, 2014, 10:42:38 PM
I confirm that I also get stripes in Crop mode with ML Grayscale ;)
The same here. When i enter crop mode camera also crashes (and need to take battery out). 5D3 113
thanks for the feedback.
so you cannot enter crop mode with mlv_rec enabled?
preview mode is enabled, right?
I am able to enter and exit crop mode when ML grayscale is activated. But the entire screen is filled with tick white and black chunky pixel stripes that acts like heavy noise, seems like a failed attempt by ML grayscale to output the screen.
Quote from: kgv5 on September 06, 2014, 08:02:26 PM
The same here. When i enter crop mode camera also crashes (and need to take battery out). 5D3 113
The camera doesn't crash for me, but I haven't tried recording in crop mode like that. Did you record in crop mode?
1.1.3
Sep 1st Nightly
Quote from: g3gg0 on September 07, 2014, 01:00:46 PM
so you cannot enter crop mode with mlv_rec enabled?
Thats correct, mlv_rec (and mlv_play) enabled and raw_twk also enabled. When i press zoom button white mess appears, sometimes it crashes just after pushing once and sometimes when i press zoom second time. It crashes every time though and cannot leave crop mode, every time i need to take battery out.
Quote from: g3gg0 on September 07, 2014, 01:00:46 PM
preview mode is enabled, right?
Right, preview is on auto.
Quote from: Kharak on September 07, 2014, 09:06:47 PM
I am able to enter and exit crop mode when ML grayscale is activated.
The same, when ML grayscale is on. When its on 'auto' it crashes.
Quote from: Kharak on September 07, 2014, 09:06:47 PM
But the entire screen is filled with tick white and black chunky pixel stripes that acts like heavy noise, seems like a failed attempt by ML grayscale to output the screen.
The camera doesn't crash for me, but I haven't tried recording in crop mode like that. Did you record in crop mode?
1.1.3
Sep 1st Nightly
No, i did not try recording because of the crashes. I am on the same build: 2014Sep01.5D3113
Any more development for this? Would be interested to see if the 7d's dual digics could be used to speed up playback.
500D does not have ProcessPathForFurikake
550D has, so newer cameras also should
6D 113 FF5ABF84 ProcessPathForFurikake
7D 203 FF3A325C ProcessPathForFurikake
6D module loaded, 5x broken, 10x ok. DNG playback looks like 5x. Sometimes crashes after playing a file. Playback def faster.
7D module loaded, 5x ok, 10x ok, recording ok. Playback a bit faster. Taking a full silent pic output is black + hang, playback is black + hang.
Live previews do not work on either camera, neither color nor BW. I'm guessing MLV preview set to auto was entering this mode for 5X.
Something to do with defect correction processing? Maybe thats why zoom is broken? What else can it do?
Quote from: 1% on September 21, 2014, 07:53:14 PM
What else can it do?
I hope for raw addition (FA_SubtractTestImage, EekoAddRaw), and I bet a lot of people hope for 12-bit.
On 700D
1280x720@24fps color all
266 frames with = 34sec
266 frames without = 54sec
I guess next thing to try is to feed it hard coded YUV422_LV_BUFFER_DISPLAY_ADDR and see if does the preview.
Nope, no method lets it work in real time :(
Best I get is a single frozen frame. At least on 6D. I can try on 7D too but I suspect results will be the same.
Quote from: 1% on September 21, 2014, 07:53:14 PM
7D module loaded, 5x ok, 10x ok, recording ok. Playback a bit faster.
So there's potential, cool.
Quote from: a1ex on September 21, 2014, 09:18:19 PM
I hope for raw addition (FA_SubtractTestImage, EekoAddRaw), and I bet a lot of people hope for 12-bit.
Sounds like you are doing some experiments :P 12-bit raw would be so awesome.
Looks like its relatively easy to do in camera dark frame subtraction... at least on silent pics.
FA_SubtractTestImage pSrc=%#lx, pDark=%#lx, pDest=%#lx",0
Doubt it for mlv/raw, esp since the live preview doesn't want to work with Furikake.
Remove hot pixels with the defect correction engine?
FA_DefectsTestImage pSrc=%#lx, pDest=%#lx",0
wonder how fast this one is.
Right, but I'm interested in adding (and averaging) two or more pictures.
FA_DefectsMergeTestImage but no info on parameters or debug message.
could this be useful for dualiso preview / processing ? 8)
We already had the preview but parts of them would get burnt into the cr2.
a1ex has some other dual ISO preview code that didn't suffer from that problem.
http://www.magiclantern.fm/forum/index.php?topic=3904.msg127999#msg127999
Wow. More fantastic news! You guys are incredible!
raw_twk is awesome! Such a great enhancement on the play side.
But it is very unstable! In the beginning when I tested it, I wrote that I didn't have problems with going in to crop mode while raw_twk was activated, but now it just seems more and more random on how it works.
In the beginning when I tested this. It was only when ML grayscale was ON together with raw_twk and I go in to 5x mode that I would get black and white stripes all over the screen.
But now it seems going in to 5x with canon preview does so too at random times.. And seems if I press zoom in too quickly so that the overlay is not properly loaded, it will crash the camera and I need to do battery pull.
Also MLV play is not working too well. Sometimes it starts playing but the screen is black and I have to restart the camera and then it plays the file normally. And same with clicking too quickly, like entering play mode and exiting immediately will crash the camera and I need to do battery pull.
Raw twk has such a good performance that it's hard for me not having it activated, even though it makes the camera crash now and then.
Hope you make a more stable release of this G3ggo!
Sep 1st. Nightly.
113
It doesn't play nice with live view on digic V. Digic IV was better. I don't think its the module, just the canon engine if its used for defect correction, probably why it didn't work for live preview.
Was a noticeable speed difference for me. Great work! I can also report the black and white lines when trying to use crop mode. Also, if I am in photo mode with Live View off on my 5D3 and try to play a MLV video, it is just a black screen with all of the on screen displays working just as if it was playing but there is no picture.
yeah thanks for the feedback.
maybe some day we know how to program all the different hardware modules to get realtime playback ;)
Anything new here?
This module is really awesome, just a bit unstable! But I find myself having a hard time not using it, even though it can cause crashes now and then.
Just tried the Module under 1.2.3 on 5D3 -- apparently it only works on 1.1.3 since it says 'Module ini't failed'...
am I right about downgrading back to 1.1.3 for this particular module?
Thanks!
revival of this mlv_play speedup module (http://www.magiclantern.fm/modules/modules/raw_twk.mo/raw_twk.mo)
thanks to Greg's clean code of what DARK is doing (http://www.magiclantern.fm/forum/index.php?topic=13408.msg172108#msg172108), i implemented the 14->16 conversion directly into raw_twk.
this results in a much cleaner, non-firmware dependent code and might be more portable.
i am just using DARK in passthru mode, not subtracting anything. no need for this :)
works fine on 5D3.113, may work on others, but test on your own risk!
please report in if it works for you
I,d like to try this on a 550D and 7D but I get linking error
tcc: error: undefined symbol 'engio_write'
[E] failed to link to modules.
Will try it out on a 5D mark III when I get my hands on one soon.
I would also like to try on a 70D
ah sorry, yes.
i cherry picked https://bitbucket.org/hudson/magic-lantern/commits/91d96bf744b52b7d8addef8024f2c9b94a659c90 into my unified.
it is from branch raw-bit-depth (https://bitbucket.org/hudson/magic-lantern/commits/branch/raw-bit-depth)
You'll be able to use it in next nightly.
I can confirm it works on the 5D mark II using raw-bit-depth branch. Black and white playback is not too far from realtime, and colour is much faster too, but both have split jello image thingy going on. Really good improvement!
Hm, wasn,t b&w realtime already?
Sorry Danne, I had that idea stuck in my head from a few days of SD-CF adapter usage, but testing it right now on a 1066x card it is almost perfect, but at least for me on the 5D mark II, theres still a slight difference between 'all' and 'exact' in black and white, so not 100% real time. Yeah sorry about the black and white nonsense, I probably assumed that conclusion. But yes there is a massive difference in colour playback when enabling/disabling the module.
Error message for loading raw_twk.mo "tcc: error: underfined simbol 'engio_write'" for
magiclantern-crop3x.2016Sep05.5D3113 build.
Unfortunately crop.mo will be not there, i guess.
@g3gg0 -- This is probably a known issue or has been reported in the past but felt this was worth mentioning at the least.
While playing back 10/12-bit files in mlv_play (with raw_twk.mo enabled) which looks really nice and efficient.
However, on the 5D3 while going into 5x/10x zoom mode (not crop_rec.mo) for us to shoot larger resolutions such as 2.8k or 3.2k -- with the raw_twk.mo enabled the camera freezes up and gets stuck in 5x/10x mode forcing me to pop the battery out for a reboot.
Current workaround is to shoot and do your thing while having the raw_twk.mo DISABLED temporarily (unless you don't intend to shoot higher res) and then once all is done in which we can go ahead and ENABLE raw_twk.mo back on for a decent mlv_play previewing with 10/12-bit files. Is this a bug?
This needs more testing. Is centred 5x zoom mode from the crop_rec module in conjunction with raw_twk to blame or not? I tested a while ago and I had a freeze when previewing but after reboot all went to normal even with the raw_twk module included. I tried to reproduce but I couldn,t. However maybe better to put the centred 5x crop module to it,s paces over at the crop_rec thread and also test it cleanly without any other bleeding edge modules.
http://www.magiclantern.fm/forum/index.php?topic=17021.msg174918#msg174918
Seems the crashing with 5x zoom is related to raw_twk after all.
http://www.magiclantern.fm/forum/index.php?topic=5601.msg175803#msg175803
Also. Any chance to get the overlay buttons working(play,next, pause) on the canon eos m with raw_twk?
thanks, will think about how to do that properly
@g3gg0
Thanks for the raw_twk module. It works well on 10/12/14 bit files with the current build for 5D3-123 provided by dfort.
I love the ability to zoom while playing the video.
Great work!
On EOS M the mlv_play speed up looks like it's working.
Too bad the camera will crash ( "Threads failed to start"; needs battery pull ) when having the module loaded and trying to record RAW (mlv_rec): https://bitbucket.org/snippets/Licaon_Kter/EqkKy
its due to the EOS M having not enough free thread ids available.
would need some serious rework of code to save threads.
Quote from: g3gg0 on January 04, 2017, 12:20:48 PM
its due to the EOS M having not enough free thread ids available.
would need some serious rework of code to save threads.
NO idea what you are talking about.
That being said, those need to be hooked (reserved, whatever) all the time even if not mlv_playing anything? (don't mind me, this is way over my head)
mlv_play doesn't use any threads when it is not started.
however when starting playback it needs 3 threads (or tasks to name it like it is named in the OS).
so maybe some other module starts threads when not invoked - which ones do you have active?
Quote from: g3gg0 on January 04, 2017, 03:05:12 PM
so maybe some other module starts threads when not invoked - which ones do you have active?
so: dual_iso, ettr, fileman, mlv_play, mlv_rec, mlv_snd, pic_view and silent
/LE: Ok got it, tracked it down to mlv_snd, and it happens only if sound is actually recorded, yep this makes sense.
Just a suggestion: Is it possible for raw_twk to zoom to center of the image as default? Thanks for this great tweak!
Also note that the video playback limit is 3 files as reported here:
http://www.magiclantern.fm/forum/index.php?topic=5601.msg179644#msg179644