Magic Lantern Forum

Developing Magic Lantern => Modules Development => Topic started by: g3gg0 on September 01, 2014, 02:23:08 AM

Title: [experiment] [5D3, others?] massive mlv_play speedup
Post by: g3gg0 on September 01, 2014, 02:23:08 AM
recently i've been trying hard to use hardware engines to process raw video for playback.
unfortunately i did not get far enough to say we have realtime playback.

but using an hardware engine, i was able to speed up processing at least a bit.

a example video with 600 frames, 24fps, 1920x1080, of 25 seconds length
takes 96 seconds using "color" and "all", which means it will play all frames with nice colors and no skipping.

using a tweak module "raw_twk (http://www.magiclantern.fm/modules/modules/raw_twk.mo/raw_twk.mo)" (see source in unified), which adds new methods for the latest
mlv_play to play raw with improved speeds.

warning:
a) use it on your own risk
b) only compatible with mlv_play
c) THIS IS REALLY EXPERIMENTAL, DONT DARE TO POST HERE IF IT RUINED SOME SHOOT YOU MADE!
d) 5D3 only yet

maybe this is stable enough to make use of it in ML core?

it is making use of "ProcessPathForFurikake" DSUNPACK/DARK/PACK16/WDMAC16 engines which receive 14bpp raw stream and align it correctly into 16bpp.
this eases up the way we can read out the pixel data and basically the most CPU expensive thing is rgb->yuv.

for this reason ive also improved rgb2yuv:
(http://s1.postimg.org/h842ihp4d/rgb2yuv.png)
left: original code, right: handcrafted assembly (there are also 6 words of constans not shown)


source (http://www.magiclantern.fm/modules/modules/raw_twk.c/raw_twk.c)
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: MA Visuals on September 01, 2014, 06:06:20 AM
I ran a few tests using the experimental raw_twk module on my 5d3 with both .raw and .mlv clips

Clip Info:   248 frames, 23.976fps, 1920x1080, 10.3 second clip (playback mode: Color & All)

                                       Before       After    
   Playback in-camera        38 sec       22 sec     73% speed increase
   Playback via HDMI       174 sec       97 sec      80% speed increase

Observations
- HDMI playback benefited a bit more than in-camera playback (an additional 7% speed increase)
- Relative speed increases were the same for both .raw and .mlv files
- The popup mlv_play menu was noticeably more sluggish to appear and dissappear (even more so via hdmi)
- I did not notice any other odd behavior with recording, playback, deleting or navigating clips
- No smoke or explosions (at least for me)

Overall a really nice improvement in playback speed :)
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: nikfreak on September 01, 2014, 07:55:50 AM
Wow the posted results above are really a noteworthy performance boost. Congrats.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: jpaana on September 01, 2014, 10:57:22 PM
Out of curiosity tried this on 5D3.123 (ProcessPathForFurikake is at 0xFF3CCC14) and it seemed to work fine and visibly faster than without the module. Also looked up the function from EOSM.202 (0xFF425150) but had problem with task creation.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: g3gg0 on September 01, 2014, 11:28:39 PM
yeah the EOS M is afair at the limit of tasks to be run in parallel
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: vstrglv on September 04, 2014, 04:52:37 PM
Very useful but does not work on NB 18July 5D3 133. Old API
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 04, 2014, 08:40:24 PM
If playback is set to "fast", do you get realtime playback?

Anyways, really nice improvement, I will try this on 1.1.3 if thats not a problem? As above post states an issue?
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: kgv5 on September 04, 2014, 08:59:51 PM
on 5d3 1.1.3 works fine, no problems so far. Really nice improvement :)
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 04, 2014, 10:09:05 PM
Installed on 1.1.3. on latest nightly. See significant improvement

Played a clip that took 64 seconds to play before raw_twk and after it was 35 secs.

But latest nightly wouldn't play .mlv files that were recorded before I updated to latest nightly. raw files played fine though.
Nothing to do with raw_twk though, it was not loaded when I started playing the files and noticed that the mlv files would not play, just mentioning.

Thanks a lot for this geggo! Really nice improvement. Haven't really done any recordings with the raw_twk loaded, but also just updated to latest nightly so will be hard for me to tell if I encounter problems, if it is the raw_twk module or latest nightly that causes issues.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: ayshih on September 04, 2014, 10:17:21 PM
But latest nightly wouldn't play .mlv files that were recorded before I updated to latest nightly. raw files played fine though.
Nothing to do with raw_twk though, it was not loaded when I started playing the files and noticed that the mlv files would not play, just mentioning.
Can you try manually deleting the associated IDX file and letting mlv_play re-create it?  I adjusted the IDX format recently, but the change should only affect "exact" playback.  And even then, playback should still work.  But, perhaps some bug slipped past my testing.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 04, 2014, 10:19:46 PM
Yes, makes sense that the idx is interferring as these mlv files have been played before on the earlier nightly that I was using.

Thank you, I will try this.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: ayshih on September 04, 2014, 10:28:30 PM
Actually, more useful would be if you save a copy of the old IDX file instead of deleting it.  If re-creating the IDX file fixes your issue, send me both the old and new IDX files.  Thanks!
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: vstrglv on September 05, 2014, 09:27:51 PM
Good speed-up! But magiclantern-Nightly.2014Sep01.5D3113 and raw_twk.mo  has a problem with 5x zoom ML Grayscale. White lines on the screen and  camera hangs sometimes.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 05, 2014, 10:08:54 PM
Good speed-up! But magiclantern-Nightly.2014Sep01.5D3113 and raw_twk.mo  has a problem with 5x zoom ML Grayscale. White lines on the screen and  camera hangs sometimes.

I got same build, I'll see if I get same results with Crop mode and ml grayscale.

Edit:

Confirmed
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: vstrglv on September 05, 2014, 10:21:49 PM
What is confirmed? Is there this issue or not?
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 05, 2014, 10:42:38 PM
What is confirmed? Is there this issue or not?

I confirm that I also get stripes in Crop mode with ML Grayscale ;)
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: vstrglv on September 05, 2014, 10:51:02 PM
Thank you!
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Markus on September 06, 2014, 04:15:11 PM
Got error when trying to load module on latest nightly for 5d3  123 FW. Only tried it with zacuto evf attached.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: kgv5 on September 06, 2014, 08:02:26 PM
I confirm that I also get stripes in Crop mode with ML Grayscale ;)

The same here. When i enter crop mode camera also crashes (and need to take battery out). 5D3 113
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: g3gg0 on September 07, 2014, 01:00:46 PM
thanks for the feedback.
so you cannot enter crop mode with mlv_rec enabled?
preview mode is enabled, right?
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 07, 2014, 09:06:47 PM
I am able to enter and exit crop mode when ML grayscale is activated. But the entire screen is filled with tick white and black chunky pixel stripes that acts like heavy noise, seems like a failed attempt by ML grayscale to output the screen.

The same here. When i enter crop mode camera also crashes (and need to take battery out). 5D3 113

The camera doesn't crash for me, but I haven't tried recording in crop mode like that. Did you record in crop mode?

1.1.3
Sep 1st Nightly


Title: Re: [experiment] [5D3] mlv_play speedup
Post by: kgv5 on September 07, 2014, 09:30:57 PM
so you cannot enter crop mode with mlv_rec enabled?

Thats correct, mlv_rec (and mlv_play) enabled and raw_twk also enabled. When i press zoom button white mess appears, sometimes it crashes just after pushing once and sometimes when i press zoom second time. It crashes every time though and cannot leave crop mode, every time i need to take battery out.

preview mode is enabled, right?

Right, preview is on auto.

I am able to enter and exit crop mode when ML grayscale is activated.

The same, when ML grayscale is on. When its on 'auto' it crashes.

But the entire screen is filled with tick white and black chunky pixel stripes that acts like heavy noise, seems like a failed attempt by ML grayscale to output the screen.

The camera doesn't crash for me, but I haven't tried recording in crop mode like that. Did you record in crop mode?

1.1.3
Sep 1st Nightly

No, i did not try recording because of the crashes. I am on the same build:  2014Sep01.5D3113
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: N/A on September 21, 2014, 03:32:31 AM
Any more development for this? Would be interested to see if the 7d's dual digics could be used to speed up playback.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Greg on September 21, 2014, 04:03:35 PM
500D does not have ProcessPathForFurikake
550D has, so newer cameras also should
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: 1% on September 21, 2014, 07:53:14 PM
6D 113 FF5ABF84 ProcessPathForFurikake
7D 203 FF3A325C ProcessPathForFurikake 

6D module loaded, 5x broken, 10x ok. DNG playback looks like 5x. Sometimes crashes after playing a file. Playback def faster.
7D module loaded, 5x ok, 10x ok, recording ok. Playback a bit faster. Taking a full silent pic output is black + hang, playback is black + hang.

Live previews do not work on either camera, neither color nor BW. I'm guessing MLV preview set to auto was entering this mode for 5X.

Something to do with defect correction processing? Maybe thats why zoom is broken? What else can it do?
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: a1ex on September 21, 2014, 09:18:19 PM
What else can it do?

I hope for raw addition (FA_SubtractTestImage, EekoAddRaw), and I bet a lot of people hope for 12-bit.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: mk11174 on September 21, 2014, 10:19:04 PM
On 700D
1280x720@24fps color all
266 frames with = 34sec
266 frames without = 54sec
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: 1% on September 21, 2014, 11:40:13 PM
I guess next thing to try is to feed it hard coded YUV422_LV_BUFFER_DISPLAY_ADDR and see if does the preview.

Nope, no method lets it work in real time :(

Best I get is a single frozen frame. At least on 6D. I can try on 7D too but I suspect results will be the same.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: N/A on September 22, 2014, 10:16:20 AM
7D module loaded, 5x ok, 10x ok, recording ok. Playback a bit faster.
So there's potential, cool.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: nikfreak on September 22, 2014, 10:45:06 AM
I hope for raw addition (FA_SubtractTestImage, EekoAddRaw), and I bet a lot of people hope for 12-bit.

Sounds like you are doing some experiments  :P 12-bit raw would be so awesome.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: 1% on September 23, 2014, 06:21:39 PM
Looks like its relatively easy to do in camera dark frame subtraction... at least on silent pics.

FA_SubtractTestImage pSrc=%#lx, pDark=%#lx, pDest=%#lx",0

Doubt it for mlv/raw, esp since the live preview doesn't want to work with Furikake.

Remove hot pixels with the defect correction engine?

FA_DefectsTestImage pSrc=%#lx, pDest=%#lx",0

wonder how fast this one is.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: a1ex on September 23, 2014, 06:27:59 PM
Right, but I'm interested in adding (and averaging) two or more pictures.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: 1% on September 23, 2014, 06:34:57 PM
FA_DefectsMergeTestImage  but no info on parameters or debug message.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: nikfreak on September 23, 2014, 09:42:39 PM
could this be useful for dualiso preview / processing ?  8)
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: 1% on September 23, 2014, 11:01:58 PM
We already had the preview but parts of them would get burnt into the cr2.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Audionut on September 24, 2014, 01:42:34 AM
a1ex has some other dual ISO preview code that didn't suffer from that problem.

http://www.magiclantern.fm/forum/index.php?topic=3904.msg127999#msg127999
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: kyrobb on September 24, 2014, 03:04:42 AM
Wow. More fantastic news! You guys are incredible!
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on September 25, 2014, 12:46:19 AM
raw_twk is awesome! Such a great enhancement on the play side.

But it is very unstable! In the beginning when I tested it, I wrote that I didn't have problems with going in to crop mode while raw_twk was activated, but now it just seems more and more random on how it works.

In the beginning when I tested this. It was only when ML grayscale was ON together with raw_twk and I go in to 5x mode that I would get black and white stripes all over the screen.

But now it seems going in to 5x with canon preview does so too at random times.. And seems if I press zoom in too quickly so that the overlay is not properly loaded, it will crash the camera and I need to do battery pull.

Also MLV play is not working too well. Sometimes it starts playing but the screen is black and I have to restart the camera and then it plays the file normally.  And same with clicking too quickly, like entering play mode and exiting immediately will crash the camera and I need to do battery pull.


Raw twk has such a good performance that it's hard for me not having it activated, even though it makes the camera crash now and then.

Hope you make a more stable release of this G3ggo!

Sep 1st. Nightly.

113
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: 1% on September 25, 2014, 03:31:46 AM
It doesn't play nice with live view on digic V. Digic IV was better. I don't think its the module, just the canon engine if its used for defect correction, probably why it didn't work for live preview.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: ChadMuffin on September 28, 2014, 04:33:27 PM
Was a noticeable speed difference for me. Great work! I can also report the black and white lines when trying to use crop mode. Also, if I am in photo mode with Live View off on my 5D3 and try to play a MLV video, it is just a black screen with all of the on screen displays working just as if it was playing but there is no picture.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: g3gg0 on October 03, 2014, 11:31:21 PM
yeah thanks for the feedback.
maybe some day we know how to program all the different hardware modules to get realtime playback ;)
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: Kharak on November 02, 2014, 04:38:18 AM
Anything new here?

This module is really awesome, just a bit unstable! But I find myself having a hard time not using it, even though it can cause crashes now and then.
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: DeafEyeJedi on November 02, 2014, 06:16:26 PM
Just tried the Module under 1.2.3 on 5D3 -- apparently it only works on 1.1.3 since it says 'Module ini't failed'...

am I right about downgrading back to 1.1.3 for this particular module?

Thanks!
Title: Re: [experiment] [5D3] mlv_play speedup
Post by: g3gg0 on November 16, 2016, 01:40:50 AM
revival of this mlv_play speedup module (http://www.magiclantern.fm/modules/modules/raw_twk.mo/raw_twk.mo)

thanks to Greg's clean code of what DARK is doing (http://www.magiclantern.fm/forum/index.php?topic=13408.msg172108#msg172108), i implemented the 14->16 conversion directly into raw_twk.
this results in a much cleaner, non-firmware dependent code and might be more portable.

i am just using DARK in passthru mode, not subtracting anything. no need for this :)

works fine on 5D3.113, may work on others, but test on your own risk!
please report in if it works for you


Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Danne on November 16, 2016, 09:35:40 AM
I,d like to try this on a 550D and 7D but I get linking error
tcc: error: undefined symbol 'engio_write'
[E] failed to link to modules.

Will try it out on a 5D mark III when I get my hands on one soon.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: andy kh on November 16, 2016, 09:58:53 AM
I would also like to try on a 70D
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: g3gg0 on November 16, 2016, 11:34:26 AM
ah sorry, yes.

i cherry picked https://bitbucket.org/hudson/magic-lantern/commits/91d96bf744b52b7d8addef8024f2c9b94a659c90 into my unified.
it is from branch  raw-bit-depth (https://bitbucket.org/hudson/magic-lantern/commits/branch/raw-bit-depth)
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: a1ex on November 16, 2016, 02:43:35 PM
You'll be able to use it in next nightly.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Ilia3101 on November 16, 2016, 05:43:58 PM
I can confirm it works on the 5D mark II using raw-bit-depth branch. Black and white playback is not too far from realtime, and colour is much faster too, but both have split jello image thingy going on. Really good improvement!
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Danne on November 16, 2016, 06:19:36 PM
Hm, wasn,t b&w realtime already?
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Ilia3101 on November 16, 2016, 10:26:26 PM
Sorry Danne, I had that idea stuck in my head from a few days of SD-CF adapter usage, but testing it right now on a 1066x card it is almost perfect, but at least for me on the 5D mark II, theres still a slight difference between 'all' and 'exact' in black and white, so not 100% real time. Yeah sorry about the black and white nonsense, I probably assumed that conclusion. But yes there is a massive difference in colour playback when enabling/disabling the module.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: vstrglv on November 23, 2016, 09:16:19 PM
Error message for loading raw_twk.mo "tcc: error: underfined simbol 'engio_write'" for
magiclantern-crop3x.2016Sep05.5D3113 build.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: g3gg0 on November 23, 2016, 10:02:33 PM
You'll be able to use it in next nightly.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: vstrglv on November 24, 2016, 08:05:22 AM
Unfortunately crop.mo will be not there, i guess.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: DeafEyeJedi on December 01, 2016, 09:51:03 AM
@g3gg0 -- This is probably a known issue or has been reported in the past but felt this was worth mentioning at the least.

While playing back 10/12-bit files in mlv_play (with raw_twk.mo enabled) which looks really nice and efficient.

However, on the 5D3 while going into 5x/10x zoom mode (not crop_rec.mo) for us to shoot larger resolutions such as 2.8k or 3.2k -- with the raw_twk.mo enabled the camera freezes up and gets stuck in 5x/10x mode forcing me to pop the battery out for a reboot.

Current workaround is to shoot and do your thing while having the raw_twk.mo DISABLED temporarily (unless you don't intend to shoot higher res) and then once all is done in which we can go ahead and ENABLE raw_twk.mo back on for a decent mlv_play previewing with 10/12-bit files. Is this a bug?
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Danne on December 01, 2016, 10:21:05 AM
This needs more testing. Is centred 5x zoom mode from the crop_rec module in conjunction with raw_twk to blame or not? I tested a while ago and I had a freeze when previewing but after reboot all went to normal even with the raw_twk module included. I tried to reproduce but I couldn,t. However maybe better to put the centred 5x crop module to it,s paces over at the crop_rec thread and also test it cleanly without any other bleeding edge modules.
http://www.magiclantern.fm/forum/index.php?topic=17021.msg174918#msg174918
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Danne on December 01, 2016, 02:56:57 PM
Seems the crashing with 5x zoom is related to raw_twk after all.
http://www.magiclantern.fm/forum/index.php?topic=5601.msg175803#msg175803

Also. Any chance to get the overlay buttons working(play,next, pause) on the canon eos m with raw_twk?
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: g3gg0 on December 01, 2016, 10:03:47 PM
thanks, will think about how to do that properly
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: RenatoPhoto on December 24, 2016, 05:10:02 PM
@g3gg0
Thanks for the raw_twk module.  It works well on 10/12/14 bit files with the current build for 5D3-123 provided by dfort.
I love the ability to zoom while playing the video.
Great work!
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Licaon_Kter on January 04, 2017, 04:31:49 AM
On EOS M the mlv_play speed up looks like it's working.

Too bad the camera will crash ( "Threads failed to start"; needs battery pull ) when having the module loaded and trying to record RAW (mlv_rec): https://bitbucket.org/snippets/Licaon_Kter/EqkKy
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: g3gg0 on January 04, 2017, 12:20:48 PM
its due to the EOS M having not enough free thread ids available.
would need some serious rework of code to save threads.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Licaon_Kter on January 04, 2017, 01:34:19 PM
its due to the EOS M having not enough free thread ids available.
would need some serious rework of code to save threads.
NO idea what you are talking about.
That being said, those need to be hooked (reserved, whatever) all the time even if not mlv_playing anything? (don't mind me, this is way over my head)
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: g3gg0 on January 04, 2017, 03:05:12 PM
mlv_play doesn't use any threads when it is not started.
however when starting playback it needs 3 threads (or tasks to name it like it is named in the OS).

so maybe some other module starts threads when not invoked - which ones do you have active?
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: Licaon_Kter on January 04, 2017, 05:07:43 PM
so maybe some other module starts threads when not invoked - which ones do you have active?
so: dual_iso, ettr, fileman, mlv_play, mlv_rec, mlv_snd, pic_view and silent

/LE: Ok got it, tracked it down to mlv_snd, and it happens only if sound is actually recorded, yep this makes sense.
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: RenatoPhoto on January 18, 2017, 05:46:51 PM
Just a suggestion:  Is it possible for raw_twk to zoom to center of the image as default?  Thanks for this great tweak!
Title: Re: [experiment] [5D3, others?] massive mlv_play speedup
Post by: RenatoPhoto on February 07, 2017, 02:56:44 PM
Also note that the video playback  limit is 3 files as reported here:

http://www.magiclantern.fm/forum/index.php?topic=5601.msg179644#msg179644