50D - 14bit raw video builds and test results

Started by GregoryOfManhattan, June 17, 2013, 01:53:22 AM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.


Andy600

Chipped adapters/lenses will show their default aperture. Mine defaults to f1.4 but it doesn't affect how auto ETTR works. i.e. it works the same for chipped and non-chipped lenses.

BTW I just added some frame settings from 1% to consts.h and got 6 stars back :) (it dropped to 5 when I added 'snap to 5x center' for some reason). All seems to be working well  8)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

GregoryOfManhattan

i've committed the FEATURE_LV_FOCUS_BOX_SNAP_TO_X5_RAW change - found that by comparing the 50D features to its digic iv cousin camera the 5D2 - you've confirmed it working, so lets get it into unified branch

also in 5D2 is #define FEATURE_LV_BUTTON_PROTECT // works even without the Q menu present
which doesn't compile cleanly unless we get the correct values for
//~ #define BGMT_Q 0xE
//~ #define BGMT_Q_ALT 0xE
(or see more of how this works for the 5D2.212 )

do you or 1% have that code somewhere visible on the net?
if you are getting values for constants from a legitimate dump of the firmware, they should be fine.
again comparing with the 5D2.212/constants, we see that
50D.109 is missing the HIJACKs and it looks like many of the already commented out YUV422_HD_ lines should be deleted as the constants are not used elsewhere in the code.

RE 6 star vs. 5 star - is stable recording speed a better metric than counting stars? of course loosing a buffer (if its really lost and not a display glitch) would be expected to decrease maximum stable recording speed.

absent deeper knowledge of magic lantern, i'd continue to compare 50D.109 with 5D2.212 to see what is different in the code.  the .h files are the lowest hanging fruit because adding correct values for constants should simple enable other parts of the magic lantern code.

can we find anything else which should be enabled.
and what about QR_MODE - we still need the correct values in src/raw.c


1%

QR mode values are easy.. just take a raw pic and look at meta data.

Code is up in the 6D repo. A1ex said that iso might still be int16, even though it doesn't look like it when read as int 16, I'll try both ways.

To find values for that I'd use event spy. I think middle click on the joystick is q?

Buffer is just buffer, 5 vs 6 doesn't make much diffference..  but I was getting a 29MB block before and now I'm not so I wonder what in the canon menu produces it.

"q" press looks to be 1E, release is 15. Same in play mode. 15 seems generated by more than 1 button, don't use it so much.

The commented out stuff looks to be from the old method of guessing hd/lv size not using the registers, I'd just leave as a reference... don't comment out the HD/lv buffer addresses.

KahL

Quote from: Andy600 on June 18, 2013, 11:37:22 AM
Upload a new build that fixes ETTR, ISO and Exposure Override issues that I was experiencing with manual lenses (thanks a1ex :) ):

https://bitbucket.org/andy600/andy50d/downloads/Andy600ML_June18th_50D.109.a43e030494e4_fix_UNIFIED.zip

This works and is up to date :)

Awesome sauce. Just tested this build and the two crucial functions that I needed are now added. Having the SET button to record, just the same was as it would for h264 video, and now finally it saves the RAW enabled function without my having to turn it on every time the camera goes to sleep or gets shut off.

Andy600

Quote from: KahL on June 18, 2013, 05:40:01 PM
Awesome sauce. Just tested this build and the two crucial functions that I needed are now added. Having the SET button to record, just the same was as it would for h264 video, and now finally it saves the RAW enabled function without my having to turn it on every time the camera goes to sleep or gets shut off.

Outdated - Here's one with all that and some bug fixes plus 5x crop snap to center ;)

https://bitbucket.org/andy600/andy50d/downloads/Andy600ML_June18th_50D.109.d55f33a1d395_Snap_to_5x_center.zip
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Andy600

I've uploaded 2 raw modules for testing purposes to my repo. (50D ONLY. May or may not work on other cameras)

The first (fast_raw) is 1%'s code and features extended functions: Global Draw can be disabled when hitting record (so no having to do it manually) and you can also delay the start of recording by 2, 4 or 10 seconds. 1% built the module based on multiples of 512. This method of calculation, to the best of my understanding,  reduces the load on the CPU as it keeps frame dimensions rounded-up to multiples of 16 meaning the CPU doesn't have so much math to process. The upside is that you should get faster write speeds = more/larger frames. The downside is max frame width in normal mode is reduced from 1584 to 1534 (i.e. 512x3 = 1534). This is probably more beneficial for slower cards and for achieving more frames in 5x crop mode.

The second (andy_raw) is basically the same as the first but uses the current method allowing for full 1584/2000px width recordings. (BTW, the name  ::) is just so I can identify it ATM. I'll change it to something else later)

Tip: both modules can co-exist but be sure to only have one enabled at any one time. To use, just drop them into your MODULES folder. Check they are loaded in the modules menu.

Tip: You can enable modules to individually, automatically load on boot. Select a module (modules menu), press Q (press joystick), SET - ON/OFF ;)

Tip: I'm getting fast write speeds when SRAW2 is selected in the Canon menu. I'm not sure if the gains are real but it seems to be 3-4MB/s faster write speed for both modules.

Q. What shall I do with my current raw_rec module?
A. Leave it where it is, just don't enable it at the same time as another raw module. If you find one module works well for you I suggest either deleting the others from your modules folder or setting auto-load OFF for the ones you don't need.

Q. Where do I download?
A. Read the first post

@1% - hope you don't mind me renaming your module. I couldn't use 1% because of the '%'. I've added an extended credit to the module too ;)


Update: I don't think the raw configs are working correctly so the modules are loaded but in an off state when booting.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

GregoryOfManhattan

today's unified build
https://bitbucket.org/GregoryOfManhattan/magic-lantern/downloads/ml-2013Jun19.50D.109.go.unified.7eee493f6fc9.zip

Andy600 - sounds very interesting - do you have any statistics on your two modules
it's hard to measure the 2 MB/s differences.

can you record more frames at 1920x1080 in crop/zoom mode ? (more than 140 ?)
maximum stable continuous recording can you get stable at 1920x880 or larger sizes?

see 1% posting to his 6D repository https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/a7df9e2b29f5a4cb4e8c6499a17daa8d992b7226/platform/50D.109/?at=unified

looks like a great deal of progress for the 50D.
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/a7df9e2b29f5a4cb4e8c6499a17daa8d992b7226/platform/50D.109/?at=unified

Andy600

@Gregory - I'm getting about 300 - 360 frames 1920x872 (Andy raw) and about 100-120 frames at 1920x1072 on a 600x Transcend 16gb card. That's more than I was getting with raw_rec. No problems recording continuously at 16:9 or 5:3 in normal resolution. I think I may have to adjust the frame dimensions a bit. Not been able to test much today.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

a1ex

If you look at commit history, you will notice that 512-byte rounding is an older method that I've borrowed from CHDK. Current ML uses 4096-byte rounding, and the rationale can be found in my source code comments.

Sorting and/or splitting the buffers can make a difference, but there's no one-size-fits-all solution with fixed buffers. See here for some research on buffering strategies:
http://www.magiclantern.fm/forum/index.php?topic=5582.msg52753#msg52753

Andy600

Quote from: a1ex on June 19, 2013, 08:43:06 PM
If you look at commit history, you will notice that 512-byte rounding is an older method that I've borrowed from CHDK. Current ML uses 4096-byte rounding, and the rationale can be found in my source code comments.

Sorting and/or splitting the buffers can make a difference, but there's no one-size-fits-all solution with fixed buffers. See here for some research on buffering strategies:
http://www.magiclantern.fm/forum/index.php?topic=5582.msg52753#msg52753

Thanks a1ex, I'll have a play :)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

akumiszcza

I've tested the latest builds here (fast-raw test build from yesterday) especially for auto-ettr, as my card is too slow to make serious raw video tests (although I like the fact that when I see the speed in green while choosing the resolution, it really does not lose frames).

1. In Liveview M mode it works quite well. It changes both ISO and time, the aperture is constant.
2. Sometimes when I do a photo after Auto-ETTR (in liveview M) it is greatly underexposed. If I reset configuration and set everything again, it goes back to normal. Weird.
3. In Liveview Av mode it seems to not change ISO, just +/- EV. I guess it's +/- 2 EV max then.
4. In photo mode the double half-shutter and set key options open liveview for a moment, then ML does Auto-ETTR, and photo mode comes back.
5. Sometimes this Auto-ETTR in photo mode takes a long time, and sometimes it's very quick.
6. Sometimes it's hard to trigger double haf-shutter, I need to click half shutter multiple times (it seems it doesn't work fine if the camera starts autofocusing in the meantime).
7. Autosnap and full-time Auto-ETTR seem to not work in photo mode. As I understand, autosnap should trigger another photo itself and the full-time mode should trigger Auto-ETTR after one takes an underexposed photo. Both things do not happen on my camera.

KahL

Alright so during a fashion shoot yesterday, I was getting pink frames all over the place. The last time this happened, I was shooting w/ the 60D in raw as well. Both environments here fairly hot, especially yesterday's ext. shoot. Does heat or battery power have anything to do w/ this? While I was using a full battery, it was a low MAH (about 1400), while the other was 1700 and zero pink frames.

Just trying to narrow this down. Maybe I'm off somehow.


1%

QuoteCurrent ML uses 4096-byte rounding, and the rationale can be found in my source code comments.

Mine does too, just kept the 512 rounded sizes since they were working esp. well. Probably can add back some regular ones.

The unified module from my repo works on all cameras but the hack isn't in the module.

ETTR is a little flaky on 50D because you need FPS override + expo override all the time pretty much. It works perfectly on other cameras but on 50D I get some of the same issues. Ie. button doesn't trigger it... just go in and out of play mode and it comes back.









GregoryOfManhattan

today's build  from the unified branch
https://bitbucket.org/GregoryOfManhattan/magic-lantern/downloads/mlmod-2013Jun20.50D.109.go.unified.1b42291d5085.zip


@akumiszcza - thank you for your detailed notes - i need to review the ETTR functions myself as i haven't used them for several days.

#KahL Kahleem - you've got an art video not a fashion shoot. which build led to that result?

GregoryOfManhattan

50D new unified build available:
https://bitbucket.org/GregoryOfManhattan/magic-lantern/downloads/mlmod-2013Jun21.50D.109.go.unified.9f26e15d9146.zip
many changes to modules/raw_rec.c and src/raw.c
all cameras join the 50D in being MOS

can this build hit 80MB/s ?

JulianH

I can't seem to get good speeds with that build Gregory. 65-68MB/s in all modes and all resolutions I've tried... Komputerbay 64GB 1000x.

I missed the last few builds since I was away for a few days. I might be doing something wrong. Will check some of them to see if speeds get better.

GregoryOfManhattan

posted a build with the latest code in unified branch
https://bitbucket.org/GregoryOfManhattan/magic-lantern/downloads/mlmod-2013Jun21.50D.109.go.unified.7b79e51d4d06.zip

this code will result in black borders for
regular silent pictures 74 pixels,
zoom mode silent pictures 64 pixels, and
regular mode video 74 pixels
zoom mode video will be OK.

you can see this in the image below or try it for yourself with the build.


looks like the procedure that we used on june 5th was incorrect insofar as it used the same value for zoom mode silent pictures and video - 64 pixels left offset. is there a conditional which could be used to distinguish zoom photo mode from zoom video mode? is so then it could be added to the code, though at the time, i didn't see any degradation to the zoom mode video image from the extra skip left.


skip_left   =  zoom ? 64: 74;

if mv1080crop or something like it, is defined for 50D,

skip_left   = mv1080crop ? 0 :  zoom ? 64: 74;


please note there are multiple builds being used along with creative users who combine an autexec.bin from here with a 50D_109.sym from there and perhaps a raw_rec.mo from over yonder - so its easy to confuse reported results.



a1ex

So, the borders in zoom mode depend on whether H.264 is enabled?!

So far, zoom was a completely separate video mode (that's why it's tested first).

Andy600

@Gregory - I just took 4 silent pics with movie mode on/off and normal/zoom. Non of them exhibited black borders although with movie mode off the frames are a little pink.

Then I tried some raw video (normal and zoom) and it has no black borders and no pink frames with these settings.

This is on a compile of the Unified code (must be the same as yours?)

BTW, I'm getting black borders on everything with a Tragic Lantern compile but that's not relevant atm until we are 100% correct with Unified skips etc. I'm basing my findings only on Unified code.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

akumiszcza

I got 4 crash dumps with mlmod-2013Jun21.50D.109.go.unified.9f26e15d9146.zip build. It was during filming in raw and pressing arrows on joystick to check if changing aperture works.

CRASH00.LOG
ASSERT: IsChunkSignature( hPos )
at PackMemory\PackMem.c:987, task RscMgr
lv:0 mode:3


Magic Lantern version : 2013Jun21.50D.109.go.unified.9f26e15d9146
Mercurial changeset   : 9f26e15d9146+ (unified) tip
Built on 2013-06-21 18:24:10 by [email protected].
Free Memory  : 284K + 3449K

CRASH01.LOG
ASSERT: IsSuiteSignature( hSuite )
at PackMemory\PackMem.c:986, task RscMgr
lv:0 mode:3


Magic Lantern version : 2013Jun21.50D.109.go.unified.9f26e15d9146
Mercurial changeset   : 9f26e15d9146+ (unified) tip
Built on 2013-06-21 18:24:10 by [email protected].
Free Memory  : 284K + 3454K

CRASH02.LOG
ASSERT: IsSuiteSignature( hSuite )
at PackMemory\PackMem.c:986, task RscMgr
lv:0 mode:3


Magic Lantern version : 2013Jun21.50D.109.go.unified.9f26e15d9146
Mercurial changeset   : 9f26e15d9146+ (unified) tip
Built on 2013-06-21 18:24:10 by [email protected].
Free Memory  : 284K + 3458K

CRASH03.LOG
ASSERT: IsSuiteSignature( hSuite )
at PackMemory\PackMem.c:986, task RscMgr
lv:0 mode:3


Magic Lantern version : 2013Jun21.50D.109.go.unified.9f26e15d9146
Mercurial changeset   : 9f26e15d9146+ (unified) tip
Built on 2013-06-21 18:24:10 by [email protected].
Free Memory  : 284K + 3463K

Andy600

A bit OT

@alex - Is the demosaic part purely down to the app that converts the raw files?

Maybe this info is useful?: http://www.libraw.org/articles/bayer-moire.html He seems to conclude LMMSE is the better algorithm. He also has some useful downloads http://www.libraw.org/download#stable

and this: http://www.ipol.im/pub/art/2011/g_zwld/

Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

1%

Yea, I'm using unified skips from a1ex so if your getting black borders must be wrong...

GregoryOfManhattan

black point disease is not always cured by blood letting the skip values - applying leeches can make things worse and actually spread bad blacks to other areas.

@a1ex - do not know if h.264 enable or not makes a difference.

@1% - values i provided are correct.

@akumiszcza - thank you for details - i am getting glitchy frames (silent pics) from both of yesterday's builds though i haven't had a crash myself yet. recommend you use an earlier vintage build for practical shooting.

@Andy600 -  you can be sure about the build and code sync by looking in the magic.cfg - it tells you what build is on which card - should always have the commit id.  same information is at the bottom of the info screen.
that build id should match one of the commit id's on https://bitbucket.org/hudson/magic-lantern/commits/branch/unified
so 9f26e15d9146 in the bug report from @akumiszcza matches commit 9f26e15
https://bitbucket.org/hudson/magic-lantern/commits/9f26e15d91464033c14d1633bf31cd13c0b100a3?at=unified


Andy600

@Gregory

Yes, I don't have the build I shot the video with or the one I checked DNGs with anymore but all are derived from Unified build with only unified commits. I use TortoiseHG to pull, commit and then console to hg update tip.

For instance: this is my current build

# Magic Lantern v2.3.NEXT.2013Jun22.50D109 (363a968c2785+ (unified) tip)
# Built on 2013-06-22 13:41:16 by magiclantern@magiclantern-VirtualBox
# Configuration saved on 2013/06/22 15:45:34

I have a fork of Tragic Lantern for testing but it's a totally separate repo and there is no crossover
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com