Author Topic: 12-bit (and 10-bit) RAW video development discussion  (Read 400636 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1650 on: April 29, 2018, 07:45:18 PM »
That doesn't sound "normal" and Jan 31 isn't the latest build for the 6D. Try the latest crop_rec_4k build from the experiments download page.
5D3.* 7D.206 700D.115 EOSM.203

keepersdungeon

  • Member
  • ***
  • Posts: 147
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1651 on: May 01, 2018, 11:40:55 AM »
Sorry for the late reply, I meant latest for the 10-12bit Raw video.
I tried to remove all and re-extracting a clean version. It seemed to "fix" it now it doesn't shift to the left, it zooms a bit more when I hit record than what u see on the LV before u do so but at least it's centered.
I did get couple of error msgs but they don't seem to affect anything:
1- I use to get this before, it kinda happen the 1st time when u turn on the can and u record something it says "Failed 'WBAL'. queued: 7 failed:1 (requeued)"

2- this one is new "Raw error" It happened twice but I couldn't replicate it and I'm not sure why it happened. 1st time I saw it was when I was tweaking the FPS override to 24 after 5x zoom crop. But like  I said I could replicate it.

It's kinda hard to debug without any logs. If there's something else I can do to help let me know

Sent from my Pixel using Tapatalk


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1652 on: May 02, 2018, 06:26:12 AM »
Ok--got it.

I would recommend using the crop_rec_4k builds on the 6D. It has everything that is in the 10/12bit build and a whole lot more. The 6D has relatively recently started working with lossless compression so you might want to try that out too.
5D3.* 7D.206 700D.115 EOSM.203

keepersdungeon

  • Member
  • ***
  • Posts: 147
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1653 on: May 02, 2018, 02:41:22 PM »
Ok--got it.

I would recommend using the crop_rec_4k builds on the 6D. It has everything that is in the 10/12bit build and a whole lot more. The 6D has relatively recently started working with lossless compression so you might want to try that out too.
Alright I'll give it a try. Thanks again @dfort
Any idea what is this WBAL error?

Sent from my Pixel using Tapatalk


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1654 on: May 02, 2018, 04:01:18 PM »
Any idea what is this WBAL error?

Need more information. What are your settings? Are you using mlv_lite? Auto WB? Can you get a screenshot of the error?
5D3.* 7D.206 700D.115 EOSM.203

keepersdungeon

  • Member
  • ***
  • Posts: 147
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1655 on: May 03, 2018, 09:41:32 PM »
Using mlv_rec, not in auto WB, recording in 12bit, 1824*1094, 24fps, no FPS override, in M mode, that's pretty much it. Sorry the image is a bit blurry but u can read the msg at the bottom.
It happens mostly on the 1st time u start recording after turning on the Cam.


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1656 on: May 03, 2018, 11:47:04 PM »
So this is on the 6D running the latest crop_rec_4k build? We're getting off topic. I'll point to this from the 6D topic because it hasn't been reported on any other camera that I know of.
5D3.* 7D.206 700D.115 EOSM.203

keepersdungeon

  • Member
  • ***
  • Posts: 147
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1657 on: May 04, 2018, 10:16:23 AM »
So this is on the 6D running the latest crop_rec_4k build? We're getting off topic. I'll point to this from the 6D topic because it hasn't been reported on any other camera that I know of.
No this is from the 10-12 bit build of Jan 13. I did not try the crop_rec_4k build yet. I hesitate on posting on the 6D topic or here seen that this is the build I'm using . And my apologies if I went off topic or made it confusing

Sent from my Pixel using Tapatalk


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1658 on: May 04, 2018, 05:35:26 PM »
You're not off topic if you are reporting an issue with the 10-12 bit branch but could you please try to reproduce the issue using the crop_rec_4k branch?

Another thing you might try is to reset settings to factory defaults and see if the issue is still there.
5D3.* 7D.206 700D.115 EOSM.203

blubbblubb

  • New to the forum
  • *
  • Posts: 10
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1659 on: May 06, 2018, 07:27:55 PM »
I tried the 10/12 Bit one and i cant reproduce the WBAL error (altough i had to use a lower resolution to record more than just 1-2 seconds), so maybe not a error affecting all 6D's?

zootrope

  • New to the forum
  • *
  • Posts: 5
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1660 on: May 10, 2018, 06:30:55 PM »
70D
Maybe some screenshots are useless, i don't know...
(i've used this ML version from dfort : crop_rec_4k.57614b3.2018Mar14.70D112)

Raw Rec @10bit @1080p (1824x1026)







Raw Rec @10bit @720p







Raw Rec @12bit @1080p (1824x1026)







Raw Rec @12bit @720p







Raw Rec @10bit 3x3 @1080p (1824x1026)







Raw Rec @10bit 3x3 @720p







Raw Rec @12bit 3x3 @1080p (1824x1026)







Raw Rec @12bit 3x3 @720p







Raw Rec @10bit x5 @1080p







Raw Rec @10bit x5 @720p







Raw Rec @12bit x5 @1080p







Raw Rec @12bit x5 @720p






delete my

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1661 on: May 10, 2018, 06:56:16 PM »
@zootrope - Not sure what problem you are pointing out. Are you able to record video files with that build and are the MLV files valid? Something you might try is the magiclantern-raw_video_10bit_12bit.2018Jan31.70D112.zip build that is posted on the experiments downloads page.
5D3.* 7D.206 700D.115 EOSM.203

zootrope

  • New to the forum
  • *
  • Posts: 5
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1662 on: May 10, 2018, 08:35:38 PM »
@zootrope - Not sure what problem you are pointing out. Are you able to record video files with that build and are the MLV files valid? Something you might try is the magiclantern-raw_video_10bit_12bit.2018Jan31.70D112.zip build that is posted on the experiments downloads page.
Maybe i'm doing wrong...
my previous post was a reply to this request : https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632

Just wanted to help  :-X

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1663 on: May 10, 2018, 09:02:48 PM »
Got it -- Your tests do help. Looks like you did it right.
5D3.* 7D.206 700D.115 EOSM.203

reddeercity

  • Hero Member
  • *****
  • Posts: 1667
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1664 on: May 17, 2018, 07:30:32 AM »
Try to find the default_raw_buffer for 5d2 , compiled the allocate-raw-lv-buffer branch
Quote
compile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (config-defines.h)
I did this , but I'm not getting the same info as posted here dfort https://www.magiclantern.fm/forum/index.php?topic=5601.msg197778#msg197778



and this 

Does this work on digic 4 of is this a digic 5 thing only ?

I record small raw video took some silent picture , the only thing show up is (c17ec) see second screen shot .
I'll look over the code and see if I make a mistake , oh yea I open the console to see what I'm getting
any help would great .

reddeercity

  • Hero Member
  • *****
  • Posts: 1667
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1665 on: May 17, 2018, 08:01:18 AM »
from a1ex
https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632
Quote
I've got a hypothesis about the raw buffer on EvfState models - full chunks (size = srm_buffer_size) are allocated for LiveView use (SRM_AllocateMemoryResourceForLiveViewYuv) and then divided later; the addresses end up in the ADST_GetMemAddrArray (ADdress STore?) at some point,

would this be the same on d4 -- full chunks (size = srm_buffer_size) ?
in my case
Code: [Select]
chunk #7 size=1a2bfc0 (c17ec)
chunk #8 size=179ffc0 (c17ec)
so is "c17ec" the srm buffer size or is this nothing to do with it ?

edit: I know now why it didn't work there no raw_slurp enabled in allocate-raw-lv-buffer branch for d4/5d2 , I'll have to manually input the code .
Tomorrow's task :)

reddeercity

  • Hero Member
  • *****
  • Posts: 1667
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1666 on: May 22, 2018, 06:44:12 AM »
Using my modified 10-12bit for 5D2
Code: [Select]
Magic Lantern Nightly.2017Dec10.5D2212
Camera   : 5D2
Firmware : 212
Changeset: ee95e8825183+ (raw_video_10bit_12bit_LVState-wip)
Shot a 40GB talking head interview 9:50 min.  (13663 frames) 12bit 1856x1044 23.976p+audio 44.1 kHz ,
no problem or issue , didn't have hdmi connected .
set to 800 ISO , lens set to f7 (ef24-70f2.8L) @24mm --  I had a hard time trying to find aliasing patterns
check it out , in the interview I had two ladies , one with striped clothing and looking at it I'm having a hard time
seeing  aliasing patterns , linked file below
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/M20-1417_000448.dng
I extracted with the new mlv_dump in the crop_rec_4k branch here -- I exported the frames at the same bit depth as I recorded (12bit) so each frame is 2.77MB
To exact to the correct bit depth with mlv_dump you need to add " --no-fixcp" to option if not they will be exported as 14 or 16bit no matter what bit depth you record at
Code: [Select]
mlv_dump --dng --no-fixcp M20-1417.MLVso if you record at 10bit or 12bit this will export to the same bit depth as recorded.

The other good thing is now all my slow CF Card (SanDisk-60MB/s) are now usable :) at a good resolution 10bit 1792x968 @23.976p+audio (write speed needed 49MB/s) continuous
would be nice if we could get other's to help get 10-12bit fully ported to 5d2 and all D4 cams .
wishful thinking :))

IDA_ML

  • Senior
  • ****
  • Posts: 429
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1667 on: May 22, 2018, 12:05:17 PM »
Excellent progress, Reddeercity!  I can't wait to see 10-12 bit working on the 7D too. 

timbytheriver

  • Member
  • ***
  • Posts: 113
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1668 on: May 22, 2018, 12:25:49 PM »
Great stuff redderrcity!

If the most recent experiments are yielding [almost] continuous recording in 12bit (but obviously no 3x crop mode) what exactly are the additional steps required to '...get 10-12bit fully ported to 5d2 and all D4 cams'?

Haven't you already achieved the goal?




reddeercity

  • Hero Member
  • *****
  • Posts: 1667
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1669 on: May 23, 2018, 06:43:54 AM »
@IDA_ML , thanks -- I thought 7D was partially working in 1:1 with 10-12bit on the same "LVState-wip" branch ? Maybe look under dfort bitbucket downloads .
If not I can provide a 7d test build with my changes , I don't have a 7d to test and I haven't got qmeu up and running just yet (very close) to test in the emulator.

Quote
If the most recent experiments are yielding [almost] continuous recording in 12bit (but obviously no 3x crop mode) what exactly are the additional steps required to '...get 10-12bit fully ported to 5d2 and all D4 cams'?
@timbytheriver , thanks
I can't really tell you or any one what steps to exactly take , most of this discover was really just a lucky break and some experiments push it to be some what functional .
If anything there needs to be more reverse engineering  research in to LiveView Sync .
The latest code for 10-12bit depends on "raw_slurp" which is a way to collect all the raw data from  channel"0" and use srm memory pack to write the data
(has 4 different address it writes from to speed up data transfer to the memory card). and at this time it's only fully supported in digic 5  cams .
There a big problem with LV sync ( not sure how to fix) that's what is messing up 3x crop_mode.
Quote
Haven't you already achieved the goal?
No , when there's a nightly build download in https://builds.magiclantern.fm/experiments.html for 10-12bit then I would have reach one of my goals .
There is a few broken thinks that I live with so I can use 10-12bit for production work .
1st: digital ISO (160,300,320,500, etc. ...) will lock up/freeze liveview , needs a battery pull to recover .
The reason (I believe) is the write channel I use 0x02 for "raw_slurp" (further investigation by a1ex noted that there a small amount data written to the same channel for time to time)
( the normal channel is 0x06 for basic ml nightly build in 14bit) this is the only channel that produces clean frames in bit reduction bit in 14bit there are still messed up so
only 10-12bit works no 14bit.
2nd: 3x crop_mode is broken , even in 14bit so all it's good for right now is for checking focus .
and again this is related to Liveview sync being messed up , It's much deeper then I can explain , if you read back a few pages I think you will get the idea .

So once raw_slurp is figured out on the digic 4 cams this will open them up to all the feature that digic5 cams have .

IDA_ML

  • Senior
  • ****
  • Posts: 429
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1670 on: May 23, 2018, 08:19:53 AM »
@IDA_ML , thanks -- I thought 7D was partially working in 1:1 with 10-12bit on the same "LVState-wip" branch ? Maybe look under dfort bitbucket downloads .
If not I can provide a 7d test build with my changes , I don't have a 7d to test and I haven't got qmeu up and running just yet (very close) to test in the emulator.

Yes, Dfort almost got it working without earthquake shaking on the 7D but it was not stable enough and 5x-magnification mode was not working on that build at all.

I am traveling right now but if you post a new 7D build with your changes, I will be happy to test when I come back in a week or so.

timbytheriver

  • Member
  • ***
  • Posts: 113
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1671 on: May 23, 2018, 11:25:35 AM »
Thanks. So sorting raw_slurp on the digic 4 cams is the deal-breaker to getting a stable nightly?

Maybe that could be pinned as a sticky to the head of the 5DMK2 thread so that anyone can see that as a important milestone?


IDA_ML

  • Senior
  • ****
  • Posts: 429
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1672 on: May 23, 2018, 01:40:45 PM »
Reddeercity,

On page 56 in my post 1390 I have provided detailed feedback on how the 7D works with Dfort's build that I mentioned in my previous post - the one without earthquake shaking in the Normal uncropped mode at 10/12 - bits.  I cannot find that build in his repository but I may have it somewhere on my home computer.  Unfortunately, as I said, I am traveling right now and do not have access to it.  If Dfort cannot find it either, I can look for it in a week from now, when I get back home.  Please read on from page 56.  You may find some more useful information for your further work.

Good luck!


dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3063
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1674 on: May 23, 2018, 06:43:53 PM »
I believe the build is still on my downloads page.

raw_video_10bit_12bit_LVState-wip.2018Feb04.7D203.zip

There are also some older experiments on the reddeercity repository downloads page.

My suggestion, as always, is to set up a development environment and try making some changes. For example, how much should we shave off of the SRM_BUFFER_SIZE in order to get this to work or is that not even necessary? You don't need to know how to code to try different values:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff#chg-src/raw.c
5D3.* 7D.206 700D.115 EOSM.203