Author Topic: 3K/UHD 5D2 Raw development and Other Digic IV Cams  (Read 10134 times)

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
3K/UHD 5D2 Raw development and Other Digic IV Cams
« on: April 06, 2017, 12:22:27 AM »
Ok by now a lot  if not all ml user know about crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Yes as it says I'm looking in to 3K/UHD on the 5d2 cam and being that most of the digic iv cam are similar
My hope is it will help on other cams like 7D , 50D , etc.... in reality only cam with CF Card have the potential
of recording 3K/UHD other SD cams maybe in short busts or reduce frame rate but I think the 5d2 and 7D should be capably
of 3K or close to it . That's my target , right now 10bit 2144x1076 @ 23.976p is a reality at about 60-63 MB/s Write speed
with a crop factor of 2.61 . Some may ask if this is possible on cams other then the 5d3 , will the whole discovery came from a
500d experiment from Greg , at least that's my understanding . To my knowledge a1ex reproduce the experiment on 5d3 digic v
which ended up code in the adtg_gui module  from iso_research branch . That's what I'm basing my development on and hints that a1ex gave my here ,
so the first step was to compile a adtg_gui module from the iso_research branch.
Of which I had many problem , thank to dfort for helping me with this issue the thread can read here. Once I load the module I can now get in to the cmos H & V.
I loaded the 5d3 3K registry preset in 3x crop mode and start play round .
I what so far I have discovered with the vertical registry I'm able to move the crop window from the left most leading edge to the right leading edge in 46 increments.
Stating at cmos[2] @ 0x10c <--0x40E
Code: [Select]
0x1c
0x2c
0x3c
0x4c
0x5c
0x6c
0x7c
0x8c
0x9c
0xac
0xbc
0xcc
0xec
0xfc
0x10c
0x11c
0x12c
0x13c
0x14c
0x15c
0x16c
0x17c
0x18c
0x19c
0x1ac
0x1bc
0x1cc
0x1dc
0x1ec
0x1fc
0x20c
0x21c
0x22c
0x23c
0x24c
0x25c
0x26c
0x27c
0x28c
0x29c
0x2ac
0x2bc
0x2cc
0x2dc
0x2ec

When I have any new developments  or share any info I'll post here and anyone that's wishes to help or build on my work Please join in
looking at you @Ilia3101  ::)   ;D




DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2904
  • 5D3 / M1 / 7D / 70D / SL1
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #1 on: April 06, 2017, 12:45:30 AM »
I'll be standing by you with all of my support for this. Also own a 7D and a coworker of mine owns a 5D2 whom is more than willingly to loan to me for to test along. Thanks for doing this @reddeercity!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Ilia3101

  • Member
  • ***
  • Posts: 110
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #2 on: April 06, 2017, 03:25:04 AM »
Juicy!
I will do whatever is useful. If you need me to try hundreds of values on registers I can do that. Will be back in a couple days and ready to do stuff(excited to look at the source code :o and the Gr3g resolution hack)
5D2

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #3 on: April 06, 2017, 05:42:57 AM »
@ DeafEyeJedi good to hear.

@ Ilia3101 sound good , if you go to my bitbucket_downloads there's a atdg_gui build for 5d2 that I'm using ,
get that and load the adtg_gui , raw_rec & file_manger modules keep it simple . In the submenu of adtg_gui in Advance , toggle on the digic registers
then at the bottom of the menu you will see 1:1 3K crop mode  (5d3) load that preset and go back to the root of adtg_gui and you will see 2 Green icon by cmos[1] Vertical  cmos[2]Horizontal
beware you will get lockups & liveview freeze , just pull the battery and restart etc.... . I use frame over ride @ 23.976p . Start reading the code here for hints & info on the registers plus look at this  post for more help.
From what I understand it's a combination of registers & video timers according to a1ex.

Ilia3101

  • Member
  • ***
  • Posts: 110
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #4 on: April 06, 2017, 07:02:11 AM »
Thank you. I've played around with the atgdgooi branch before, to be honest it scares me a little, as I suspect I've done damage to the sensor (I think it increased the fixed Patten noise) by playing with it. I also discovered 'green_rec' along with 'magenta_rec' and 'vertical_bar_rec'.
Will try out the 3k presets when I get back. The goal is to find the right values of those registers for the other cameras(5D2) to increase resolution in the same way? Or would it be controlled by other registers on other cameras? I guess this what we need to find to get it working. I will be reading the wiki for some homework ;D
5D2

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #5 on: April 06, 2017, 08:35:46 AM »
The goal is to find the right values of those registers for the other cameras(5D2) to increase resolution in the same way?
more or less , but first 5d2 then others .

Something interesting
cmos[1]0xaac <--0xeac
cmos[2]0x404 <--0x10E
looks like almost full view compared to the frame on the right which was not modified
the left all squeezed together .  I was using a EF24-70f2.8L at 24mm so in 3x crop mode that's 24mmx2.61=62.64mm
so I would say the left frame is very close to 24mm


full view crop mode
by RedDeerCityTV, on Flickr

Exported Jpeg from MLVProducer , I pushed the exposure to show detail.


M05-2224_00001
by RedDeerCityTV, on Flickr
Non modified registers cmos cdng 2K
https://www.dropbox.com/s/hzds2gbijp4tlzv/M05-2336_00000.dng?dl=0
Modified cmos[1] & cmos[2] cdng
https://www.dropbox.com/s/ng1zbz3xy8auopp/M05-2224_00000.dng?dl=0

It's a start  :)

Deadcode

  • New to the forum
  • *
  • Posts: 33
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #6 on: April 06, 2017, 08:49:58 AM »
Cheers! I hope you make it!

Is it possible to implement the lossless compression with older Digic 4 cameras?
With my calculations, the highest continuous resolution in crop mode might be 2736x1164 at 23,976 fps, with 2,1x crop ratio. That requires 128MB/s write speed and with the "usual" 58% compression is just about 74MB/s. That's around the max stable write speed with the 5D2 in crop mode.

Ilia3101

  • Member
  • ***
  • Posts: 110
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #7 on: April 06, 2017, 08:59:32 AM »
@Deadcode The compression is definitely possible in the 5D2 as it also has those features in the firmware that are being utilised on the 5D3 right now, but I think it takes an a1ex level of digic programming skills to do. So possibly may happen... or not. You can never predict.
@reddeercity squished crop mode would be pretty good, looks like there's a big black area on the right of that frame though. I bet getting rid of that pink corruption will be the hardest part.
Great start though!
5D2

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #8 on: April 07, 2017, 02:44:24 AM »
Yea I think compressed raw will work , I did a test merge of the compressed_raw branch in to unified source code and have a little problem with src/raw.c merge  that I hadn't had time to look in too fully but no crashes on the build .

I bet getting rid of that pink corruption will be the hardest part.

No , shouldn't be too big of a problem , I can clean up the corruption by changing addresses . Main thing I'm looking for is the full View with the right aspect ratio right now .

goldenchild9to5

  • Senior
  • ****
  • Posts: 489
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #9 on: April 07, 2017, 08:05:44 AM »
@reddeercity Great work..

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9926
  • 5D Mark Free
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #10 on: April 07, 2017, 08:11:05 AM »
Some of the values are actually bit fields. My old notes on 5D2 registers:

Code: [Select]
CMOS[0]: 0x203 at ISO 100
---- ------xx default 11, setting to 00 results in dark image with very low stdev
---- ---xxx-- ISO, values from 000 to 100, ascending (ISO 100-1600)
---- xxx----- second ISO field (as in dual ISO), disabled by default (see next bit)
---x -------- enables second ISO field (default 0)
--x- -------- enables vertical OB clamping maybe (setting it to 0 results in severe horizontal banding that looks like random walk)
-x-- -------- compresses the image horizontally (left side squashed, right side black)

CMOS[4]: 0x242 at iso 100...800 and 0x244 at iso 1600
---x -------- looks like some vertical dual ISO
-x-- -------- seems to cleanup 0.2 or 0.25 stops of shadow noise
No effect observed on the other bits.

dariSSight

  • Member
  • ***
  • Posts: 117
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #11 on: April 07, 2017, 01:45:18 PM »
@reddeercity Can't wait, but what is the atdg_gui build file for? So far the only way to use 10bit or your possible build is to push the magnify but for 5X to put in crop mode and you have to have a lens like 24-70mm 2.8L lens to view?
Canon 5D Mark II

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #12 on: April 07, 2017, 11:09:29 PM »
@reddeercity Can't wait, but what is the atdg_gui build file for?
For developing

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #13 on: April 07, 2017, 11:19:31 PM »
Some of the values are actually bit fields. My old notes on 5D2 registers:

Code: [Select]
CMOS[0]: 0x203 at ISO 100
-x-- -------- compresses the image horizontally (left side squashed, right side black)
Thanks  :) , interesting that's on cmos[2] 0x404 reproduces this .
I found more on Greg's work last night read thought the CMOS/ADTG/Digic register investigation on ISO thread , and it seems
I may have been adjusting the wrong registers (cmos 1&2) it seems he was adjusting cmos 5 &6 for digic iv cams
I'll give that a try .
No ISO related , seems it has more in common to 5D3

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #14 on: April 08, 2017, 09:51:39 AM »
Some more update/process , Getting cleaner image now with some surprise's  :D


M07-2145_00001
by RedDeerCityTV, on Flickr   5643x1076



M07-2149_00157
by RedDeerCityTV, on Flickr    5643x1076

Ok I had to do a little post work to get it to the right Aspect Ratio   ::), I still getting the compressed image but it's seem to be all there.
I crop off the black area then just increased the horizontal by 300% and left the vertical unchanged


M07-2145_00001
by RedDeerCityTV, on Flickr


M07-2149_00157
by RedDeerCityTV, on Flickr

Screenshots from crop mode , showing the capture area that's what recorded for reference only


5D2 Liveview
by RedDeerCityTV, on Flickr

I'm having problem with the FPS timer A , it doesn't want to go below 610 , I have no problem with timer B in fact I had it up to 3000 that help clean up the frame
it seems timer A is lock and I think that what stopping me for getting the full width capture not compressed to 2K frame , I had at about 17 fps to get a clean image

Or do I have to play with the Head Timers? I know there's three Can't find any info on them , I believes I need to get timer A to 455 or in that range and timer B about 2200 .

Cheers
Edit: This is frame with FHD 1856x1044 I used on my registers crop full view test , for reference only
 
M07-2337_00001
by RedDeerCityTV, on Flickr

Ilia3101

  • Member
  • ***
  • Posts: 110
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #15 on: April 08, 2017, 11:09:44 AM »
Good work. Does the black area go away if you move the crop window by the right amount? Yay now we have the 5D3 3x1 binning from crop rec!
Edit: technically the black bar can't go away unless you set the resolution to < 1881. Also that's insane spectrum ratio at 5.25:1
@a1ex genius. Are the registers 12 bit? That's weird ::)
5D2

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #16 on: April 09, 2017, 07:28:34 AM »
I'm getting more stability now ,
the registers I'm using is:
ADTG1 [105f]N-----0xa41
CMOS[1]------------0x2c7
CMOS[2]------------0x404

Exact FPS
FPS timer A --------610 (can't  lower it for some reason)
FPS timer B -------2000
main clock---------24.00 Mhz
Actual FPS---------19.672

Image is very clean now , no artifacts.
but image is still compressed to the left , I believe that because of FPS timer A is not be able to lowered from 610
I think there's something I'm missing , I feel I'm getting very close now If I can figure out the video timer problem .

Edit:

ADTG1 [105f]N-----0xab56
CMOS[1]------------0x2c7
CMOS[2]------------0x404
CMOS[5]------------0xc05

New setting that open up all registers , before I couldn't figure out way I didn't see the registers at where documented in other posts.
By accident I started to play around with CMOS[5] and BOOM there is all these Registers  :o
there go on forever . Now I access more RAW H&V and video timer stuff.
 


Ilia3101

  • Member
  • ***
  • Posts: 110
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #17 on: April 10, 2017, 04:48:41 PM »
@reddeercity I've now tried the iso-research branch and setting registers, your settings didn't work too well for me, I managed to get the squish working with cmos[0], but black level kept going up and down causing it to all look ugly, setting cmos[1] and others to values you recommend didn't help much other than locking up the camera. Basically i've not had any success.
But one more thing: when the correct registers/settings to unlock resolution are found, will it automatically be detected in the raw_rec or mlv module so the higher resolution can be set straight away?(I mean: does it auto detect max resolution?)
5D2

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 946
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #18 on: April 10, 2017, 04:57:54 PM »
once the sweet spot(s) for 5D2 are found you could for e.g. adjust/activate your new "crop mode / resolution" via ML menu for the moment. Check a1ex's 2016 implementation
70D.112 & 100D.101

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #19 on: April 11, 2017, 02:29:16 AM »
@Ilia3101 ,when selecting registers I find it's more temperamental the warmer the sensor gets . There is a certain  order you have to do and of course the video timers have to adjusted or it will not work not matter what you do. 1st set to low light , video timer A ( low as it can go) , video timer B (2000 and up) works in the adtg_gui  advance menu enable digic registers then apply the 1:1 3k preset (which sets cmos[1] & cmos[2] the I adjust cmos[2] first for width then cmos[1] for height next is ADTG1[105f]N (x5 & x10 zoom shutter blanking), move the frame up & down by either by shorting or lengthen shutter duration, to fix pink frame you may have to find a different registers in cmos[1] oh yea also enable cmos[5] helps with pink hi-lights you may also need to play with frame over ride a bit , seems that the frame is clean around 18-20 fps not sure why. Also It looks like I may have got the 1:1 Crop_record working on the 5D2 so you don't have to be in zoom mode for it to work  ;D  As it say 1:1  but at 1856 not 5616 but I can move the capture area cross the sensor just like 3x crop ,  plus in that mode I was able to record a H264 in 1:1when I turn off raw video but the problem I having and you know there's always a problem  ::) I can only record 1856 width(center crop) but full height which is compressed vertically , once I stretch it out it's 1865x3744 .

1:1 Crop_recording with 3K preset , This is Not 3x Crop mode but 1:1 , like I said with compressed vertically . 1856x1044


M10-021100001
by RedDeerCityTV, on Flickr 

After I resized the vertical by about 258% I get 1856x2694 close to full height of the sensor but at 16x9
 

1856x2694-M10-021100057
by RedDeerCityTV, on Flickr

1:1 crop_record H264 from the 5D2 , turn off the raw video and hit record  :) , Maybe Highly Compressed h264 3K or 4K  :P :P
Just click on the image and ficker will link to the file on page. or watch off dropbox see link below


MVI_8653
by RedDeerCityTV, on Flickr

So how do I know if it's truly 1:1 , will I don't really only thing that tells me is the fact there no moiré or aliasing pattern
look at the the fine hole on my MacPro even in h264 I can't tell , thou the lens I use was 24mm so no lens correction
which give the distortion width at the top .

H264 1:1 crop_record MVI_8653.MOV source file link from my dropbox if any one what to download 45Mb

 

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #20 on: April 11, 2017, 09:01:21 AM »
Doing some more quick tests in 1:1 LiveView with the following setting
adtg1[1061] 0x449
cmos[1] 0x3e0
cmos[2] 0x104
cmos[5] 0xc21

23.976 from 24
Video Timer A 584 (FT +12)
Video Timer B 3422 (FT +1672)
Actual frame rate 12.00

crop_record 1:1 (Not 100% sure it working but seems to setup a lot easier and centered
1:1 3K preset that's it .

3000x1250

M10_3000x1250
by RedDeerCityTV, on Flickr


3000x1250

M10_2_3000x1250
by RedDeerCityTV, on Flickr

There's a little trick I did to get this to work , I post tomorrow on what the special thing is  8)
The hint is in the frame size   ;) 

ProRes files of the 3K test  , not big  about 100Mb each
M10_1.mov
M10_2.mov

Ilia3101

  • Member
  • ***
  • Posts: 110
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #21 on: April 11, 2017, 09:27:40 AM »
I see the hint: 1250 ;) Maybe you got something like the vertical compression in the post before but sideways? just a guess...
5D2

goldenchild9to5

  • Senior
  • ****
  • Posts: 489
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #22 on: April 12, 2017, 01:00:39 AM »
@reddeercity Great job  :) Now I wanna buy a 5D2 as a back-up to my 5D3. 

Greg

  • Hero Member
  • *****
  • Posts: 550
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #23 on: April 12, 2017, 02:51:07 AM »
This will not work on old cameras (500D, 5D2 etc) until you find a way to change the EDMAC size.

Old cameras do not support #ifdef CONFIG_EDMAC_RAW_SLURP
https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-429

Maybe patch here :
Code: [Select]
LiveViewMg:000929e0:00:00: *** ConnectWriteEDmac(0x10, 0x0), from ff1f7134
LiveViewMg:000929e0:00:00: *** SetEDmac(0x10, 0x463390a4, 0xff44e5c8, 0x20000000), from ff1f7164

Or try "#define CONFIG_ALLOCATE_RAW_LV_BUFFER"
https://bitbucket.org/hudson/magic-lantern/src/11f405b62b312294d705e8e7bb8688b820bb9eea/src/raw.c?at=crop_rec_4k&fileviewer=file-view-default#raw.c-126

reddeercity

  • Hero Member
  • *****
  • Posts: 1315
Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams
« Reply #24 on: April 12, 2017, 03:05:24 AM »
Thanks for the info Greg I looking for something like this